Unknown nodes everywhere

I am running both the client and the server on 5.8.0-dev.
The proxy config is this:
{
	"Servers": {
		"Lobby": {
			"Addr": "<IP_ADDRESS>:30002",
		}
	},
	"BindAddr": ":30000",
	"TelnetAddr": ":30001",
	"NoPlugins": false,
	"CmdPrefix": ">",
	"UserLimit": 100,
	"ForceDefaultSrv": true,
	"List": {
		"Enable": true,
		"Addr": "vps.brn.systems",
		"Interval": 300,
		"Name": "BRN Systems",
		"Desc": "The BRN Systems network",
		"URL": "brn.systems",
		"Creative": true,
		"Dmg": true,
		"PvP": true,
		"Game": "BRN Systems",
		"FarNames": true
	},
	"Groups":{
		"admin": [
			"cmd_shutdown",
			"cmd_find",
			"cmd_addr",
			"cmd_alert",
			"cmd_send",
			"cmd_players",
			"cmd_reload",
			"cmd_group",
			"cmd_perms",
			"cmd_gperms",
			"cmd_server",
			"cmd_kick",
			"cmd_ban",
			"cmd_unban",
			"cmd_uptime",
			"cmd_help",
			"cmd_usage",
			"interact",
			"home",
			"shout"
		],
		"default": [
			"interact",
			"shout"
		]

	},
	"UserGroups": {
		"BRNSystems": "admin"
	}

}

And the minetest lobby config is quite long, since i copied the example config.
This is a screenshot of a singlenode world with some cobblestone.

Is this a bug?

This is the log of my minetest client in verbose. Node defs and item defs are very small(Probably dont have anything in them)
Even my wieldhand is undefined.
Linux vezicka 6.6.3-arch1-1 x86_64
 17:51:15 up 22:08,  1 user,  load average: 2,37, 2,07, 1,92
minetest --verbose
2023-12-01 17:51:27: INFO[Main]: Using system-wide paths (NOT RUN_IN_PLACE)
2023-12-01 17:51:27: INFO[Main]: Detected share path: /usr/local/share/minetest
2023-12-01 17:51:27: INFO[Main]: Detected user path: /home/bruno/.minetest
2023-12-01 17:51:27: INFO[Main]: Detected cache path: /home/bruno/.cache/minetest
2023-12-01 17:51:27: INFO[Main]: Using static locale directory /usr/local/share/locale
2023-12-01 17:51:27: INFO[Main]: Minetest 5.8.0-dev-cdb4a5d8c
2023-12-01 17:51:27: INFO[Main]: with SER_FMT_VER_HIGHEST_READ=29, BUILD_TYPE=Release
2023-12-01 17:51:27: INFO[Main]: RUN_IN_PLACE=0
2023-12-01 17:51:27: INFO[Main]: USE_CURL=1
2023-12-01 17:51:27: INFO[Main]: USE_GETTEXT=1
2023-12-01 17:51:27: INFO[Main]: USE_SOUND=1
2023-12-01 17:51:27: INFO[Main]: STATIC_SHAREDIR="/usr/local/share/minetest"
2023-12-01 17:51:27: INFO[Main]: STATIC_LOCALEDIR="/usr/local/share/locale"
2023-12-01 17:51:27: VERBOSE[Main]: httpfetch_init: parallel_limit=8
2023-12-01 17:51:27: INFO[Main]: Gettext: domainname="minetest" path="/usr/local/share/locale"
2023-12-01 17:51:27: INFO[Main]: Message locale is now set to: LC_CTYPE=en_US.UTF-8;LC_NUMERIC=C;LC_TIME=sk_SK.UTF-8;LC_COLLATE=en_US.UTF-8;LC_MONETARY=sk_SK.UTF-8;LC_MESSAGES=en_US.UTF-8;LC_PAPER=sk_SK.UTF-8;LC_NAME=en_US.UTF-8;LC_ADDRESS=en_US.UTF-8;LC_TELEPHONE=en_US.UTF-8;LC_MEASUREME
2023-12-01 17:51:27: INFO[Main]: NT=sk_SK.UTF-8;LC_IDENTIFICATION=en_US.UTF-8
2023-12-01 17:51:27: INFO[Main]: Searching worlds...
2023-12-01 17:51:27: INFO[Main]:   In /home/bruno/.minetest/worlds: test
2023-12-01 17:51:27: INFO[Main]: 1 found.
2023-12-01 17:51:27: [Main]: Automatically selecting world at [/home/bruno/.minetest/worlds/test]
2023-12-01 17:51:27: INFO[Main]: Using world gameid [minetest]
2023-12-01 17:51:27: INFO[Main]: Audio: Global Initialized: OpenAL 1.1 ALSOFT 1.23.1, using OpenAL Soft
2023-12-01 17:51:27: INFO[Main]: Irrlicht: Irrlicht Engine version 1.9.0mt13
2023-12-01 17:51:27: INFO[Main]: Irrlicht: Linux 6.6.3-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 29 Nov 2023 00:37:40 +0000 x86_64
2023-12-01 17:51:27: INFO[Main]: Irrlicht: Using renderer: OpenGL 4.6
2023-12-01 17:51:27: INFO[Main]: Irrlicht: AMD Radeon Graphics (raphael_mendocino, LLVM 16.0.6, DRM 3.54, 6.6.3-arch1-1): AMD
2023-12-01 17:51:27: INFO[Main]: Irrlicht: OpenGL driver version is 1.2 or better.
2023-12-01 17:51:27: INFO[Main]: Irrlicht: GLSL version: 4.6
2023-12-01 17:51:27: INFO[Main]: Using the OpenGL video driver
2023-12-01 17:51:27: INFO[Main]: Waiting for other menus
2023-12-01 17:51:27: INFO[Main]: Waited for other menus
2023-12-01 17:51:27: INFO[Main]: Audio: Initialized: OpenAL
2023-12-01 17:51:27: INFO[Main]: GUIEngine: Initializing Lua
2023-12-01 17:51:27: VERBOSE[Main]: Loading and running script from /usr/local/share/minetest/builtin/init.lua
2023-12-01 17:51:27: INFO[Main]: Initializing asynchronous environment
2023-12-01 17:51:27: VERBOSE[Main]: Loading and running script from /usr/local/share/minetest/builtin/init.lua
2023-12-01 17:51:27: INFO[Main]: Initializing asynchronous environment
2023-12-01 17:51:27: VERBOSE[Main]: Loading and running script from /usr/local/share/minetest/builtin/init.lua
2023-12-01 17:51:27: INFO[Main]: Initializing asynchronous environment
2023-12-01 17:51:27: VERBOSE[Main]: Loading and running script from /usr/local/share/minetest/builtin/init.lua
2023-12-01 17:51:27: INFO[Main]: Initializing asynchronous environment
2023-12-01 17:51:27: INFO[Main]: SCRIPTAPI: Initialized main menu modules
2023-12-01 17:51:27: VERBOSE[Main]: Loading and running script from /usr/local/share/minetest/builtin/init.lua
2023-12-01 17:51:27: INFO[Main]: Searching worlds...
2023-12-01 17:51:27: INFO[Main]:   In /home/bruno/.minetest/worlds: test
2023-12-01 17:51:27: INFO[Main]: 1 found.
2023-12-01 17:51:27: VERBOSE[AsyncWorker-1]: httpfetch_caller_alloc_secure: allocating<PHONE_NUMBER>531404198
2023-12-01 17:51:27: INFO[AsyncWorker-1]: Mod performs HTTP request with URL servers.minetest.net/list?proto_version_min=37&proto_version_max=43
2023-12-01 17:51:27: VERBOSE[AsyncWorker-0]: httpfetch_caller_alloc_secure: allocating<PHONE_NUMBER>962887105
2023-12-01 17:51:27: INFO[AsyncWorker-0]: Mod performs HTTP request with URL servers.minetest.net/geoip
2023-12-01 17:51:27: VERBOSE[AsyncWorker-2]: httpfetch_caller_alloc_secure: allocating<PHONE_NUMBER>6707818668
2023-12-01 17:51:27: INFO[AsyncWorker-2]: Mod performs HTTP request with URL https://content.minetest.net/api/updates/?type=mod&type=game&type=txp&protocol_version=43&engine_version=5.8.0-dev
EKEY_CODE not found, X11 keycode: 237
EKEY_CODE not found, X11 keycode: 237
2023-12-01 17:51:30: INFO[Main]: GUIEngine: Deinitializing scripting
2023-12-01 17:51:30: INFO[Main]: Audio: Deinitializing...
2023-12-01 17:51:30: INFO[Main]: ~MenuTextureSource() before cleanup: 18 after: 3
2023-12-01 17:51:30: INFO[Main]: Searching worlds...
2023-12-01 17:51:30: INFO[Main]:   In /home/bruno/.minetest/worlds: test
2023-12-01 17:51:30: INFO[Main]: 1 found.
2023-12-01 17:51:30: INFO[Main]: Selected world: [--world parameter] [/home/bruno/.minetest/worlds/test]
2023-12-01 17:51:30: INFO[Main]: SourceImageCache::getOrLoad(): Loading path "/usr/local/share/minetest/textures/base/pack/progress_bar.png"
2023-12-01 17:51:30: INFO[Main]: SourceImageCache::getOrLoad(): Loading path "/usr/local/share/minetest/textures/base/pack/progress_bar_bg.png"
2023-12-01 17:51:30: INFO[Main]: Attempting to use OpenAL audio
2023-12-01 17:51:30: INFO[Main]: Audio: Initialized: OpenAL
2023-12-01 17:51:30: INFO[Main]: MeshUpdateManager: using 4 threads
2023-12-01 17:51:30: INFO[Main]: SourceImageCache::getOrLoad(): Loading path "/usr/local/share/minetest/textures/base/pack/minimap_mask_round.png"
2023-12-01 17:51:30: INFO[Main]: SourceImageCache::getOrLoad(): Loading path "/usr/local/share/minetest/textures/base/pack/minimap_overlay_round.png"
2023-12-01 17:51:30: INFO[Main]: SourceImageCache::getOrLoad(): Loading path "/usr/local/share/minetest/textures/base/pack/minimap_mask_square.png"
2023-12-01 17:51:30: INFO[Main]: SourceImageCache::getOrLoad(): Loading path "/usr/local/share/minetest/textures/base/pack/minimap_overlay_square.png"
2023-12-01 17:51:30: INFO[Main]: SourceImageCache::getOrLoad(): Loading path "/usr/local/share/minetest/textures/base/pack/player_marker.png"
2023-12-01 17:51:30: INFO[Main]: SourceImageCache::getOrLoad(): Loading path "/usr/local/share/minetest/textures/base/pack/object_marker_red.png"
2023-12-01 17:51:30: INFO[Main]: Connecting to server at <IP_ADDRESS>:30000
2023-12-01 17:51:30: INFO[Main]: Client::peerAdded(): peer->id=1
2023-12-01 17:51:30: INFO[Main]: Client packetcounter (30s): sum=0 avg=0/s
2023-12-01 17:51:30: INFO[Main]: Client: TOCLIENT_HELLO received with serialization_ver=29, auth_mechs=2, proto_ver=42, compression_mode=0.  in textdomain "__builtin"
2023-12-01 17:51:32: INFO[Main]: Client: Detached inventory update: "i3_bag_content_BRNSystems", mode=update
2023-12-01 17:51:32: INFO[Main]: Client: Detached inventory update: "i3_bag_BRNSystems", mode=update
2023-12-01 17:51:38: INFO[Main]: Game lost focus
2023-12-01 17:51:39: INFO[Main]: Game lost focus
2023-12-01 17:51:40: INFO[Main]: Client: avg_rtt=0.015
2023-12-01 17:51:41: INFO[Main]: Game lost focus
2023-12-01 17:51:50: INFO[Main]: Client: avg_rtt=0.015
2023-12-01 17:52:00: INFO[Main]: Client packetcounter (30s): sum=40 avg=1.33333/s
2023-12-01 17:52:00: INFO[Main]: cmd 2 (TOCLIENT_HELLO) count 1
2023-12-01 17:52:00: INFO[Main]: cmd 3 (TOCLIENT_AUTH_ACCEPT) count 1
2023-12-01 17:52:00: INFO[Main]: cmd 32 (TOCLIENT_BLOCKDATA) count 1
2023-12-01 17:52:00: INFO[Main]: cmd 39 (TOCLIENT_INVENTORY) count 2
2023-12-01 17:52:00: INFO[Main]: cmd 41 (TOCLIENT_TIME_OF_DAY) count 1
2023-12-01 17:52:00: INFO[Main]: cmd 42 (TOCLIENT_CSM_RESTRICTION_FLAGS) count 2
2023-12-01 17:52:00: INFO[Main]: cmd 47 (TOCLIENT_CHAT_MESSAGE) count 2
2023-12-01 17:52:00: INFO[Main]: cmd 51 (TOCLIENT_HP) count 1
2023-12-01 17:52:00: INFO[Main]: cmd 52 (TOCLIENT_MOVE_PLAYER) count 1
2023-12-01 17:52:00: INFO[Main]: cmd 56 (TOCLIENT_MEDIA) count 1
2023-12-01 17:52:00: INFO[Main]: cmd 58 (TOCLIENT_NODEDEF) count 1
2023-12-01 17:52:00: INFO[Main]: cmd 60 (TOCLIENT_ANNOUNCE_MEDIA) count 1
2023-12-01 17:52:00: INFO[Main]: cmd 61 (TOCLIENT_ITEMDEF) count 1
2023-12-01 17:52:00: INFO[Main]: cmd 65 (TOCLIENT_PRIVILEGES) count 1
2023-12-01 17:52:00: INFO[Main]: cmd 66 (TOCLIENT_INVENTORY_FORMSPEC) count 2
2023-12-01 17:52:00: INFO[Main]: cmd 67 (TOCLIENT_DETACHED_INVENTORY) count 6
2023-12-01 17:52:00: INFO[Main]: cmd 69 (TOCLIENT_MOVEMENT) count 1
2023-12-01 17:52:00: INFO[Main]: cmd 73 (TOCLIENT_HUDADD) count 2
2023-12-01 17:52:00: INFO[Main]: cmd 75 (TOCLIENT_HUDCHANGE) count 1
2023-12-01 17:52:00: INFO[Main]: cmd 77 (TOCLIENT_HUD_SET_PARAM) count 4
2023-12-01 17:52:00: INFO[Main]: cmd 78 (TOCLIENT_BREATH) count 1
2023-12-01 17:52:00: INFO[Main]: cmd 81 (TOCLIENT_LOCAL_PLAYER_ANIMATIONS) count 1
2023-12-01 17:52:00: INFO[Main]: cmd 86 (TOCLIENT_UPDATE_PLAYER_LIST) count 2
2023-12-01 17:52:00: INFO[Main]: cmd 96 (TOCLIENT_SRP_BYTES_S_B) count 1
2023-12-01 17:52:00: INFO[Main]: cmd 97 (TOCLIENT_FORMSPEC_PREPEND) count 1
2023-12-01 17:52:00: INFO[Main]: cmd 99 (TOCLIENT_SET_LIGHTING) count 1
2023-12-01 17:52:00: INFO[Main]: Client: avg_rtt=0.015

Oops, didnt mean to close

TL;DR Try with Minetest 5.7.0 and stop exposing telnet publicly.
Unknown nodes
This probably isn't a bug. Each commit only supports one Minetest version (usually the patch number doesn't matter though: a proxy commit that works with 5.7.0 shouldn't break with 5.7.1 if it existed). This can't really be changed due to different packet serialization approaches (you can ask for the technical details if you want to know them).
Minetest 5.8.0-dev isn't supported, only 5.7.0 is on the latest commits. The official 5.8.0 is going to be supported, maybe even out of the box.
At first glance https://github.com/minetest/minetest/commit/aea9242a968ad4cf92fe528ce42e40066b35e10b seems to have introduced this issue but I can't test this yet.
Dev versions aren't supported by this proxy because doing so would be a monumental task. I'd like to solve this but as I said before coming up with a solution for this is very difficult.
Please try again using Minetest 5.7.0. The proxy logs would also be very interesting as they contain some information on serialization failures.
Concerns with your config
Please never expose the telnet interface to the public internet. It grants everyone the ability to run privileged chat commands without authentication.
The telnet listening address defaults to ip6-localhost for a reason. You should only use secure methods to access the telnet server, including the following:

SSH into the server and run the telnet client there (let the server listen on (ip6-)localhost)
Use an encrypted tunnel (VPN, etc.) to do the same and prevent public access (firewall)
Limit the source IP addresses using a firewall (WARNING: IP spoofing is still possible and can potentially be used to execute commands without getting a response)

Method 1 is probably the best one. If anyone else has access to the system you should consider blocking telnet connections from loopback.
Also there is no need to set options to their default value unless you're doing it for stylistic reasons ("NoPlugins": false, and "CmdPrefix": ">", can be removed).

It is running on an oracle cloud vps, so there is another layer of firewall(the telnet is accessible only from that machine)
The logs of the proxy is this (I didnt see any errors):
[minetest@brnoracle bin]$ ./mt-multiserver-proxy
2023/12/01 18:42:10 load config
2023/12/01 18:42:10 [proxy] version: v0.0.0-20231126154428-f3c57a3b3585
2023/12/01 18:42:10 [proxy] load plugins
2023/12/01 18:42:10 [proxy] listen [::]:30000
2023/12/01 18:42:10 [proxy] listen telnet [::]:30001
2023/12/01 18:42:15 [<IP_ADDRESS>:53258] -> connect
2023/12/01 18:42:16 [content Lobby] <- *mt.ToCltItemDefs: github.com/HimbeerserverDE/mt.ItemDef: 1 bytes of trailing data
2023/12/01 18:42:16 [content Lobby] <- *mt.ToCltNodeDefs: 1 bytes of trailing data
2023/12/01 18:42:16 [<IP_ADDRESS>:53258 BRNSystems] <-> handshake completed
2023/12/01 18:42:16 [server Lobby] -> connect
2023/12/01 18:42:16 [<IP_ADDRESS>:53258 BRNSystems] -> *mt.ToSrvPlayerPos: trailing data: 00
2023/12/01 18:42:16 [<IP_ADDRESS>:53258 BRNSystems] -> *mt.ToSrvPlayerPos: trailing data: 00
2023/12/01 18:42:17 [server Lobby] <- *mt.ToCltItemDefs: github.com/HimbeerserverDE/mt.ItemDef: 1 bytes of trailing data
2023/12/01 18:42:17 [server Lobby] <- *mt.ToCltNodeDefs: 1 bytes of trailing data
2023/12/01 18:42:17 [server Lobby] <-> handshake completed
2023/12/01 18:42:17 [<IP_ADDRESS>:53258 BRNSystems] -> *mt.ToSrvPlayerPos: trailing data: 00
2023/12/01 18:42:17 [<IP_ADDRESS>:53258 BRNSystems] -> *mt.ToSrvPlayerPos: trailing data: 00
2023/12/01 18:42:17 [server Lobby] <- *mt.ToCltAORmAdd: github.com/HimbeerserverDE/mt.AOAdd: github.com/HimbeerserverDE/mt.AOInitData: 28 bytes of trailing data
2023/12/01 18:42:17 [server Lobby] <- *mt.ToCltAOMsgs: github.com/HimbeerserverDE/mt.IDAOMsg: 28 bytes of trailing data
2023/12/01 18:42:20 [<IP_ADDRESS>:53258 BRNSystems] <-> disconnect
2023/12/01 18:42:20 [server Lobby] <-> disconnect

There are errors: "trailing data". This has happened before and hints at changes to those packet types that break compatibility. Trailing data errors prevent packets from being processed. In this case node and item definitions from the game servers aren't processed so the proxy sends empty / builtin defs to the client as you suspected. This is most likely caused by 5.8.0-dev.
You can still try stable 5.7.0.
In any case, thanks for the logs. They have already provided valuable info on what needs to be done for 5.8.0 support once it's released.

Thanks. I will try 5.7.0.
Btw. How many breaking changes could 5.8.0 have introduced?

Does this proxy support split packets?
When I looked at the communication in wireshark, it seemed to send item data and node definitions in split packets.
But in the proxied connection, I have not seen a split packet.

Thanks. I will try 5.7.0. Btw. How many breaking changes could 5.8.0 have introduced?

I'd say there have been changes to all packets found in the proxy logs: item defs, node defs, player position and active object messages. Figuring out the details and updating requires reading the Minetest source code though. Anything in src/network/ is interesting but some serialization changes happen in other files like src/nodedef.* or src/itemdef.*.
It looks surprisingly bad compared to some of the other updates like the one to 5.4.1.

Does this proxy support split packets? When I looked at the communication in wireshark, it seemed to send item data and node definitions in split packets. But in the proxied connection, I have not seen a split packet.

Yes, the proxy has full split packet support. The reason you don't see any is that the packets are small enough to be transmitted without fragmentation.
Technical background: The proxy doesn't actually forward anything. It reads the high-level payload of a packet, does some processing and wraps the final result in a new packet of the same type. Low-level metadata (peer ID, channel, reliability, splitting, etc.) is automatically recalculated because it may need to be different (for example the true client peer ID is not necessarily equal to the one the real server assigns to the proxy). We call this a layer 7 proxy because it works on OSI layer 7 (Minetest connections, not UDP). The low-level info doesn't affect high-level packet handling (this applies to Minetest too).

Can confirm that 5.8.0-dev is severely broken with similar log messages, but 5.7.0 is working fine. Proxy commit: 478f2b2db4e0, version: v0.0.0-20231201174547-478f2b2db4e0.

mtstuff.pcapng.zip
If it helps, here is the complete communication between server, proxy and client.
port 30000 is proxy
server runs on 30001
The game is void(smallest one)
Proxy config:
{
        "Servers": {
                "ServerName1": {
                        "Addr": "<IP_ADDRESS>:30001"
                }
        },
        "BindAddr": ":30000",
        "TelnetAddr": ":3000"
        }
}

Marking this as "required" as Minetest 5.8.0 was officially released yesterday. Support will be added soon depending on how severe the changes are.

It took some time but Minetest 5.8.0 is now supported by the proxy (merged #145) and tested to be working. This breaks compatibility with 5.7. Please update the proxy to the latest version using the installation instructions in README.md and try again.
In order to prevent mismatched versions in the future please set strict_protocol_version_checking = true in your minetest.conf file. This way you'll get an error message instead of a broken game.

Since this issue has been inactive for about two months now and my own testing confirms that it is now fixed I'll close this. Please reopen it or file a new issue if you're still experiencing the problem after updating the proxy to the latest version.


Request : enable longer exposure times than 30s in non bulb mode

Is there a way to enable longer exposures than 30s in non bulb mode? Can the behaviour of when the camera saves full 14bit raw be influenced in addition? Thanks in advance,
Ben

+1
It's found that the Bulb mode will cause 'star eater problem'. If  longer exposures than 30s in M mode can be enabled, it is a good news for astro-photographer.
Thank you!

+1

+1

+1. Just a note with 3.3 firmware they eat stars after 3.2 seconds. Even taking pictures of the city at night long than 3.2 seconds will eat the detail of the lights in the city. See here for examples:
http://community.sony.com/t5/Alpha-SLT-DSLR-Cameras/star-eater-algorithm-in-Bulb-mode-shots-please-offer-alternative/td-p/619350


Allow !Unpin Futures in select!

This adds support for directly select!ing on !Unpin Futures as long
as those Futures are returned by expressions by stack-pinning the
Futures.
For the common case where the futures are the result of an async fn
call this will increase the ergonomics of the macro a lot, since no
explicit binding and stack pinning is required.
Partially resolves #1811 (it does not affect the borrow-checking
requirements)

This has already been merged as part of #1874


Organized point clouds

(This issue continues the discussion started in #342 about organized point clouds)
Organized point clouds are an attractive way to represent 3D point sets coming from range sensors. The main advantage of them is that they are akin to matrices, thus easier to traverse. Furthermore, the neighboring set of each point is known a priori, so operations like normal computations could potentially skip the K-NN step, reducing computation time and CPU/memory taken in building a K-D tree.
Existing implementations of organized point clouds (pcl, sensor_msgs/PointCloud2) represent the range measurements in a flat data structure akin to a matrix, where each cell corresponds to a range sensor bin. Thus the range sensor FOV is discretized horizontally and vertically. Missing points are usually filled with NaN.
I'm opening this issue to discuss how organized point clouds could be represented within the framework of libpointmatcher. Some points for discussion I can think of right now:

Whether we can re-use/extend the current DataPoints class to host the missing point ordering information.
If it would make sense to actually use a different representation. For example, a matrix with multiple channels, one for each descriptor (similar to a cv::Mat from OpenCV)

I'm currently working on speeding up surface normals computation on DataPoints structures. I'm planning to implement an 'Index Image', a structure that contains a mapping between a 2D matrix (an organized point cloud grid) and 1D (DataPoints) elements. This approach is inspired by Serafin et al
I consider it a better option than depth images / depth buffers, as it seems more general and straightforward to perform a mapping in 3D space with a direction table, than to map point clouds from arbitrary sensors into depth images (for example, dome LIDARs).
We plan to use this as a mean to speed up feature sampling filters, initially.


docker demon failed

$ docker-osx-dev start
[INFO] Starting Docker and Vagrant
Bringing machine 'boot2docker' up with 'virtualbox' provider...
==> boot2docker: Importing base box 'blinkreaction/boot2docker'...
==> boot2docker: Matching MAC address for NAT networking...
==> boot2docker: Setting the name of the VM: nodeschool-hk_boot2docker
==> boot2docker: Clearing any previously set network interfaces...
==> boot2docker: Preparing network interfaces based on configuration...
    boot2docker: Adapter 1: nat
    boot2docker: Adapter 2: hostonly
==> boot2docker: Forwarding ports...
    boot2docker: 2375 => 2375 (adapter 1)
    boot2docker: 2376 => 2376 (adapter 1)
    boot2docker: 22 => 2222 (adapter 1)
==> boot2docker: Running 'pre-boot' VM customizations...
==> boot2docker: Booting VM...
==> boot2docker: Waiting for machine to boot. This may take a few minutes...
    boot2docker: SSH address: <IP_ADDRESS>:2222
    boot2docker: SSH username: docker
    boot2docker: SSH auth method: private key
    boot2docker: Warning: Connection timeout. Retrying...
==> boot2docker: Machine booted and ready!
==> boot2docker: Checking for guest additions in VM...
==> boot2docker: Configuring and enabling network interfaces...
==> boot2docker: Rsyncing folder: /Users/abtris/Sites/nodeschool-hk/ => /Users/abtris/Sites/nodeschool-hk
==> boot2docker:   - Exclude: [".vagrant/", "npm-debug.log", ".grunt", ".git", ".sass-cache/", ".DS_Store", ".AppleDouble", ".LSOverride", ".cordova", "node_modules", ".vagrant"]
==> boot2docker: Mounting shared folders...
    boot2docker: /vagrant => /Users/abtris/Sites/nodeschool-hk
==> boot2docker: Running provisioner: shell...
    boot2docker: Running: inline script

$ docker build .
Sending build context to Docker daemon
FATA[0000] An error occurred trying to connect: Post https://localhost:2375/v1.18/build?cpusetcpus=&cpushares=0&dockerfile=Dockerfile&memory=0&memswap=0&rm=1&t=: tls: oversized record received with length 20527

$ echo $DOCKER_HOST
tcp://localhost:2375

Idea how solve this? With normal boot2docker, I don't have any this type of problems.

If I try vagrant ssh
docker@boot2docker:~$ docker ps
FATA[0000] Get http:///var/run/docker.sock/v1.18/containers/json: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?

some problem as another disussion: unset DOCKER_CERT_PATH DOCKER_TLS_VERIFY helps.


Generative models using ConditionalDists.jl

This includes a lot of vanilla models that can be implemented via ConditionalDists.jl and GenerativeModels.jl

VAE finished

[ ] VAE
[ ] WAE
[ ] AAE
[ ] WAE with Vamp
[ ] AAE with Vamp
[ ] GAN
[ ] fmGAN

Implemented via https://github.com/aicenter/GenerativeAD.jl/pull/31, https://github.com/aicenter/GenerativeAD.jl/pull/34, https://github.com/aicenter/GenerativeAD.jl/pull/38 and https://github.com/aicenter/GenerativeAD.jl/pull/43.


Syscall dependency feature request

Hi,
The current syzkaller implementation feeds the syscalls with random data, while in some case that syscalls are needed to be executed in order and params to syscalls are needed to be global available.
For example for ion and its user driver xx below, the call sequences are:
/* allocate ion /
syz_open_dev("xx", ...) fd_xx
/ suppose arg1 (inout ptr) is
struct arg1 {
handle int32 // handle has the ion handler allocated by below ioctl call
}
ioctl$xx_ion(fd_xx, cmd, arg1)
/* Open another device yy which needs to use the handle allocated above */
syz_open_dev("yy", ...) fd_yy
ioctl$yy(fd_yy, cmd, arg1->handle)
If not open ion first, we cannot fuzz ioctl$yy(fd_yy, cmd, arg1->handle), so can you please consider this new feature to pass arg1->handle to 2nd syscall?
Thanks

In this case we need to pass a value returned from one syscall to another syscall. That's what resources are for. That's used throughout the descriptions, e.g. open/socket/pipe return fd that needs to be passed to other syscalls. Look for examples.
Also, we already have ion descriptions. Check out how that's described:
https://github.com/google/syzkaller/blob/master/sys/linux/ion.txt


#159570479 Update npm scripts to improve app startup time

What does this PR do?
Update npm scripts to improve app startup time
Description of Task to be completed?

Remove npm prestart script
Add heroku-postbuild script

How should this be manually tested?
N/A
Any background context you want to provide?
To improve app startup time
What are the relevant pivotal tracker stories?(if applicable)
#159570479

Pull Request Test Coverage Report for Build 183

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 99.147%

Totals

Change from base Build 181:
0.0%

Covered Lines:
1250

Relevant Lines:
1253

💛  - Coveralls


Log web app save confirmation text

Fixes vscode-azureappservice#1509

Save confirmation text is now logged as well as put in the notification.
Like this:

So for this issue (vscode-azureappservice#1509), I think we were actually more interested in updating the Updated "file.name"... output line.
If you look here, appendLog accepts a resourceName parameter.  We'd rather use that so that the message includes the app's name when updating the file.
For example:
10:40:21 AM: weininger-02: Updated "hostingstart.html".
The point of this issue is that we include the app name so that the user can see which resource was updated.  The test team just included the Saving "hostingstart.html" message as an example of a message that included the resource name in the notification.

@nturinski Ok, I see now. I have got this working. Do we want to include the resource name in all output lines or just the "Updated {filename}" log?

As discussed offline, let's prefix all of the file download/uploading output with the resource name.


LibWeb: Remove reference counting for CSS::StyleProperties

AK::CopyOnWrite already does reference counting, so there is no need to do it again.

CI says this breaks Text/input/interpolation-longhand-properties.html

Yep, some CSS::StyleProperties still need to be passed by reference, working on it.


Cannot access blob

Unable to retrieve child resources.
Details:
["FetchError:request to https://outputstorage.blob.core.windows.net/?include=metadata&comp=list failed, reason: connect ETIMEDOUT <IP_ADDRESS>:443"]

@fishtoo this looks like Storage Explorer is trying to connect to a proxy and failing. Do you have proxy settings set:

At the app level? You can verify by checking Settings (gear icon on the left hand side) -> Application - > Proxy.
At an OS level via environment variables? To verify this, look for environment variables HTTPS_PROXY or HTTP_PROXY.

funny enough it is not trying to connect to a proxy in my case, but is should. I had to add the proxy to the system:
netsh winhttp set proxy proxy:port
But I also have the same proxy defined in the settings. Why does Blob storage access not follow the application proxy settings?

@michelvankessel what version of storage explorer are you using?
Do you know what all that command does? Is it possible that for the machine you are on, without running that command, the proxy address was blocked? As far as our testing can show, setting the app proxy settings will result in storage explorer using that proxy address.

@michelvankessel we just read your comment on another issue. Can you try accessing queues without having run the netsh command?

For some reason it didn't work immediately. the steps I took

open storage explorer (first time)
add Azure AD account from dialog, account is added but cannot list subscriptions,
message": "connect ETIMEDOUT <IP_ADDRESS>:443", which makes sense, because I don't have a proxy set.
Set proxy and apply settings. This will show my subscriptions
At this point I can scroll most storage objects, but as soon a I reach Blob Containers I get the "unable to retrieve child objects"
reason: connect ETIMEDOUT <IP_ADDRESS>:443" < it seems the proxy is not used for Blob Containers and Files Shares. Queues and tables don't seem to generate errors
At this point I have to restart/reload the application to get the new proxy settings going

Funny enough, disabling the proxy settings does have an immediate effect on browsing storage objects.
the specific order that generates the issue, is adding AAD account without proxy and setting the proxy later.
note: the netsh settings was not the solution, but restarting/reloading the app was.

@michelvankessel

the netsh settings was not the solution, but restarting/reloading the app was.

This makes sense since our current proxy setting system don't propagate proxy settings to all process and require a full restart. Storage Explorer will be able to propagate proxy settings without restarting since 1.17.0.

We did some rework on our proxy features in 1.20 and 1.21, which includes dynamically propagating updated proxy settings in Storage Explorer so that a full restart is no longer needed. Please try upgrade to 1.21.


[OpenAI] Update the min dep version of @azure-rest/core-client

Packages impacted by this PR
@azure/openai
Issues associated with this PR
https://github.com/Azure/azure-sdk-for-js/issues/27006
Describe the problem that is addressed by this PR
OperationOptions was exported in<EMAIL_ADDRESS>so we need to use this minimum version in openai.
What are the possible designs available to address the problem? If there are more than one possible design, why was the one in this PR chosen?
N/A
Are there test cases added in this PR? (If not, why?)
N/A
Provide a list of related PRs (if any)
Command used to generate this PR:**(Applicable only to SDK release request PRs)
Checklists

[x] Added impacted package name to the issue description
[ ] Does this PR needs any fixes in the SDK Generator?** (If so, create an Issue in the Autorest/typescript repository and link it here)
[ ] Added a changelog (if necessary)

API change check
API changes are not detected in this pull request.


proposal: x/lint: type omission lint fires even when type is necessary for documentation purposes

I wasn't sure whether to file this as a bug or a proposal, but I would like to remove or disable-by-default the type omission lint. It is the most common false positive I have, and fires often even when the type is necessary for the identifier to show up in the correct place in the documentation.
For example, in mellium.im/xmpp/compress this lint fires for the following:
var (
    // LZW implements stream compression using the Lempel-Ziv-Welch (DCLZ)
    // compressed data format.
    LZW Method = lzwMethod
)

However, without the type, the list of supported compression methods does not show up under the Method type. Instead it shows up at the top of the package level docs and it is no longer obvious that these are existing methods that can be passed to New. In such a small package it doesn't really matter, but in larger packages it can be absolutely necessary to group values under their type.
This isn't technically a problem with Go code (which is what the linter is made to find), but I do think the lint is often encouraging bad practice and unreadable docs, which seems like a good enough reason to get rid of it since it doesn't really provide any benefit that I can see but does have this down side.
Moved from golang/go#26274

Ping.

cc: @andybons

We refined the scope of lint to be issues outlined in golang.org/wiki/CodeReviewComments
If there isn't anything there covering this particular use case (I haven't looked in a while) I don't see an issue with removing it.

See https://golang.org/cl/191317 which didn't get linked here for some reason.


[BUG]: Adding Models to Projects also adds these to Data

Describe the bug
When I add a model to a project, it shows up in both models and data
To Reproduce
Add a model to the project
Go to the project
This is listed in both the models and the data tabs
Expected behavior
Should be listed in the models only
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.

I cannot reproduce this - I think this might be the most recently used items.


[HUDI-2338] Hoodie update reject clustering using SparkRejectClusteringStrategy

https://issues.apache.org/jira/browse/HUDI-2338
What is the purpose of the pull request
Hudi now support async clustering in HoodieDeltaStreamer and StructStreaming and support offline clustering through HoodieClusteringJob.
Data update conflicts with clustering is one of the more common scenarios. And now hudi can only reject data using SparkRejectUpdateStrategy and failed the ingestion.
Sometimes, we think that clustering is an optimization service that runs in the background, and data ingestion has a higher priority than it.
So this tickets add a new UpdateStrategy named SparkRejectClusteringStrategy.
This SparkRejectClusteringStrategy will reject and failing clustering job and let data update success.
Brief change log
When update happened after clustering plan created and before clustering executed.
 1. There will be a request replace commit.

 2. SparkRejectClusteringStrategy will create a clustering reject file under .tmp dir named xxx.replacement.request.reject.

 3. Before perform clustering job, hudi can check this reject file using SparkRejectClusteringStrategy.validateClustering() function.

      3.1 if reject file is exists then abort this clustering plan and remove reject file.

When update happened after clustering executed but not finished.
 1. There will be a inflight replace commit.

 2. SparkRejectClusteringStrategy will create a clustering reject file under .tmp dir named xxx.replacement.inflight.reject.

 3. Before clustering job finished and committed, hudi can check this reject file using SparkRejectClusteringStrategy.validateClustering() function.

      3.1 if reject file is exists then failed this clustering execution and remove reject file.

This PR is already tested on our env and added UTs which are necessary.
As you can see, the update action in<PHONE_NUMBER>3028.commit can success and create a clustering reject file under .hoodie/.tmp dir
Also the clustering job in<PHONE_NUMBER>2927.replacecommit.requested if failed because of SparkRejectClusteringStrategy

Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end.
Added HoodieClientWriteTest to verify the change.
Manually verified the change by running a job locally.

Committer checklist

[ ] Has a corresponding JIRA in PR title & commit

[ ] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

@hudi-bot run azure

@satishkotha if you are up for it, this is a good PR to get your feedback on

@zhangyue19921010 Would this solve the problem completely. IIUC, I think there are still possible race conditions between when replacecommit.rejected file is created and when clustering validation check is performed.
Anyway, my high level thought is that it is better to find a way to integrate this with Multi-writer conflict resolution mentioned here https://cwiki.apache.org/confluence/display/HUDI/RFC+-+22+%3A+Snapshot+Isolation+using+Optimistic+Concurrency+Control+for+multi-writers#RFC22:SnapshotIsolationusingOptimisticConcurrencyControlformultiwriters-OptimisticConcurrencyusingatomicrenameswithConflictResolution
If we integrate with multi-writer, this can be done in a generic for any two operations (instead of adding very specific strategy for clustering.) Some changes may be needed in multi-writer implementation to enforce priority of operations (clustering is lower priority than ingestion). But this is better long term IMO.  Happy to discuss more details/other alternatives.

@satishkotha @vinothchandar Thanks for your attention. Let me explain in more detail :
First of all this rejectClustering strategy will do clustering validation three times :

Before clustering performed, checking xxx.replacement.request.reject.
After clustering finished, checking xxx.replacement.inflight.reject.
Before clustering committing which will create clustering commit file

There's actually a race condition that reject file could be created during clustering committing. This step will create a clustering commit file and I believe it's a quick move.
      table.getActiveTimeline().transitionReplaceInflightToComplete(replaceCommitInflightInstant,
          Option.of(metadata.toJsonString().getBytes(StandardCharsets.UTF_8)));

Also Thanks for @satishkotha reminding, I add another validation after clustering commit created, if reject file created during clustering committing, and there's a reject file remainder after clustering committed. We has to throws an exception here to avoid losing update data.
In short this reject clustering strategy can reject request clustering, inflight clustering and finished but not committed clustering. Although when clustering is committing, we can't reject it anymore and have to throw exception.

Anyway, my high level thought is that it is better to find a way to integrate this with Multi-writer conflict resolution mentioned here https://cwiki.apache.org/confluence/display/HUDI/RFC+-+22+%3A+Snapshot+Isolation+using+Optimistic+Concurrency+Control+for+multi-writers#RFC22:SnapshotIsolationusingOptimisticConcurrencyControlformultiwriters-OptimisticConcurrencyusingatomicrenameswithConflictResolution
If we integrate with multi-writer, this can be done in a generic for any two operations (instead of adding very specific strategy for clustering.) Some changes may be needed in multi-writer implementation to enforce priority of operations (clustering is lower priority than ingestion). But this is better long term IMO. Happy to discuss more details/other alternatives.

I strongly agree that multi-write can solve this conflict more completely. But the disadvantage of this solution is that Introduced external dependencies like ZooKeeper. As the data lake becomes larger, ZooKeeper or something else maybe become the bottleneck which needs to maintain and tune.
So that maybe just introduce a new strategy is more convenient to use.
Anyway, each solution has its own advantages and disadvantages :)

Happy to discuss more details/other alternatives. Thanks in advanced.


Render classes have bad javadoc

This is a minor problem (especially since I am excluding them from javadoc generation anyway).
But the Render classes have methods like this (and others with problems):
    /**
     * Renders the passed in model.
     * @param model a model assumed never to be null.
     * @param appendable the appendable to write to.
     * @throws IOException if there is an error writing to the appendable
     */
    public void execute(dk.mada.jaxrs.generator.mpclient.dto.tmpl.CtxInterface model, Appendable a) throws java.io.IOException {
        execute(model, a, this.formatter, this.escaper);
    }

And the javadoc compiler is not happy:
/home/jskov/git/openapi-jaxrs-client/modules/generator/build/generated/sources/annotationProcessor/java/main/dk/mada/jaxrs/generator/mpclient/dto/tmpl/CtxInterfaceRenderer.java:89: error: @param name not found
     * @param appendable the appendable to write to.
              ^
/home/jskov/git/openapi-jaxrs-client/modules/generator/build/generated/sources/annotationProcessor/java/main/dk/mada/jaxrs/generator/mpclient/dto/tmpl/CtxInterfaceRenderer.java:90: error: reference not found
     * @throws IOException if there is an error writing to the appendable
       ^
/home/jskov/git/openapi-jaxrs-client/modules/generator/build/generated/sources/annotationProcessor/java/main/dk/mada/jaxrs/generator/mpclient/dto/tmpl/CtxInterfaceRenderer.java:92: warning: no @param for a
    public void execute(dk.mada.jaxrs.generator.mpclient.dto.tmpl.CtxInterface model, Appendable a) throws java.io.IOException {
                ^
/home/jskov/git/openapi-jaxrs-client/modules/generator/build/generated/sources/annotationProcessor/java/main/dk/mada/jaxrs/generator/mpclient/dto/tmpl/CtxInterfaceRenderer.java:92: warning: no @throws for java.io.IOException
    public void execute(dk.mada.jaxrs.generator.mpclient.dto.tmpl.CtxInterface model, Appendable a) throws java.io.IOException {

Yep the zero dep has incorrect javadoc. That is a bug.
I need to make some module like like the spring ones that Java javadoc run on them.
(that is why regular mode generates correct javadoc).


Unexpected crash in 3.2b

The daemon was very stable for about a week, but it finally crashed with the following crash report:
2020-06-15 16:00:55 UTC [Info] failed to reset stream (this means it was probably closed successfully): $error
	error: "RPC #709468 failed: \"internal RPC error error: unknown stream_idx\""
2020-06-15 16:01:59 UTC [Info] failed to reset stream (this means it was probably closed successfully): $error
	error: "RPC #709687 failed: \"internal RPC error error: unknown stream_idx\""
2020-06-15 16:11:01 UTC [Fatal] Unhandled top-level exception: $exn
Generating crash report
	exn: "(monitor.ml.Error\n  (\"writer buffer has data older than\" (maximum_age 2m)\n    (beginning_of_buffer\n       \"{\\\"seqno\\\":711124,\\\"method\\\":\\\"validationComplete\\\",\\\"body\\\":{\\\"seqno\\\":584095,\\\"is_valid\\\":false}}\\\n      \\n{\\\"seqno\\\":711125,\\\"method\\\":\\\"validationComplete\\\",\\\"body\\\":{\\\"seqno\\\":584096,\\\"is_valid\\\":false}}\\\n      \\n{\\\"seqno\\\":711126,\\\"method\\\":\\\"validationComplete\\\",\\\"body\\\":{\\\"seqno\\\":584097,\\\"is_valid\\\":false}}\\\n      \\n{\\\"seqno\\\":711127,\\\"method\\\":\\\"validationComplete\\\",\\\"body\\\":{\\\"seqno\\\":584098,\\\"is_valid\\\":false}}\\\n      \\n{\\\"seqno\\\":711128,\\\"method\\\":\\\"validationComplete\\\",\\\"body\\\":{\\\"seqno\\\":584099,\\\"is_valid\\\":false}}\\\n      \\n{\\\"seqno\\\":711129,\\\"method\\\":\\\"validationComplete\\\",\\\"body\\\":{\\\"seqno\\\":584100,\\\"is_valid\\\":false}}\\\n      \\n{\\\"seqno\\\":711130,\\\"method\\\":\\\"validationComplete\\\",\\\"body\\\":{\\\"seqno\\\":584101,\\\"is_valid\\\":false}}\\\n      \\n{\\\"seqno\\\":711131,\\\"method\\\":\\\"validationComplete\\\",\\\"body\\\":{\\\"seqno\\\":584102,\\\"is_valid\\\":false}}\\\n      \\n{\\\"seqno\\\":711132,\\\"method\\\":\\\"validationComplete\\\",\\\"body\\\":{\\\"seqno\\\":584103,\\\"is_valid\\\":false}}\\\n      \\n{\\\"seqno\\\":711133,\\\"method\\\":\\\"validationComplete\\\",\\\"body\\\":{\\\"seqno\\\":584104,\\\"is_valid\\\":false}}\\\n      \\n{\\\"seqno\\\":711134,\\\"method\\\":\\\"validationComplete\\\",\\\"body\\\":{\\\"seqno\\\":584105,\\\"is_valid\\\":false}}\\\n      \\n{\\\"seqno\\\":711135,\\\"method\\\":\\\"validationComplete\\\",\\\"body\\\":{\\\"s\")\n    (writer\n      ((id 14)\n        (fd\n          ((file_descr 55)\n            (info\n              (\"child process\"\n                (stdin (pid 28544) (prog coda-libp2p_helper) (args ()))\n                src/process.ml:62:17))\n            (kind Fifo) (supports_nonblock true) (have_set_nonblock true)\n            (state (Open Empty))\n            (watching ((read Not_watching) (write (Watch_once Empty))))\n            (watching_has_changed false) (num_active_syscalls 1)\n            (close_finished Empty)))\n        (monitor\n          (((name (id 248)) (here ()) (id 248) (has_seen_error false)\n             (is_detached false))\n            ((name \"\") (here ()) (id 228) (has_seen_error false)\n              (is_detached true))))\n        (inner_monitor\n          (((name (id 249)) (here ()) (id 249) (has_seen_error false)\n             (is_detached true))\n            ((name \"\") (here ()) (id 228) (has_seen_error false)\n              (is_detached true))))\n        (background_writer_state Running) (background_writer_stopped Empty)\n        (syscall Per_cycle) (bytes_received 672416539)\n        (bytes_written 672316817) (scheduled_bytes 264) (scheduled_back 264)\n        (back 99722) (close_state Open) (close_finished Empty)\n        (close_started Empty) (num_producers_to_flush_at_close 0)\n        (flush_at_shutdown_elt (<opaque>))\n        (check_buffer_age\n          (((writer <opaque>) (maximum_age 2m)\n             (bytes_received_at_now_minus_maximum_age 672317521)\n             (bytes_received_queue\n               (672317961 672318577 672318841 672318929 672319369 672319897\n                 672321683 672322827 672333189 672344003 672354085 672354525\n                 672355229 672355669 672356109 672358159 672358775 672359655\n                 672360095 672370817 672373395 672373747 672374451 672374715\n                 672375331 672376035 672378085 672379405 672379933 672380813\n                 672380989 672381253 672382863 672384007 672394277 672395157\n                 672395509 672396565 672406631 672407247 672410001 672410089\n                 672410617 672411145 672411761 672412201 672415131 672415219\n                 672416011 672416451 672416539))\n             (times_received_queue\n               ((2020-06-15 09:09:02.222426000-07:00)\n                 (2020-06-15 09:09:06.043597000-07:00)\n                 (2020-06-15 09:09:08.047991000-07:00)\n                 (2020-06-15 09:09:09.048483000-07:00)\n                 (2020-06-15 09:09:12.057494000-07:00)\n                 (2020-06-15 09:09:16.079104000-07:00)\n                 (2020-06-15 09:09:17.081425000-07:00)\n                 (2020-06-15 09:09:20.092527000-07:00)\n                 (2020-06-15 09:09:25.831544000-07:00)\n                 (2020-06-15 09:09:26.837571000-07:00)\n                 (2020-06-15 09:09:27.844615000-07:00)\n                 (2020-06-15 09:09:28.854799000-07:00)\n                 (2020-06-15 09:09:29.856786000-07:00)\n                 (2020-06-15 09:09:30.875843000-07:00)\n                 (2020-06-15 09:09:31.876259000-07:00)\n                 (2020-06-15 09:09:32.876660000-07:00)\n                 (2020-06-15 09:09:34.881631000-07:00)\n                 (2020-06-15 09:09:38.899038000-07:00)\n                 (2020-06-15 09:09:42.904821000-07:00)\n                 (2020-06-15 09:09:45.990195000-07:00)\n                 (2020-06-15 09:09:47.000096000-07:00)\n                 (2020-06-15 09:09:50.507250000-07:00)\n                 (2020-06-15 09:09:54.517261000-07:00)\n                 (2020-06-15 09:09:57.557865000-07:00)\n                 (2020-06-15 09:09:58.558850000-07:00)\n                 (2020-06-15 09:10:01.626516000-07:00)\n                 (2020-06-15 09:10:04.642490000-07:00)\n                 (2020-06-15 09:10:05.757633000-07:00)\n                 (2020-06-15 09:10:09.515714000-07:00)\n                 (2020-06-15 09:10:13.532847000-07:00)\n                 (2020-06-15 09:10:16.576910000-07:00)\n                 (2020-06-15 09:10:17.579646000-07:00)\n                 (2020-06-15 09:10:19.645320000-07:00)\n                 (2020-06-15 09:10:20.650648000-07:00)\n                 (2020-06-15 09:10:23.827351000-07:00)\n                 (2020-06-15 09:10:24.831569000-07:00)\n                 (2020-06-15 09:10:30.841109000-07:00)\n                 (2020-06-15 09:10:31.848250000-07:00)\n                 (2020-06-15 09:10:33.860126000-07:00)\n                 (2020-06-15 09:10:34.865542000-07:00)\n                 (2020-06-15 09:10:35.870771000-07:00)\n                 (2020-06-15 09:10:36.873692000-07:00)\n                 (2020-06-15 09:10:39.900398000-07:00)\n                 (2020-06-15 09:10:40.902828000-07:00)\n                 (2020-06-15 09:10:43.915067000-07:00)\n                 (2020-06-15 09:10:47.939906000-07:00)\n                 (2020-06-15 09:10:52.170006000-07:00)\n                 (2020-06-15 09:10:55.191436000-07:00)\n                 (2020-06-15 09:10:56.196771000-07:00)\n                 (2020-06-15 09:10:59.224958000-07:00)\n                 (2020-06-15 09:11:00.228352000-07:00)))\n             (bytes_seen 672416539) (too_old (Full ())))))\n        (consumer_left Empty) (raise_when_consumer_leaves true)\n        (open_flags (Full (Ok (wronly)))) (line_ending Unix))))\n  (\"Caught by monitor (id 248)\"))"
2020-06-15 16:11:44 UTC [Info] failed to reset stream (this means it was probably closed successfully): $error
	error: "RPC #711025 failed: \"internal RPC error error: unknown stream_idx\""

Restarting produced this crash:
2020-06-16 16:16:40 UTC [Info] Setting current protocol version to "0.1.0" from compile-time config
2020-06-16 16:16:40 UTC [Info] Daemon started process of kind "Prover" with pid 31715
2020-06-16 16:16:41 UTC [Info] Daemon started process of kind "Verifier" with pid 31734
2020-06-16 16:16:41 UTC [Info] Registering async shutdown handler: "Close transition frontier, if exists"
2020-06-16 16:16:41 UTC [Info] libp2p peer ID this session is $peer_id
	peer_id: "12D3KooWDV2Ry7tUpG3vMHgb5hpFQzB2TcSttM6eWgmMHRoDyztx"
2020-06-16 16:16:42 UTC [Fatal] Unhandled top-level exception: $exn
Generating crash report
	exn: "(monitor.ml.Error\n  (Failure\n    \"Failed to connect to libp2p_helper process: RPC #2 failed: \\\"initializing helper error: failed to read value pointer from vlog file: {Fid:42 Len:581 Offset:16049153}: Unable to find log file. Please retry\\\"\")\n  (\"Raised at file \\\"stdlib.ml\\\", line 33, characters 17-33\"\n    \"Called from file \\\"src/deferred1.ml\\\", line 17, characters 40-45\"\n    \"Called from file \\\"src/job_queue.ml\\\" (inlined), line 131, characters 2-5\"\n    \"Called from file \\\"src/job_queue.ml\\\", line 171, characters 6-47\"\n    \"Caught by monitor coda\"))"

It seems like the first crash report file got erased when the new ones were created. Sorry I don't have the first crash report file, but here is another one.
coda_crash_report_2020-06-16_16-18-31.949259.tar.gz

The daemon was very stable for about a week, but it finally crashed with the following crash report:
2020-06-15 16:00:55 UTC [Info] failed to reset stream (this means it was probably closed successfully): $error
	error: "RPC #709468 failed: \"internal RPC error error: unknown stream_idx\""
2020-06-15 16:01:59 UTC [Info] failed to reset stream (this means it was probably closed successfully): $error
	error: "RPC #709687 failed: \"internal RPC error error: unknown stream_idx\""
2020-06-15 16:11:01 UTC [Fatal] Unhandled top-level exception: $exn
Generating crash report
	exn: "(monitor.ml.Error\n  (\"writer buffer has data older than\" (maximum_age 2m)\n    (beginning_of_buffer\n       \"{\\\"seqno\\\":711124,\\\"method\\\":\\\"validationComplete\\\",\\\"body\\\":{\\\"seqno\\\":584095,\\\"is_valid\\\":false}}\\\n      \\n{\\\"seqno\\\":711125,\\\"method\\\":\\\"validationComplete\\\",\\\"body\\\":{\\\"seqno\\\":584096,\\\"is_valid\\\":false}}\\\n      \\n{\\\"seqno\\\":711126,\\\"method\\\":\\\"validationComplete\\\",\\\"body\\\":{\\\"seqno\\\":584097,\\\"is_valid\\\":false}}\\\n      \\n{\\\"seqno\\\":711127,\\\"method\\\":\\\"validationComplete\\\",\\\"body\\\":{\\\"seqno\\\":584098,\\\"is_valid\\\":false}}\\\n      \\n{\\\"seqno\\\":711128,\\\"method\\\":\\\"validationComplete\\\",\\\"body\\\":{\\\"seqno\\\":584099,\\\"is_valid\\\":false}}\\\n      \\n{\\\"seqno\\\":711129,\\\"method\\\":\\\"validationComplete\\\",\\\"body\\\":{\\\"seqno\\\":584100,\\\"is_valid\\\":false}}\\\n      \\n{\\\"seqno\\\":711130,\\\"method\\\":\\\"validationComplete\\\",\\\"body\\\":{\\\"seqno\\\":584101,\\\"is_valid\\\":false}}\\\n      \\n{\\\"seqno\\\":711131,\\\"method\\\":\\\"validationComplete\\\",\\\"body\\\":{\\\"seqno\\\":584102,\\\"is_valid\\\":false}}\\\n      \\n{\\\"seqno\\\":711132,\\\"method\\\":\\\"validationComplete\\\",\\\"body\\\":{\\\"seqno\\\":584103,\\\"is_valid\\\":false}}\\\n      \\n{\\\"seqno\\\":711133,\\\"method\\\":\\\"validationComplete\\\",\\\"body\\\":{\\\"seqno\\\":584104,\\\"is_valid\\\":false}}\\\n      \\n{\\\"seqno\\\":711134,\\\"method\\\":\\\"validationComplete\\\",\\\"body\\\":{\\\"seqno\\\":584105,\\\"is_valid\\\":false}}\\\n      \\n{\\\"seqno\\\":711135,\\\"method\\\":\\\"validationComplete\\\",\\\"body\\\":{\\\"s\")\n    (writer\n      ((id 14)\n        (fd\n          ((file_descr 55)\n            (info\n              (\"child process\"\n                (stdin (pid 28544) (prog coda-libp2p_helper) (args ()))\n                src/process.ml:62:17))\n            (kind Fifo) (supports_nonblock true) (have_set_nonblock true)\n            (state (Open Empty))\n            (watching ((read Not_watching) (write (Watch_once Empty))))\n            (watching_has_changed false) (num_active_syscalls 1)\n            (close_finished Empty)))\n        (monitor\n          (((name (id 248)) (here ()) (id 248) (has_seen_error false)\n             (is_detached false))\n            ((name \"\") (here ()) (id 228) (has_seen_error false)\n              (is_detached true))))\n        (inner_monitor\n          (((name (id 249)) (here ()) (id 249) (has_seen_error false)\n             (is_detached true))\n            ((name \"\") (here ()) (id 228) (has_seen_error false)\n              (is_detached true))))\n        (background_writer_state Running) (background_writer_stopped Empty)\n        (syscall Per_cycle) (bytes_received 672416539)\n        (bytes_written 672316817) (scheduled_bytes 264) (scheduled_back 264)\n        (back 99722) (close_state Open) (close_finished Empty)\n        (close_started Empty) (num_producers_to_flush_at_close 0)\n        (flush_at_shutdown_elt (<opaque>))\n        (check_buffer_age\n          (((writer <opaque>) (maximum_age 2m)\n             (bytes_received_at_now_minus_maximum_age 672317521)\n             (bytes_received_queue\n               (672317961 672318577 672318841 672318929 672319369 672319897\n                 672321683 672322827 672333189 672344003 672354085 672354525\n                 672355229 672355669 672356109 672358159 672358775 672359655\n                 672360095 672370817 672373395 672373747 672374451 672374715\n                 672375331 672376035 672378085 672379405 672379933 672380813\n                 672380989 672381253 672382863 672384007 672394277 672395157\n                 672395509 672396565 672406631 672407247 672410001 672410089\n                 672410617 672411145 672411761 672412201 672415131 672415219\n                 672416011 672416451 672416539))\n             (times_received_queue\n               ((2020-06-15 09:09:02.222426000-07:00)\n                 (2020-06-15 09:09:06.043597000-07:00)\n                 (2020-06-15 09:09:08.047991000-07:00)\n                 (2020-06-15 09:09:09.048483000-07:00)\n                 (2020-06-15 09:09:12.057494000-07:00)\n                 (2020-06-15 09:09:16.079104000-07:00)\n                 (2020-06-15 09:09:17.081425000-07:00)\n                 (2020-06-15 09:09:20.092527000-07:00)\n                 (2020-06-15 09:09:25.831544000-07:00)\n                 (2020-06-15 09:09:26.837571000-07:00)\n                 (2020-06-15 09:09:27.844615000-07:00)\n                 (2020-06-15 09:09:28.854799000-07:00)\n                 (2020-06-15 09:09:29.856786000-07:00)\n                 (2020-06-15 09:09:30.875843000-07:00)\n                 (2020-06-15 09:09:31.876259000-07:00)\n                 (2020-06-15 09:09:32.876660000-07:00)\n                 (2020-06-15 09:09:34.881631000-07:00)\n                 (2020-06-15 09:09:38.899038000-07:00)\n                 (2020-06-15 09:09:42.904821000-07:00)\n                 (2020-06-15 09:09:45.990195000-07:00)\n                 (2020-06-15 09:09:47.000096000-07:00)\n                 (2020-06-15 09:09:50.507250000-07:00)\n                 (2020-06-15 09:09:54.517261000-07:00)\n                 (2020-06-15 09:09:57.557865000-07:00)\n                 (2020-06-15 09:09:58.558850000-07:00)\n                 (2020-06-15 09:10:01.626516000-07:00)\n                 (2020-06-15 09:10:04.642490000-07:00)\n                 (2020-06-15 09:10:05.757633000-07:00)\n                 (2020-06-15 09:10:09.515714000-07:00)\n                 (2020-06-15 09:10:13.532847000-07:00)\n                 (2020-06-15 09:10:16.576910000-07:00)\n                 (2020-06-15 09:10:17.579646000-07:00)\n                 (2020-06-15 09:10:19.645320000-07:00)\n                 (2020-06-15 09:10:20.650648000-07:00)\n                 (2020-06-15 09:10:23.827351000-07:00)\n                 (2020-06-15 09:10:24.831569000-07:00)\n                 (2020-06-15 09:10:30.841109000-07:00)\n                 (2020-06-15 09:10:31.848250000-07:00)\n                 (2020-06-15 09:10:33.860126000-07:00)\n                 (2020-06-15 09:10:34.865542000-07:00)\n                 (2020-06-15 09:10:35.870771000-07:00)\n                 (2020-06-15 09:10:36.873692000-07:00)\n                 (2020-06-15 09:10:39.900398000-07:00)\n                 (2020-06-15 09:10:40.902828000-07:00)\n                 (2020-06-15 09:10:43.915067000-07:00)\n                 (2020-06-15 09:10:47.939906000-07:00)\n                 (2020-06-15 09:10:52.170006000-07:00)\n                 (2020-06-15 09:10:55.191436000-07:00)\n                 (2020-06-15 09:10:56.196771000-07:00)\n                 (2020-06-15 09:10:59.224958000-07:00)\n                 (2020-06-15 09:11:00.228352000-07:00)))\n             (bytes_seen 672416539) (too_old (Full ())))))\n        (consumer_left Empty) (raise_when_consumer_leaves true)\n        (open_flags (Full (Ok (wronly)))) (line_ending Unix))))\n  (\"Caught by monitor (id 248)\"))"
2020-06-15 16:11:44 UTC [Info] failed to reset stream (this means it was probably closed successfully): $error
	error: "RPC #711025 failed: \"internal RPC error error: unknown stream_idx\""

I had this same crash

Luke is uploading the coda-config associated with not being able to restart. The writer buffer has data older than\" (maximum_age 2m-bug is notorious for being hard to find. Given that it only happens after the node is up for a long time, we won't prioritize fixing it -- but we do want to prioritize fixing the ability restart when this bug happens.
We should open a new issue to track the restart problem

I had the same issue after 6 days on my google cloud vm. Has there been any auto restart recommendation yet that would mitigate this problem? Seems that people are using different scripts to auto restart. Is there any recommended way?

Closing this issue as it is stale for a long time.


feat: xml output omit defaultNS identifier

fixes #438
BREAKING CHANGE

breaks for the following reasons: see https://github.com/CycloneDX/cyclonedx-python-lib/issues/438#issuecomment-1731052115

superseded by #458


Create !crisis, !mentalhealth and !emergency commands

Re-merge of #435 after Nick did some stuff with the tests. Content is identical.
All are aliases for the same thing - provide a list of crisis resources for
quick reference.

Thanks for the reviews everyone, and thanks @nicklambourne for fixing the tox problem! 😄


[PR/623 - pt 2] Use scheduling constraints to find best fit TEs

Context
This PR is the second one of an effort to split a larger PR into manageable parts.
In the preceding PR, we introduced "scheduling constraints," composed of machine and assignment attributes, to guide scheduling decisions.
The focus of this PR is to employ these constraints optimally to identify the best Task Executors.
Task Executors will continue to register in the system, carrying essential metadata for facilitating on-demand scheduling. However, they now also carry "allocation attributes" to support the newly introduced scheduling logic. While initially the registered TEs were indexed solely by core count, we now allow multiple "versions" of TEs for each core count. For instance:

2 cores, 16GB | jdk:8
2 cores, 32GB | jdk:8
2 cores, 16GB | jdk:17

With this system, there might be multiple SKUs available for a given core count. Efficient and safe scheduling requires us to search for a single SKU that fits a request the best. To achieve this, we will change our indexing strategy for Task Executors: instead of purely using cores count, we will use a group key that encapsulates both machine definition and allocation attributes.
The scheduling search process now involves two steps. First, we need to ensure that the allocation attributes specified in the request can be satisfied by the TEs. Second, the selected TEs should be the "best fit" for the request in terms of CPU and memory. The first step is achieved by passing the required allocation attributes (and potential default values) along with the request (via config) to the RC actor. The second step is accomplished through a new interface introduced to calculate the fitness score of two machine definitions -- the one from the request and the one from the registered TE. An implementation of this interface is provided, which computes the fitness score based on a CPU/memory ratio of 18/1.
Important note
To maintain focus on scheduling, this PR intentionally neglects the pending scheduling cache logic, which is currently flawed as it groups by cores, while it should use all scheduling constraints. This will be resolved in a future PR, the third in this series.
Unresolved items, marked with TODO in the PR, have been left deliberately to reduce complexity. An example would be the passage of the fitness function through the config; this and other items will be taken care of in subsequent PRs.
Tests

Added more unit tests
verified mantis-helm works

Checklist

[ ] ./gradlew build compiles code correctly
[ ] Added new tests where applicable
[ ] ./gradlew test passes all tests
[ ] Extended README or added javadocs where applicable

closing in favor of https://github.com/Netflix/mantis/pull/638


stdlib: fix the build with macOS long tests

When building swift on macOS with long tests, the build would fail with
undefined references to these symbols.  The undefined symbols were
expected to be present in swiftCore against which we link.  However,
because the symbols are marked as internal, they would be dead stripped
before the dylib was constructed.  As a result, we would end up with
undefined references to these symbols.  Expose the additional internal
interfaces for now so that we can build the tests.

Replace this paragraph with a description of your changes and rationale. Provide links to external references/discussions if appropriate.

Resolves SR-NNNN.

CC: @jrose-apple
Sorry Jordan, don't really know who is a good person to run this past.

@swift-ci please smoke test

What the heck is @_silgen_name("")? I've never seen that before, and it doesn't seem valid to me.

The right answer here is probably to either use @_versioned internal or @_silgen_name("somethingActuallyValid") internal, both of which preserve Swift-side access control while allowing linking from the runtime .o files.

@jrose-apple well, the interesting thing is that the names are used in the mangled form, so Im not sure that the @_silgen_name is needed at all.  @_versioned internal sounds reasonable, but that won't export it will it?

That does export it, but ideally this wouldn't be exported, just not dead-stripped. Maybe that's what _silgen_name("") is trying to do? But it'd be better to just pick a name and not bother with the mangled form.

@jrose-apple unfortunately, it would need to be exported for the tests to actually use it from the stdlib.

But there aren't any tests using it from the stdlib.

@jrose-apple oh, I thought I mentioned that this was being used by the long tests.

@swift-ci please smoke test

I saw that, but I can't find a test that is actually doing it.

The references are from the runtime:

Undefined symbols for architecture x86_64:
"__T0s13_getErrorCodeSiSPyxGs0B0RzlF", referenced from:
__swift_stdlib_bridgeErrorToNSError in ErrorObject.mm.o
"__T0s23_getErrorDomainNSStringyXlSPyxGs0B0RzlF", referenced from:
__swift_stdlib_bridgeErrorToNSError in ErrorObject.mm.o
"__T0s29_getErrorUserInfoNSDictionaryyXlSgSPyxGs0B0RzlF", referenced from:
__swift_stdlib_bridgeErrorToNSError in ErrorObject.mm.o
"__T0s32_getErrorEmbeddedNSErrorIndirectyXlSgSPyxGs0B0RzlF", referenced from:
dynamicCastValueToNSError(swift::OpaqueValue*, swift::TargetMetadataswift::InProcess const*, swift::WitnessTable const*, swift::DynamicCastFlags) in Casting.cpp.o
_dynamicCastToExistential(swift::OpaqueValue*, swift::OpaqueValue*, swift::TargetMetadataswift::InProcess const*, swift::TargetExistentialTypeMetadataswift::InProcess const*, swift::DynamicCastF

These show up when building:
unittests/runtime/LongTests/SwiftRuntimeLongTests
unittests/runtime/SwiftRuntimeTests

@_silgen_name("") is a temporary hack that has the same effects on symbol visibility as @_silgen_name("whatever") but does not change the symbol's name.

@gparker42 Can we just…not do that? I don't see any reason why the runtime would want to reference these by mangled name rather than non-mangled name.

The runtime tests shouldn't be linking to both the runtime .a and the standard library. We should pick one and stick with it. This has been causing problems elsewhere too (warnings on Darwin about duplicate class implementations).
Can we fix that instead? Or is that hard to fix?

Hah, I just saw that and was about to look into that error too.  I didnt see the .a in the linker invocation, which is why I was under the impression that it was supposed to pull it from the stdlib.  If that is simply me being incapable of reading, then, I think fixing that is better if it isn't too terribly difficult (for some value of terribly difficult).

It links the runtime .a, which references symbols in the stdlib. But it already loads the stdlib, I think. (If not, we should make the .a symbols weak or something so that the runtime tests can still build.)

(It might link a bunch of runtime object files directly. I don't remember.)

Yes, it does link against the runtime, but does not include the stdlib objects.  I suppose that we should be able to avoid the double runtime link.

IIRC the test executable links directly to the runtime's C++ code but not its Swift code. unittests/runtime/Stdlib.cpp contains empty placeholders for some Swift functions that are required to build but not to run. We may need to add these functions to that file.

(And the test executable does this because it calls some non-exported runtime functions directly.)

@swift-ci please smoke test

The linux build is a out of space issue, I think that this should be okay, it also locally passes for me.  @jrose-apple or @gparker42 could we get this merged?

@swift-ci please smoke test Linux platform


[$500] Android - Chat - PDF preview is opening in a smaller version

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: <IP_ADDRESS>
Reproducible in staging?: Y
Reproducible in production?: Y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: Applause - Internal Team
Slack conversation:
Action Performed:

Open a conversation
Upload a PDF file
Close the app
Open the app
Navigate to the chat with the PDF

Expected Result:
PDF preview should load the correct size for the first time
Actual Result:
PDF preview is opening in a smaller version and gets to the correct size later
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android: Native
[ ] Android: mWeb Chrome
[x] iOS: Native
[ ] iOS: mWeb Safari
[ ] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos
Add any screenshot/video evidence

https://github.com/Expensify/App/assets/78819774/596e9f1c-f279-4c33-b9a9-2e55c9428767
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~018e002ff9a3c1a28c
Upwork Job ID:<PHONE_NUMBER>679159808
Last Price Increase: 2024-01-05

Proposal
Please re-state the problem that we are trying to solve in this issue.
The PDF preview shows as small first and then resized to normal
What is the root cause of that problem?
The PDF preview is rendered by ImageRenderer and the width and height are retrieved from the HTML attribute
https://github.com/Expensify/App/blob/f5a3c0202128a8e0e77607defbbdcb4f30f80ad3/src/components/HTMLEngineProvider/HTMLRenderers/ImageRenderer.js#L48-L49
However, both width and height are 0, and if you notice the code above, we fall back to undefined. Because it's undefined, the ThumbnailImage gets the default width and height of 200.
https://github.com/Expensify/App/blob/f5a3c0202128a8e0e77607defbbdcb4f30f80ad3/src/components/ThumbnailImage.tsx#L74
The image component then loads the image and updates it to the correct size.
https://github.com/Expensify/App/blob/f5a3c0202128a8e0e77607defbbdcb4f30f80ad3/src/components/ThumbnailImage.tsx#L87-L95
What changes do you think we should make in order to solve the problem?

Have a cache to cache the image size similar to the Lightbox.js component. Then we use this cached size as the fallback instead of undefined.
https://github.com/Expensify/App/blob/f5a3c0202128a8e0e77607defbbdcb4f30f80ad3/src/components/Lightbox.js#L17
https://github.com/Expensify/App/blob/f5a3c0202128a8e0e77607defbbdcb4f30f80ad3/src/components/Lightbox.js#L70
We currently have the cache implemented for native only on the Image component here.
https://github.com/Expensify/App/blob/f5a3c0202128a8e0e77607defbbdcb4f30f80ad3/src/components/Image/index.native.js#L12-L14
https://github.com/Expensify/App/blob/f5a3c0202128a8e0e77607defbbdcb4f30f80ad3/src/components/Image/index.native.js#L61
I think it's weird to have the cache implementation exposed by a component, so I think we can create a new global file to handle the cache
(the cache only lives on memory, so it will be cleared when the app closed)

Proposal
Please re-state the problem that we are trying to solve in this issue.
Pdf attachment preview after upload is opened in smaller size and later resized to actual size.
What is the root cause of that problem?
The root cause of the issue is that after the pdf is uploaded, in ImageRenderer.js, the imageWidth and imageHeight fallback to undefined because the html attributes 'data-expensify-width' and 'data-expensify-height' are set to 0.
https://github.com/Expensify/App/blob/f5a3c0202128a8e0e77607defbbdcb4f30f80ad3/src/components/HTMLEngineProvider/HTMLRenderers/ImageRenderer.js#L48
The ThumbnailImage prop sets default values for imageWidth and imageHeight to 200.
https://github.com/Expensify/App/blob/f5a3c0202128a8e0e77607defbbdcb4f30f80ad3/src/components/ThumbnailImage.tsx#L74
The ImageWithSizeCalculation component eventually calls GetSize() from react native image api, to return the actual image size.
https://github.com/Expensify/App/blob/f5a3c0202128a8e0e77607defbbdcb4f30f80ad3/src/components/ThumbnailImage.tsx#L102
RNImage.GetSize()
https://github.com/Expensify/App/blob/f5a3c0202128a8e0e77607defbbdcb4f30f80ad3/src/components/Image/index.js#L39
Then after GetSize, the size is recalculated based on the actual size provided by GetSize()
https://github.com/Expensify/App/blob/f5a3c0202128a8e0e77607defbbdcb4f30f80ad3/src/components/ThumbnailImage.tsx#L89
What changes do you think we should make in order to solve the problem?
I tried prefetching and caching approach using react native image api prefetch and querycache. Even after the image is prefetched, react native does not return size immediately using GetSize(). The approach I suggest to use is using persistentstorage (Onyx) for storing image data i.e. imgurl, width and height. Once stored in Onyxdb, the image data can be easily retreived using imgUrl and then passed into ImageRenderer.js and then to ThumbnailImage as imageWidth and imageHeight props.
With the caching approach, getSize would be called each time the user switches reports and the activity indicator would be displayed each time but with the persistent storage the image data is retrieved immediately from Onyxdb and there is no delay in displaying the correct size.
What alternative solutions did you explore? (Optional)

@hoangzinh let me know your thoughts here. This seems like a tiny issue though, so if any of these proposals take a lot of time and effort to complete, I'd prefer we close instead of "fix", because the PDF still loads fine.

@hoangzinh the persistent storage approach I mentioned is simple and effective, some feedback would be much appreciated. Thanks

@twisterdotcom please do take few moments and review the video posted in proposal and comment on current behavior. The proposal is to not have the activity loader displayed each time the user switches reports.

@hoangzinh Thanks for feedback. I have added another poc video to demonstrate proposal from uploading attachment to preview loading.

@hoangzinh @twisterdotcom In the video poc I have covered the flow from user loading the chat first time and adding attachment to the user switching reports. As demonstrated, the issue does not occur the first time chat is loaded or after uploading attachment. In the video attached by tienifr, the attachment is preloaded and does not demonstrate the entire flow and the issue could reoccur the first time user load chat or after uploading attachment.

@hoangzinh I am curious how @tienifr proposal is unique from others.
@tienifr please kindly also share video of the entire flow from user uploading attachment to user switching reports so that becomes clear as how the proposal is unique from others.

I terminated the app completely and reopened the app again.
@tienifr the closing the app and opening the app part is clear but what is not clear is the actual uploading of the attachment and after the attachment is loaded whether the issue remains.
@hoangzinh The issue is also occurring on the web besides other platforms. I have also made the solution clear that is not to show the image if the size is not known but I have provided clear video demonstrating that no resizing takes place after attachment is loaded and the first time chat is loaded.

TBH, all the solutions seem to do what is required from a product perspective as long as @tienifr's also works from upload, which I assume it does.
@hoangzinh please choose the solution you think makes the most technical sense and we can assign an internal engineer to verify as well.

@twisterdotcom @hoangzinh I also propose to split not all but some of the bounty should the review determine both solutions meet product perspective :smile: and I suggest not to leave things on assumptions because that is how regressions happen, uploading another video would take maybe 5 minutes.

I don't think it's over-engineered. We did the same approach for Lightbox.js which is a pretty simple approach
https://github.com/Expensify/App/blob/964548a2adb3ed9e2572ad33456759176a01b76c/src/components/Lightbox.js#L17
https://github.com/Expensify/App/blob/964548a2adb3ed9e2572ad33456759176a01b76c/src/components/Lightbox.js#L70-L74

Yes, that's correct

@hoangzinh The first load would not be an issue, i can provide another video if that would assist.

@hoangzinh Yes, I agree that the first load requirement was supposed to be discussed in proposal and I left that out because I did not want to give out too much before the proposal was awarded. To get the image to load on first load need to add useEffect in ReportActionItemFragment

@hoangzinh Please review revised proposal, I have included part on how to get first load. Also the issue could probably get resolved on the backend if data-expensify-width and data-expensify-height were provided, depends how busy the back-end team is.

Also I think caching width/height of images to Onyx seems over-engineered to me.

@hoangzinh The temp ids for invited members are also being stored in onyx when the user invites members to workspace, storing image meta data in onyx is related to that and would follow similar pattern.

@hoangzinh regarding the specifics of first load, I can share the implementation details after the proposal is awarded. I assure the proposal does handle first load as demonstrated in the poc. Thanks for understanding.

Thanks for the review, @hoangzinh; I'm leaning towards closing this issue as you mentioned here. Currently, I'm focused on fixing wave issues for the backend. If it can be fixed purely on the frontend, it would be great; we can wait a bit longer in case there's a better proposal.

@hoangzinh regarding the specifics of first load, I can share the implementation details after the proposal is awarded. I assure the proposal does handle first load as demonstrated in the poc. Thanks for understanding.

@sourcecodedeveloper given the comment above, feel free to provide more details in your current proposal maybe you can change @hoangzinh's mind :)

@marcochavezf Thanks
@hoangzinh I can try and elaborate in some more detail and perhaps get the proposal.
I would add useEffect in ReportActionItemFragment. The useEffect will have dependency of [fragment.html]
https://github.com/Expensify/App/blob/753ce92ea9a1858dd2d576eeff692df527ea3ecb/src/pages/home/report/ReportActionItemFragment.js#L96

When the user uploads the attachment the first time, the fragment.html is set to ATTACHMENT_UPLOADING_MESSAGE_HTML and the message Uploading attachment... is displayed in the chat. I would add fragment.html as dependency to useEffect and listen for updates to fragment.html.

I would also add useState variable isAttachmentLoading and set the value to true while the attachment is uploaded.

The useEffect would trigger on value change of fragment.html. After the attachment is uploaded, the fragment.html contains the image preview source, in the useEffect I would check the value of fragment.html and obtain the imgUrl and then check if the imgUrl has already been processesed previously and stored in Onyx key store. If the imgUrl has previously been processed, I would set isAttachmentLoading to false and return from useEffect.

If the imgUrl has not been previously processed i.e the lookup in key store would return null, I would then proceed to call RNImage.GetSize(imgUrl) and in the success callback for GetSize store the image width and height in keystore and set isAttachmentLoading to false.

The useState variable isAttachmentLoading would be used for displaying the previous message Uploading attachment... through out the attachment upload + the call to RNImage.GetSize(if needed). Once the isAttachmentLoading is set to false, the image preview is ready to be displayed in first load.

@marcochavezf Thanks
@hoangzinh I can try and elaborate in some more detail and perhaps get the proposal.
I would add useEffect in ReportActionItemFragment. The useEffect will have dependency of [fragment.html]
https://github.com/Expensify/App/blob/753ce92ea9a1858dd2d576eeff692df527ea3ecb/src/pages/home/report/ReportActionItemFragment.js#L96

When the user uploads the attachment the first time, the fragment.html is set to ATTACHMENT_UPLOADING_MESSAGE_HTML and the message Uploading attachment... is displayed in the chat. I would add fragment.html as dependency to useEffect and listen for updates to fragment.html.

I would also add useState variable isAttachmentLoading and set the value to true while the attachment is uploaded.

The useEffect would trigger on value change of fragment.html. After the attachment is uploaded, the fragment.html contains the image preview source, in the useEffect I would check the value of fragment.html and obtain the imgUrl and then check if the imgUrl has already been processesed previously and stored in Onyx key store. If the imgUrl has previously been processed, I would set isAttachmentLoading to false and return from useEffect.

If the imgUrl has not been previously processed i.e the lookup in key store would return null, I would then proceed to call RNImage.GetSize(imgUrl) and in the success callback for GetSize store the image width and height in keystore and set isAttachmentLoading to false.

The useState variable isAttachmentLoading would be used for displaying the previous message Uploading attachment... through out the attachment upload + the call to RNImage.GetSize(if needed). Once the isAttachmentLoading is set to false, the image preview is ready to be displayed in first load.

when user sign out and sign in again, then go to any chat that has images. What would you show in the chat report when isAttachmentLoading is true?

@hoangzinh Interesting point. After the user signs out, the onyx storage is cleared and signing back in would mean the image size is no longer stored in onyx and needs to be reloaded.
While isAttachmentLoading is true, I would need to show either attachment loading placeholder text or some success activity indicator.
Option 1:
The existing implementation when uploading attachment uses placeholder const when attachment is uploading,
ATTACHMENT_UPLOADING_MESSAGE_HTML: 'Uploading attachment...',
poc_loader1.webm
Option 1 would be to add another placeholder const for reloading sizes i.e ATTACHMENT_LOADING_MESSAGE_HTML: 'Loading Attachment...'
Option 2:
Instead of using placeholder message, the success activity loader can be used. Attached poc for activity loader,
poc_loader2.webm

@tienifr's https://github.com/Expensify/App/issues/34041#issuecomment-1879078487 provided a simple fix, but if we show a large image or network is slow, the image is disappear for a short period. I think it's worse than currently (temporary show in a smaller version then show in a correct version).

@hoangzinh I would not be showing large image and their proposal does not cover first load and also does not use any kind of cache or onyx, they would need to show activity loader even after file is uploaded and each time user switch report call getSize. Instead of showing image, I would be displaying either Option 1 or Option 2.
What I can suggest for the proposal, something to benefit both the FE & BE. The ImageRenderer already has 'data-expensify-width' and 'data-expensify-height' but the BE is not sending the correct values. In the ImageRenderer I can add check to only load image width and height from onyx if the 'data-expensify-width' and 'data-expensify-height' are undefined. Then depending on when and if the BE starts sending correct width/height, the priority would be given to the values from BE.
And the other thing regarding BE, after sign out/signin, the BE would also need to fetch image sizes and send to FE by api call, the report on FE would need to show some form of activity loader from Option 1 or Option 2, while the image size is retrieved from BE. Also once the BE has returned the correct width/height, storing the values in onyx storage would be beneficial for immediate use when switching reports. Otherwise, the BE would have to return image width/height each time the user goes from one chat to another. So even if the BE returns correct width and height, storing the image width/height from the BE in onyx storage would be beneficial for displaying images when the user goes from one chat to another.

I'm focused on fixing wave issues for the backend.

On the other hand, does not look like @marcochavezf would be available any time soon which means it would go to waste and issue would get closed.

@hoangzinh how about considering the proposal for if the BE implements correct width and height, then retrieving width and height from onyx would still provide faster loading when the user goes from one chat to another, otherwise the BE would have to fetch image with and height each time user switches reports.

@marcochavezf intrigued for your thoughts here? I'm worried we might be wasting a lot of time on something trivial and not super noticeable for the end user.
I'm also not sure we care that much about having the same behavior on first login vs every other time you can recover some onyx cache. The app has infinite sessions, so I kind of think if we have to load it once incorrectly for it to load correctly every other time (unless you sign out), that's an acceptable solution.

Not overdue.


init fails with timeout on db with 10,000+ tables

When I try to init in a large database (10.600 tables); fails with timeout
[mysql] 2018/09/13 09:44:47 packets.go:36: read tcp <IP_ADDRESS>:52735->XXX.XXX.XXX.XXX:3306: i/o timeout
2018-09-13 09:44:47 [ERROR] Cannot examine schemas on XXX.XXX.XXX.XXX:3306: Error querying information_schema.tables: invalid connection
2018-09-13 09:44:47 [DEBUG] Exit code 2

Thank you for trying out Skeema! The client-side read timeout (e.g. max time Skeema will wait to get a network response) defaults to 5 seconds, but is configurable. However, there may be other problems encountered when trying to use Skeema on a database with such a high table count. A few things to try or check:

You can configure a higher network read timeout using connect-options. For example, try skeema init --connect-options="readTimeout=30s" to use a 30-second timeout. Once you find a good value, you can permanently configure this by putting a line like connect-options="readTimeout=30s" in a global config file (/etc/skeema, /usr/local/etc/skeema, ~/.skeema) or in the instance-level config file created by skeema init.

With 10000 tables, skeema may be making upwards of 50000 database queries. This means it is essential to run Skeema from the same datacenter as mysql, to reduce network latency. For example, if your database instance is in AWS, be sure to run Skeema from a VM in the same availability zone as your db, rather than running it from your local machine.

Given the extremely high table count, are many of these tables automatically generated/managed by your application? If so, it might make sense to exclude those tables from being managed by Skeema, using the ignore-table regex option.

In general, schema management tools need to query table metadata for all tables, which can be slow when you have a huge number of tables. On the database side, there are a few global settings that can help here.

Ensure that mysql's innodb_stats_on_metadata global option is disabled. If this is enabled, it would severely hurt Skeema's performance if there's a large table count. Luckily it's only enabled by default in 5.5.

Have you optimized your database settings for your high table count in general? Some things to possibly configure include table_definition_cache, innodb_open_files, and possibly several options relating to persistent stats -- but it depends a lot on your database version/flavor, workload, and environment.

Closing due to inactivity


Port AvahiDBus provider to DBus#

This branch ports Mono.Zeroconf.Providers.AvahiDBus from NDesk.DBus to DBus#.

Actually, cancel this. Judging by https://github.com/mono/Mono.Zeroconf/commit/3e2358e67a398b65fe1385446d889c75f5525392, removing TrapSignals and UntrapSignals isn't such a good idea.


Fix compilation of fallback.rs

The fallback.rs backend has several small compilation issues; this PR addresses them.

Oops, thanks!


how to use piou loss in Mask RCNN-based methods?

Hello, I want to replace Piou loss with Smoothl1 loss in the Mask rcnn-based instance segmentation method. I want to confirm whether I only need to replace Smoothl1 loss in RPN and Roi head with Piou loss? In addition, batch['cat_spec_mask'] in ctdet_angle.py is the segmentation label of the instance? Looking forward for your response, thank you.

Hello,and I want to know if you have solved this problem replaceing  Piou loss loss  with Smoothl1 loss  in the Mask rcnn-based instance segmentation method??


feat(consensus): improve logging and metrics for untimely messages

Logging
Addresses  #2320.
For instance, the new logging of timely/untimely messages is:
$ go test -v -run TestPBTSEnableHeight | grep timely
I[2024-02-27|10:50:38.046] prevote step: Proposal is not timely; prevoting nil module=consensus height=4 round=0 proposal_timestamp_difference=-999.792ms lower_bound=-500ms upper_bound=2.5s
D[2024-02-27|10:50:38.974] prevote step: Proposal is timely             module=consensus height=4 round=1 proposal_timestamp_difference=200µs lower_bound=-500ms upper_bound=2.5s
D[2024-02-27|10:50:38.989] prevote step: Proposal is timely             module=consensus height=5 round=0 proposal_timestamp_difference=106µs lower_bound=-500ms upper_bound=2.5s

Notice that untimely messages are logging in INFO level, while timely messages are logging in DEBUG level.
Metrics
Addresses  #2323.
Metrics calculation refactored. Now they are computed when moving to the prevote step, instead of when receiving a (valid) proposal.

PR checklist

[ ] Tests written/updated
[ ] Changelog entry added in .changelog (we use unclog to manage our changelog)
[ ] Updated relevant documentation (docs/ or spec/) and code comments
[ ] Title follows the Conventional Commits spec

Is this ready for review?

Addresses  #2323

Can we say it partially addresses it? We used that issue to describe our QA test cases with @hvanz, so I wouldn't like to close #2323 until those tests are run.
Or, should we move the test cases to another issue?


test: allow the v1.x nightly to be manually triggered

In order to allow testing the nightly on v1.x without having to wait a full day
PR checklist

[X] Tests written/updated
[ ] Changelog entry added in .changelog (we use unclog to manage our changelog)
[ ] Updated relevant documentation (docs/ or spec/) and code comments
[ ] Title follows the Conventional Commits spec

No need to change this workflow. We can use the e2e-manual workflow that performs the same tests.


Allow configuring location of overlayfs

What is the problem you're trying to solve
We have encountered issues with containers that end up doing a lot of disk i/o on the overlayfs (instead of using a proper mount via a k8s emptyDir etc) and end up starving containerd leading to timeouts on various CRI operations. For things like k8s emptyDirs, we have been able to isolate containerd by keeping the containerd root location on a different drive than the kubelet but since the overlayfs directory is hardcoded to be under the containerd root, we are unable to do so in that case.
Describe the solution you'd like
Allowing the location of the overlayfs to be configured when configuring the overlayfs plugin would help provide the flexibility to do so if the operator wants to.
Additional context

Related to https://github.com/containerd/containerd/issues/4485 which is where we saw the effects of this problem

Two workarounds:

Substitute root = "/var/lib/containerd" in /etc/containerd/config.toml to an arbitrary path
Bind-mount an arbitrary directory on /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs

@AkihiroSuda thanks for suggesting the workarounds.

We found that doing this doesn't really fix the core problem here which is that we would like to isolate containerd's disk i/o from disk i/o being driven by workloads inside a container that can compete for resources.
This is precisely what we are currently doing as a workaround to this problem :)

The reason for this feature request was because we think Option 2 forces us to rely on containerd implementation details like the name of the directory io.containerd.snapshotter.v1.overlayfs which we are hoping to avoid.

We would be more than happy to contribute a patch if this configurability sounds reasonable

:+1:


Typescript SDK: How to get workflow Id while creating via putWorkflow

I'm creating workflow via admin.putWorkflow but it does not return anything.
So, in this case, how I can get a workflow Id in case I have to delete it later?
I couldn't find anything in the sdk or docs.
SDK: Typescript
Thanks.

Hi @ahsansher, unfortunately that SDK method does not return anything at the moment. The best option is to use the REST client to list workflows and get the workflow by name from there. We can add support for returning the workflow ID directly - it should be a very easy change if someone would like to pick it up: https://github.com/hatchet-dev/hatchet-typescript/blob/46bb07239d054761b67ab4cd1572383c71774bee/src/clients/admin/admin-client.ts#L81.

Okay @abelanger5. I'll have a look.
Thanks.

@abelanger5 I have created a fix for this in the typescript-sdk repo.
I've cloned the repo, and implemented fix on a new branch, now trying to push that branch but getting remote: Permission to hatchet-dev/hatchet-typescript.git denied.
can you please share how exactly I can push my branch and open a PR?
Thanks.

Hi @ahsansher, you'll need to push to a fork of the repo, and then you can create a PR from the branch in your fork to the branch in the main repo.
More docs here: https://docs.github.com/en/get-started/exploring-projects-on-github/contributing-to-a-project

@abelanger5 I have created a PR for this fix here: https://github.com/hatchet-dev/hatchet-typescript/pull/346
Please have a look. Thanks.

Closing as https://github.com/hatchet-dev/hatchet-typescript/pull/346 was merged


Self Hosting - Feedback

Two issues I wanted to raise, neither were major obstacles.
First, when you run your any-sync-tools and create the YAML files, the output is called sync_1.yml and your instructions don't make it clear that it should be used for the any-sync-node application.
Second, and more important, it defaults to a storage location of db and if that folder doesn't exist, it throws an error. I'm not sure what the best way to fix this is, either

Fix the any-sync-tools utility so it prompts for a directory and creates it if it doesn't exist
Have the script simply try to create the directory if it doesn't exist.

Hi @ChadDaemon, thank you for the feedback!
We have made an effort to address the correct configuration allocation in step 5 of the how-to.
We will consider how to better handle the issue of a non-existing folder. Thank you for bringing it to our attention!

Haha, I got so wrapped up in following the README in the individual repos I forgot about your how-to article.

@ChadDa3mon if you are interested, I'm working on docker images to simplify self hosting. I got the coordinator image built and running, and I ran into the same issue for the sync node. Also for the file node I've some issues to fix. Feel free to take a look here:
https://github.com/SamBouwer/any-docker

I had the same issue with the sync node here, too.
I would advise against the script (any-sync-network) trying to create folders as you could potentially run any-sync-network on a different machine to just create the config files for deployment, hence creating directories on that machine wouldn't help.
A word of warning in the self-hosting instructions would definitely be helpful.

Am I right to assume that the client apps MUST be compiled ? I can't use the the app from the App Store ?

Am I right to assume that the client apps MUST be compiled ? I can't use the the app from the App Store ?

Correct, the network configuration settings are embedded/hardcoded into the app when compiling, so you could compile your own Anytype client, but it will connect to the Anytype hosted nodes.

@olluz Just to add here: we are already working on a network configuration inside the app. I hope we will deliver this in the September release.


implementing hidden agent

Description
[When an conversational agent is registered with a hidden model, then the agent will be treated as hidden too. We don't want to expose any information for the hidden model.]
Issues Resolved
[List any issues this PR will resolve]
Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Are you building a hidden agent or removing the hidden model from Get/Search agent response?

Are you building a hidden agent or removing the hidden model from Get/Search agent response?

hidden agent.

CI failed

Task :opensearch-ml-plugin:jacocoTestCoverageVerification FAILED
[ant:jacocoReport] Rule violated for class org.opensearch.ml.action.agents.TransportSearchAgentAction: branches covered ratio is 0.5, but expected minimum is 0.7

CI failed

Task :opensearch-ml-plugin:jacocoTestCoverageVerification FAILED
[ant:jacocoReport] Rule violated for class org.opensearch.ml.action.agents.TransportSearchAgentAction: branches covered ratio is 0.5, but expected minimum is 0.7

Yeah, I'm working on it.