waiting very long for "x11docker=ready" | Xorg in rootless podman | bubblewrap setups

In Fedora CoreOS (in a virtualbox VM), with latest images of docker.io/x11docker/xserver and docker.io/x11docker/fluxbox, using the 7.4.2 version of x11docker script:
x11docker -D -V --backend=podman --desktop x11docker/fluxbox
loops apparently forever with:
...
DEBUGNOTE[16:27:10,027]: waitforlogentry(): tailstdout: Waiting since 703s for log entry "x11docker=ready" in store.info
DEBUGNOTE[16:27:10,027]: waitforlogentry(): tailstderr: Waiting since 703s for log entry "x11docker=ready" in store.info
...

I'm assuming docker.io/x11docker/{xserver,fluxbox} are your images.  Am I wrong?  Also, doesn't the x11docker script test such things?

Also tried x11docker/openbox and x11docker/kde-plasma, and I get the same looping behavior.

The above turned out to be a python issue (I was trying to use a containerized python, and it wasn't working right).  After I installed python3 (uncontainerized), I get a different problem: (EE) xf86OpenConsole: Cannot open virtual console 8 (Permission denied).  So I sudo chowned /dev/tty8, but that resulted in (EE) xf86OpenConsole: Switching VT failed .

Thank you for the report!

The above turned out to be a python issue (I was trying to use a containerized python, and it wasn't working right).

Was this a custom setup of yours or something more common that x11docker should check and be able to handle?

I get a different problem: (EE) xf86OpenConsole: Cannot open virtual console 8 (Permission denied). So I sudo chowned /dev/tty8, but that resulted in (EE) xf86OpenConsole: Switching VT failed .

Likely x11docker wants to run Xorg on a different tty, but your system is not configured to allow this.
You can either run with sudo x11docker [...] or configure Xorg to allow the start. Compare https://github.com/mviereck/x11docker/wiki/Setup-for-option---xorg

The setup is very pure:  Start with Fedora CoreOS (with comes with podman), install absolutely nothing else on it (although I needed python for your script - more on that later...), and use the x11docker script with xserver container and one or more window manager or desktop environment containers to get a choice of desktop enviornments running on it.  I can use either VirtualBox or Qemu/KVM to house the CoreOS install for experimentation, but eventually my goal is for a bare metal install of CoreOS (with no additional installs on it) with a fully podman containerized single user no remote access desktop environment.
I will try sudo x11docker and report back, but I want to run x11docker completely rootless.  If I wanted to configure Xorg, but am using the x11docker/xserver container, would I need to rebuild the xserver container to do so, or is there a path into its configuration from some x11docker script arg?  Note: I may want to build my own xserver container anyway as I don't need nxagent or xpra (or xfishtank!), also probably not the hacked MIT-SHM (because nothing will be remote), but would benefit from virtualbox-guest-x11.  Do you have advice on doing so?  I see that the x11docker script is checking for config and labeling of the xserver container.
BTW: about x11docker script requirement for python.  It seems the requirement is very light.  Possibly the script would work with just using podman inspect --format, or by using jq (which is available probably wherever podman or docker are, and comes installed on CoreOS).  Of course, my case is extreme, as CoreOS does not have any version of python installed, and I don't want to install one (although I did so due to x11docker's requirement).

I will try sudo x11docker and report back, but I want to run x11docker completely rootless. If I wanted to configure Xorg, but am using the x11docker/xserver container, would I need to rebuild the xserver container to do so, or is there a path into its configuration from some x11docker script arg?

Oh, right, you are already using x11docker/xserver. The configuration of Xwrapper.config or running as root for --xorg should only be needed if using Xorg from host.
Here on Debian I don't have a configured Xwrapper.config on host but in the image x11docker/xserver only.
The lines in the Dockerfile are:
# configure Xorg wrapper
RUN echo 'allowed_users=anybody' >/etc/X11/Xwrapper.config && \
    echo 'needs_root_rights=yes' >>/etc/X11/Xwrapper.config

TTY switching works fine without root.
IIRC this succeeded in a fedora desktop VM, too.
I might set up a fedora CoreOS VM to reproduce your issue.

Note: I may want to build my own xserver container anyway as I don't need nxagent or xpra (or xfishtank!), also probably not the hacked MIT-SHM (because nothing will be remote), but would benefit from virtualbox-guest-x11. Do you have advice on doing so? I see that the x11docker script is checking for config and labeling of the xserver container.

You can reduce the Dockerfile of x11docker/xserver to your needs. Below a proposal for a Dockerfile reduced to Xorg. I've removed some of the tools, too (including the cute xfishtank); the LABEL list of available tools might be wrong now, would need a closer check:
FROM debian:bullseye

# cleanup script for use after apt-get
RUN echo '#! /bin/sh\n\
env DEBIAN_FRONTEND=noninteractive apt-get autoremove --purge -y\n\
apt-get clean\n\
find /var/lib/apt/lists -type f -delete\n\
find /var/cache -type f -delete\n\
find /var/log -type f -delete\n\
exit 0\n\
' > /apt_cleanup && chmod +x /apt_cleanup

# X servers
RUN apt-get update && \
    env DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
        xserver-xorg \
        xserver-xorg-legacy && \
    /apt_cleanup

# Window manager openbox with disabled context menu
RUN apt-get update && \
    env DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
        openbox && \
    sed -i /ShowMenu/d         /etc/xdg/openbox/rc.xml && \
    sed -i s/NLIMC/NLMC/       /etc/xdg/openbox/rc.xml && \
    /apt_cleanup

# tools
RUN apt-get update && \
    env DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
        catatonit \
        procps \
        psmisc \
        psutils \
        x11-utils \
        x11-xkb-utils \
        x11-xserver-utils \
        xauth \
        xinit && \
    /apt_cleanup

# configure Xorg wrapper
RUN echo 'allowed_users=anybody' >/etc/X11/Xwrapper.config && \
    echo 'needs_root_rights=yes' >>/etc/X11/Xwrapper.config

# HOME
RUN mkdir -p /home/container && chmod 777 /home/container
ENV HOME=/home/container

LABEL options='--xorg'
LABEL tools='catatonit cvt glxinfo setxkbmap \
             xauth xdpyinfo xdriinfo xev \
             xhost xinit xkbcomp xkill xlsclients xmessage \
             xmodmap xprop xrandr xrefresh xset xsetroot xvinfo xwininfo'
LABEL options_console='--xorg'
LABEL windowmanager='openbox'

ENTRYPOINT ["/usr/bin/catatonit", "--"]

BTW: about x11docker script requirement for python. It seems the requirement is very light. Possibly the script would work with just using podman inspect --format, or by using jq (which is available probably wherever podman or docker are, and comes installed on CoreOS). Of course, my case is extreme, as CoreOS does not have any version of python installed, and I don't want to install one (although I did so due to x11docker's requirement).

I am not entirely happy about the python dependency. x11docker also supports nerdctl that does not support nerdctl inspect --format yet. I've also tried jq but found that it is not installed by default everywhere, but it seemed that python is a widespread standard that can be expected.
I consider to check for jq if python is not installed and use it in that case.

Running sudo x11docker... does not work as it won't find the user's pulled images when running as root.  In other words sudo podman images returns nothing even though the user runing podman images has x11docker/xserver, x11docker/fluxbox among others.

Running sudo x11docker... does not work as it won't find the user's pulled images when running as root. In other words sudo podman images returns nothing even though the user runing podman images has x11docker/xserver, x11docker/fluxbox among others.

Ok, right; I confused a bit here, sorry.
Now I remember: Running Xorg within a container works only with rootful podman.
So you would need at least x11docker/xserver in rootful podman.
To run containers in rootless podman, but with root for the Xorg container:
sudo x11docker --backend=podman --rootless --xorg [...]

To avoid the need of sudo, you would have to install Xorg on host.
I failed to get Xorg running in rootless podman and I am not sure if it is possible at all.

Isn't the --xorg going to bypass my --xc=podman requirement to use the container version of xserver?  It seems to do that when I try it, as it gives me the x11docker ERROR: Did not find a possibility to provide a display. error message.  Dropping --xorg and keeping --xc=podman, I get back to my earlier (EE) xf86OpenConsole: Cannot open virtual console 8 (Permission denied) error in the log, even though now using sudo.  Could this be a selinux issue about consoles?
Note that a wayland-based x11docker desktop such as x11docker/kde-plasma does no better.  I have seen that Fedora Silverblue and Kinoite (the KDE variety of the same family) run wayland as the user, not as root.  So that suggests the possibility of a container doing so on CoreOS.  But, using x11docker/kde-plasma as the target desktop container did not change matters - I again get the same Cannot open virtual console 8... error in the log file.

BTW: if you want to get CoreOS up and running in VirtualBox quickly, I can help with that.  It's not your typical install-from-ISO-as-CD distro.

Isn't the --xorg going to bypass my --xc=podman requirement to use the container version of xserver?

You don't need to specify option --xc if image x11docker/xserver is available. x11docker uses it automatically (and tells so in a message).
Regardless if the image is available or --xc is specified, you can choose a desired X server, here with --xorg.

It seems to do that when I try it, as it gives me the x11docker ERROR: Did not find a possibility to provide a display. error message. Dropping --xorg and keeping --xc=podman, I get back to my earlier (EE) xf86OpenConsole: Cannot open virtual console 8 (Permission denied) error in the log, even though now using sudo.

I am not sure yet if I understand right.
Do you have x11docker/xserver now if you check sudo podman images? If not, and if Xorg is not installed on host, x11docker will find no Xorg that it could start with sudo x11docker [...].
The special example sudo x11docker --backend=podman --rootless --xorg [...] runs a rootful podman for x11docker/xserver, but a rootless podman for the desired container.
Or maybe I misunderstand you? Please show me your commands and the resulting error messages.

Note that a wayland-based x11docker desktop such as x11docker/kde-plasma does no better. I have seen that Fedora Silverblue and Kinoite (the KDE variety of the same family) run wayland as the user, not as root. So that suggests the possibility of a container doing so on CoreOS. But, using x11docker/kde-plasma as the target desktop container did not change matters - I again get the same Cannot open virtual console 8... error in the log file.

It doesn't matter which image you use because x11docker sets up Wayland or X before the container is started.
Running X or Wayland from console always needs some sort of privileges. Aside from obvious sudo there are possibilities with suid X or some obscure logind privilege setups. So even if your process list shows that X or Wayland are running as an unprivileged user, something in background gave some privileges to the process.
I admit that I don't understand all ways how privileges are granted.

BTW: if you want to get CoreOS up and running in VirtualBox quickly, I can help with that. It's not your typical install-from-ISO-as-CD distro.

Thank you for offer! I'll try to reproduce in my regular fedora VM.
But very likely it is just the issue that Xorg in container does not run with rootless podman. x11docker should catch that case and print a message instead of producing an error.

Running podman images as user, I have: x11docker/xserver, x11docker/fluxbox, x11docker/openbox, and x11docker/kde-plasma.  But, sudo podman images sees no images, as they're all in the user's container storage, not root's.
Running in the x11docker git clone directory:
sudo ./x11docker --backend=podman --rootless --xorg --desktop x11docker/fluxbox

This produces
x11docker ERROR: Did not find a possibility to provide a display.
...

error message.  I can't copy-n-paste it or transfer it out of the VirtualBox VM easily with CoreOS, because no VirtualBox guest extensions are present.  If you need the whole output and/or log file, I will work on a transfer ability.
If I instead run
sudo ./x11docker --backend=podman --rootless --xc=podman --desktop x11/fluxbox

that's when I get
(EE) xf86OpenConsole: Cannot open virtual console 8 (Permission denied)

in the ~/.cache/x11docker/x11docker.log file
If I need other privileges to run rootless, I can worry about getting them later.

I can't copy-n-paste it or transfer it out of the VirtualBox VM easily with CoreOS, because no VirtualBox guest extensions are present. If you need the whole output and/or log file, I will work on a transfer ability.

Thank you! I don't need the full text, I just needed to sort the commands and their error messages.
The first error is correct:

Running in the x11docker git clone directory:

sudo ./x11docker --backend=podman --rootless --xorg --desktop x11docker/fluxbox

This produces

x11docker ERROR: Did not find a possibility to provide a display.

Please provide x11docker/xserver in rootful podman. Than this should work. So run sudo podman pull x11docker/xserver or build the reduced Dockerfile example above.
Your second command sudo ./x11docker --backend=podman --rootless --xc=podman --desktop x11/fluxbox should have produced the same error, but it did not. This is likely an x11docker bug. It seems that it does not try to use rootful podman for the Xorg container although it should. I'll check this.

I did a podman pull x11docker/xserver as root, and now things are much closer to working.  I am getting an X server with a gray background and mouse tracking, but no fluxbox desktop in it (which would have a root menu and a toolbar, by default).  Also, no way to exit, except by ACPI shutdown.  This with the sudo ./x11docker --backend=podman --rootless --xorg --desktop x11docker/fluxbox command.  Same thing for x11docker/kde-plasma.

I'll bet it isn't finding x11docker/fluxbox or any other desktop the user has pulled.  Maybe I should run two x11docker instances, one as root for the server the other as user for the desktop?  Is there a way to do that?

Surprisingly, the fluxbox desktop appeared after a very long delay.  I didn't need to start a separate x11docker instance as I thought, just waiting.   Hopefully this long delay is a one-time issue.

I've tried to reproduce the issue but now have the unrelated problem that my root partition has not enough space left for image x11docker/xserver to pull it in rootfull podman.
If you somehow can provide me the log file ~/.cache/x11docker/x11docker.log , I might find a hint what is blocking the process.

Also, no way to exit, except by ACPI shutdown.

At least CTRL+ALT+F(n) should be possible to switch to another console.

Maybe I should run two x11docker instances, one as root for the server the other as user for the desktop? Is there a way to do that?

That is basically possible, but should not be needed.

Surprisingly, the fluxbox desktop appeared after a very long delay. I didn't need to start a separate x11docker instance as I thought, just waiting. Hopefully this long delay is a one-time issue.

You could specify another tty with option --vt , e.g.--vt=8, and switch back to your current tty (check it with tty) with CTRL+ALT+F(n). Than you can read the x11docker terminal output. if you add --debug, this might give some useful hints.

It is late here, and I am tired. I'll look at this tomorrow again.

I have both fluxbox and openbox working, but not kde-plasma.  It looks like the x11docker/fluxbox and x11docker/openbox containers have no X client apps, such as xterm, in them.  Of course they are designed to be used as base layers for other containers, so I will do that.
I noticed that on distros where I can start X as non-root (via the startx -> xinit route), they have Xorg.wrap, a setuid for doing just that.  But a more secure scheme with just the necessary capabilities is probably possible: some setcap'ed way of launching the x11docker script as the user instead of root.

I have both fluxbox and openbox working

Do you still have the long startup delay? It can be as well a podman issue that I once had, too. It can be solved temporary with podman system prune to clean up the podman storage.

but not kde-plasma

kde-plasma needs --init=systemd. Did you set this option?
Also add option --desktop for desktop environments, otherwise x11docker will run a window manager that might cause issues.

But a more secure scheme with just the necessary capabilities is probably possible: some setcap'ed way of launching the x11docker script as the user instead of root.

The ideal way would be to be able run run Xorg with rootless podman. We might give it a try again and ask the podman developers for help. x11docker already gives the bare minimum of capabilities to the container of x11docker/xserver that is needed to run Xorg.
One guess: Xorg in rootless podman might fail because podman's root in container is not root on host.

I tried podman system prune, and it did prune some things.  I also pulled the x11docker/mate image to try a mid-sized X11 desktop instead of the large wayland kde-plasma desktop to see if I can get anything else beyond fluxbox and openbox.  However, I'm getting mysterious errors with mate that suggests a podman bug - go backtraces, but only after a very long delay even after having done podman system prune.  From the looks of things, it is something running within the desktop container or launched from podman for the desktop container called 'exe' running as the user.  The behavior is strange: it isn't doing much but very slowly reading and writing disk - despite the disk being a fast SSD, the rates are 500K read/sec and 2M write/sec.  With very low cpu usage.  So, WTF is that?
I will keep --init=systemd in mind with kde-plasma, but all indications are that I never got close to that problem due to podman issues prior to that.  I always have the --desktop option on.
I don't think Xorg.wrap would fail to run just because it is in a rootless container.  But it would fail to deliver the necessary capabilities, which the rootless container didn't inherit from the user.  I think that the necessary capabilities have to be delivered from the outside-in starting with the x11docker script itself.  However, I know only enough about capabilities to know I don't know enough about capabilities :(  But I do think this can be fixed externally to podman, assuming podman doesn't intentionally drop excess capabilities it has inherited when run rootless.

And mate is finally running!  After over an hour of that exe process doing whatever.

Do you have a way to send me ~/.cache/x11docker/x11docker.log?
A useful logfile would be after you have terminated a container that as such a long startup delay.
I would like to find out what causes the delay.
I've build the Dockerfile above reduced to Xorg in x11docker/xserver in rootful podman. Here a startup of x11docker/fvwm is pretty fast.

Once they finally start up the first time, each desktop has pretty fast (~10sec) startup times after that.  So I will have to try a new one.  I will try fvwm and send you the log if it is slow.  If that isn't slow, I will try XFCE or LXDE.
I figured out a way to get files out of a vbox without guest additions by using a USB drive.  I would rather have a way to mount something rw in the vbox that is ro outside, and there's probably a way to do that, but the USB drive trick works without further thought, so it wins.
I think that exe app must be part of podman itself.  It isn't just waiting and writing small things to a log file.  It's writing at a steady 2M/sec rate for over an hour, yet not growing the vbox disk image substantially over that time, hence not appending most of that to a log.  I've seen what your logging does when it waits for that event - probably only writing at a few bytes/sec rate, and that would be pure appending.

Once they finally start up the first time, each desktop has pretty fast (~10sec) startup times after that. So I will have to try a new one. I will try fvwm and send you the log if it is slow. If that isn't slow, I will try XFCE or LXDE.

You mean, once you have waited a long time for the first startup, later startups with x11docker are fast?
That sounds pretty much like a podman issue, no an x11docker issue. However, to be sure it makes sense to check the log file.
It might be worth to compare with rootful podman. For example, run:
sudo podman pull x11docker/fvwm
sudo x11docker --backend=podman --desktop x11docker/fvwm

fvwm was fast as user.  Nothing interesting in the log, but I've saved it in case it may prove useful by comparison to something slow.

You mean, once you have waited a long time for the first startup, later startups with x11docker are fast?

Yes.  On to XFCE as user.  If that's slow, I'll try it as root.  Actually, I will pull it as root, then podman image scp it to the user, so I don't waste network bandwidth.
Also, if it is slow, I will try pstree to determine the provenance of that exe process.

Just an idea: Maybe podman is pulling the image again instead of using the one you have pulled before?
x11docker sets option --pull=never so this should not happen.

Maybe podman is pulling the image again instead of using the one you have pulled before?

There's no network activity, either.  I just had to forcibly stop the vm that was trying to start XFCE - I was logged in on another console as root spying on it with pstree and other things, and something I did caused the vm to go crazy.  But, I did see the exe process has as its only arg an image overlay file name, so it must be part of podman.
I'll take a look at the log file...

No log file.  So I restarted XFCE, and will be a bit more careful while spying on it.
doing ls -l /proc/3744/exe where 3744 is the pid of that exe process shows that it is /usr/bin/podman itself, run with a different name (probably doing arg[0] conditionalization).

Found this:
https://github.com/containers/podman/issues/7866
which suggests my goal of a fully rootless containerized X server is a dead end, at least with kernel behavior as of two years ago.  That's a bummer, if true.  It surprises me that there is no way to pass a small set of capabilities into a rootless namespace container if the invoking user had those capabilities on the way in.

Found this:
https://github.com/containers/podman/issues/7866

That's really a bummer. Good that you found this. I did several attempts meanwhile to get Xorg running, but to no avail. Now I know that I should give up.
I found a ticket related to the very slow startup of your podman containers, maybe it contains a useful hint: https://github.com/containers/podman/issues/13226
I also found the ticket where I once got the hint to use podman system prune: https://github.com/containers/podman/issues/6288

The XFCE desktop finished loading, and appears to work. Attached is the x11docker.log file

Thank you for the log file! At the first glance everything seems to run correctly, but podman is just slow. I'll look closer at which point exactly it slows down.

I had a closer look at the startup delay and found that one great delay happens when x11docker runs command podman logs $Containername ... that serves to catch container output.
x11docker did this even before pid 1 of container was up.
Now x11docker waits with this command until pid 1 of container is running.
Maybe this makes a difference.

I found a ticket related to the very slow startup of your podman containers, maybe it contains a useful hint: https://github.com/containers/podman/issues/13226

No, mine has graphDriverNam: overlay.  Which makes sense since this is Fedora CoreOS, and they wouldn't give that the wrong fs for podman.
As for podman system prune, there does seem to be quite a bit to prune after runs in the user's storage.  In the root's storage, I'm getting some kind of error now.
I thought the slowdown was occurring after X11 initialized and before the desktop started, so the root's containers are not the slow ones, it's the user's that are slow.  Is that correct?  That's what it looks like on the console when I get one of those very slow runs (still only the first run for each desktop image): the console goes gray and I see an "X" mouse cursor tracking the mouse very quickly, and the hour+ delay is between that and the desktop environment showing up.
I might have caused that root error running podman system prune when my VM went crazy that time, and I had to hard shutdown the VM.  I can rm the whole thing and reload x11docker/xserver, and see what happens.
Back to rootless container for X11.  I have 2 goals for containerizing X11 (and eventually wayland): to use a stable distinct distro (Debian stable or Alpine, for example) version of X11 so that it can be updated at will separately from the base OS, and also get very good rootless sandbox security for it.  I can still do the first part, but not the second.  So what is the "best" sandboxing that can be done?  It may not be running podman as root.  X11 running in a rootfull container with something setuid to start that container might be less secure than X11 running as user started by Xorg.wrap.

Now x11docker waits with this command until pid 1 of container is running.
Maybe this makes a difference

I'll pull your changes and try it.
BTW: thanks for all of this help!

I ran your newest x11docker script, I podman rmi'ed the x11docker/xfce image and repulled it from the repo.  I also cleaned out the root's image store and reinstalled the xserver image there.  It ran with about a 15min delay between X starting (gray console + "X" mouse cursor tracking the mouse) and XFCE starting.  The log is attached.
xfce2-x11docker.log

I thought the slowdown was occurring after X11 initialized and before the desktop started, so the root's containers are not the slow ones, it's the user's that are slow. Is that correct?

Yes, that is correct. Xorg runs in rootful podman, and if the grey screen with the mouse cursor appears reasonable fast, than this part is ok.
The slowdown happens in rootless podman after podman run [...].
The second log indicates that delay is during podman inspect $Containername. I have changed x11docker debug output a bit to make this more obvious, but I am already pretty sure.
I would conclude that podman just needs veeery long until the container is accessible for a podman inspect command. But podman inspect does not fail but waits until the container is ready.
It is interesting that you never seem to have the issue with rootful podman.
This should be reported to podman. However, they would need something easier to reproduce than the full x11docker setup.
 Normally you should get an error message of lxterminal pretty soon ("Display not found").
What happens if you run podman inspect CONTAINERID in parallel?

Back to rootless container for X11. I have 2 goals for containerizing X11 (and eventually wayland): to use a stable distinct distro (Debian stable or Alpine, for example) version of X11 so that it can be updated at will separately from the base OS, and also get very good rootless sandbox security for it. I can still do the first part, but not the second. So what is the "best" sandboxing that can be done? It may not be running podman as root. X11 running in a rootfull container with something setuid to start that container might be less secure than X11 running as user started by Xorg.wrap.

Basically two possibilities are left:

Xorg on host.  Runs with setuid wrapper Xorg.wrap. The host is not protected from possible hardness security issues in Xorg.
Xorg in rootful container. Xorg's access to host is restricted to a few device files and few capabilities. Disadvantage: Needs root password at every startup.

Some custom setuid wrapper to run a rootful Xorg container is possible, but does not sound great.
Though, we could separate the containers and somehow make a suid script from the first line. Example:
read Xenv < <(sudo x11docker --xorg --backend=podman --printenv --xonly --desktop)
env $Xenv x11docker --backend=podman --hostdisplay x11docker/xfce

(I had to make a fix to make this work, please update before you try it.)

I again pulled your latest x11docker changes.  I pulled the lxde image and tried podman run x11docker/lxde lxterminal, and it printed the error almost instantaneously.  Not enough time to try a podman inspect in parallel.  But, then when I use that lxde image to start the full x11docker experience, it again stalls in between X server and desktop, with that exe process doing its thing.  There was also a go backtrace and core dump (but ulimit -c is 0 by default, so nothing got saved).  But the x11docker/lxde did eventually complete.  Log file attached.
lxde1_x11docker.log
I am not convinced that the podman developers would find the full x11docker setup too cumbersome to deal with.  They should be able to get considerable detail from debugging exe once they reproduce the case.  And it should be reproducible, as there's just not much involved.  Anyway, we can let them decide.  I will file an issue with them, but I should first start over with a completely fresh VM to make sure it is reproducible.
More thoughts on a custom sandbox for Xorg: it occurs to me that there are alternative ways to rootless containers.  One is relying on the kernal.unprivileged_userns_clone=1 flag to enable setup of namespaces by non-root users, which is what podman and bubblewrap (flatpak) do.  The other is a setuid exe for setting up the namespace, like what firejail does.  I wonder if these setuid sandboxers suffer the same inability to inherit capabilities into a rootless container?  Probably not unless they take explicit steps to do so, as the process is root at the point when the namespaces are established.  So, maybe they have an advantage (although the bubblewrap people have long stated that firejail being setuid itself is an extreme disadvantage).
Another alternative is to run Xorg as a dedicated "system" unprivileged user with no ability to read the real user's files, no ability to do networking, read mounted file systems, etc.  I'm sure selinux rules can be formulated to lock it down even further.  This would be a type of sandbox without namespaces, but still very well locked down.  It might be possible to make this user unable to read anything in /proc for example.

My streamlined attempt to reproduce the delay problem failed.  In this attempt, starting with a fresh VM, I pulled only the xfce desktop as user (and of course x11server pulled as root).  The sudo ./x11docker ... test took only ~2min for the XFCE desktop to appear.
This could mean that the original VM's user image store was somehow corrupted, or that there is some interference when there are many images in the store, or just bad luck on my part.
So I am not going to bother reporting it until it crops up again.
I'm going to continue to pursue better security for rootless Xorg.  If you'd like, I can keep you posted.  If I do come up with something, I may start a github project to continue to refine it publicly.
Again, thanks so much for all of the help!

More thoughts on a custom sandbox for Xorg: it occurs to me that there are alternative ways to rootless containers. One is relying on the kernal.unprivileged_userns_clone=1 flag to enable setup of namespaces by non-root users, which is what podman and bubblewrap (flatpak) do. The other is a setuid exe for setting up the namespace, like what firejail does. I wonder if these setuid sandboxers suffer the same inability to inherit capabilities into a rootless container?

I assume the capability restrictions only occur if the container is based on rootless userns.
It seems to me that capability namespacing comes along with userns remapping. (Is there really a thing like capability namespacing?)
And rootless containers are only allowed with userns remapping.
Note, I am not sure at all, it is just the picture I get yet.
Setuid setups are essentially the same as a rootful container and have no limits. Here we would have to look which namespaces are used. I doubt that e.g. firejail uses all possibilities to isolate its containers. But I don't know and might be wrong.
One quick attempt to get a rootful Xorg podman container without providing the password could be a bash script that runs Xorg with x11docker only and gets an entry in /etc/sudoers with NOPASSWD.
As long as this script does not take arguments, the risk is reasonable low.
(side note, directly setuid on scripts is not possible and is denied by the kernel.)

Another alternative is to run Xorg as a dedicated "system" unprivileged user with no ability to read the real user's files, no ability to do networking, read mounted file systems, etc. I'm sure selinux rules can be formulated to lock it down even further. This would be a type of sandbox without namespaces, but still very well locked down. It might be possible to make this user unable to read anything in /proc for example.

An interesting thought. Myself i am not a fan of selinux, I feel it as another layer of complexity. And Xorg would still run as root thanks to Xorg.wrap. Some logind tricks to run Xorg rootless do some magic in background that I don't understand either.
I prefer simple setups that I understand.
AFAIK in BSD it is possible to drop capabilities for a process before it starts and it cannot gain them back.
It even goes so far that the entire system can lock itself down to a point where a reboot is needed to regain root privileges.

My streamlined attempt to reproduce the delay problem failed. In this attempt, starting with a fresh VM, I pulled only the xfce desktop as user (and of course x11server pulled as root). The sudo ./x11docker ... test took only ~2min for the XFCE desktop to appear.

~2min still sounds a bit long, although much better than before. Here it takes ~20sec, and my computer is ~12 years old. Some slowdown might be caused by using a VM, but I doubt that this explains everything.

I'm going to continue to pursue better security for rootless Xorg. If you'd like, I can keep you posted. If I do come up with something, I may start a github project to continue to refine it publicly.

I am quite interested! Maybe I'll have some ideas, too, and I am willing to test and check out things.

Again, thanks so much for all of the help!

I quite appreciate our conversation, and thanks for your close looks on what is going on! And thank you for giving x11docker a shot even after having so much trouble to get it work.

I am able to get a rootless Xorg server up in an Alpine VM without using Xorg.wrap, and get a working fluxbox desktop!  I bypassed Xorg.wrap using a crafted ~/.xserverrc file that directly execs /usr/libexec/Xorg, and tested that it bypasses Xorg.wrap by making /etc/X11/Xwrapper.config not allow anything but root:
$ cat ~/.xserverrc
#!/bin/sh
exec /usr/libexec/Xorg -nolisten tcp "$@"

$ cat /etc/X11/Xwrapper.config
allowed_users=rootonly
need_root_rights=yes

I tested that Xwrapper.config before creating ~/.xsessionrc to make sure it prevented Xorg.wrap from launching a rootless Xorg.   Also, running getcap -r / returns nothing: no files have capability extended attributes on this Alpine system.
Next, I installed bubblewrap, and changed ~/.xserverrc to be:
$ cat ~/.xserverrc
#!/bin/sh
exec /usr/bin/bwrap --dev-bind / / --cap-drop ALL --unshare-all -- /usr/libexec/Xorg -nolisten tcp "$@"

And it still works!  But much slower (~15sec startup delay) for some reason, and has problems exiting, probably due to no pid1 zombie management(?).  But, that's a rootless namespace sandbox without any capabilities, as /usr/bin/bwrap is not setuid on Alpine.
What would happen if you used Alpine instead of Debian for the xserver image, with these changes minus bwrap?  How is Alpine getting around the need for cap_sys_tty_config?  What would happen if this is tried in a Debian VM?
About selinux: it is already enabled on all Fedora, including CoreOS, and mostly stays out of the way.  I was just brainstorming other ways I could lock down Xorg in CoreOS without using a namespace sandbox.

I am beginning to think that the only issue with a rootless Xorg container in my CoreOS setup is that the user is not in the video or input groups.  And I can't add the user to those groups, either.  I think this might be a CoreOS bug, so I'm asking about it in the Fedora discussion group about CoreOS:
https://discussion.fedoraproject.org/t/groupadd-usermod-dont-always-work-in-coreos/41735
I have fine-tuned the Alpine bubblewrap sandbox around Xorg some more without much difficulty.  It really looks as though if the necessary /devs are bound in the container, and the user is in video and input groups, a rootless Xorg container should just work.  Perhaps you might add a test to a rootless Xorg container to check if the user is a member of the video and input groups?

Sorry for my late response! I am a bit distracted these days.
Your tests with bubblewrap are quite interesting! I'll check it out, too.
I wonder if it is possible to create a container of x11docker/xserver with podman (or docker) and to use its file system with bubblewrap.
x11docker might provide to bubblewrap a host Xorg as well as a bubblewrapped Xorg from an x11docker/xserver container.

What would happen if you used Alpine instead of Debian for the xserver image, with these changes minus bwrap?

I'll try this. Alpine images are a bit hard to set up because one has to figure out several dependencies (and the nxagent build has to be done different). However, a test image with Xorg and a few tools is not too hard.
Overall alpine could provide a smaller image than the current debian based one.

How is Alpine getting around the need for cap_sys_tty_config?

Alpine does not use systemd/logind, so it must be different.
I found that a logged in tty belongs to the logged in user. This might be the key. (I've also tried to add group tty to rootless podman container user, but to no avail. Maybe the groups are namespaced as well?)
$ tty
/dev/tty2
$ ls -l /dev/tty2
crw------- 1 lauscher tty 4, 2 27. Aug 16:35 /dev/tty2

and the user is in video and input groups, a rootless Xorg container should just work. Perhaps you might add a test to a rootless Xorg container to check if the user is a member of the video and input groups?

I found that in rootful container with non-root container user it is enough to add the container user to desired groups. x11docker already adds the x11docker/xserver user to groups video and render. Groups tty and input could be added as well. (Experimentally I already tried that for Xorg in rootless podman.)

I am beginning to think that the only issue with a rootless Xorg container in my CoreOS setup is that the user is not in the video or input groups. And I can't add the user to those groups, either. I think this might be a CoreOS bug, so I'm asking about it in the Fedora discussion group about CoreOS:
https://discussion.fedoraproject.org/t/groupadd-usermod-dont-always-work-in-coreos/41735

Odd. And x11docker might need to check those additional group files. An example for useless complexity. sigh

I wonder if it is possible to create a container of x11docker/xserver with podman (or docker) and to use its file system with bubblewrap.
x11docker might provide to bubblewrap a host Xorg as well as a bubblewrapped Xorg from an x11docker/xserver container.

You can podman-image-mount it somewhere, then bind that as the root in the bubblewrap sandbox.

Alpine does not use systemd/logind, so it must be different.

Do you think sysVinit or runit systems would all allow Xorg to run without cap_sys_tty_config?

I found that a logged in tty belongs to the logged in user. This might be the key.

That occurs in my CoreOs install as well, so it's not the only key.
BTW: I directly edited /etc/group and /etc/gshadow on my CoreOs install so that the user is in video, input and render groups, but still cannot start a containerized rootless Xorg.  Am I doing it right?:
x11docker --vt=7 --backend=podman --xc=podman --desktop x11docker/fluxbox
which gives me the error Did not find the possibility to provide a display. even though the user has the x11docker/xserver image.
I may install bubblewrap and Xorg directly in CoreOs to see if I can replicate the success I had in Alpine that way.  I also have a Debian VM that I can boot either with sysVinit or systemd  (the MX distro allows the choice at boot time), so I can try this there as well.  But, on this Debian, none of startx, xinit or Xorg are suid or have getcap.  So my guess is that a rootless bubblewrap sandboxed Xorg would work there, assuming startx works as it is supposed to without a sandbox.

The Debian experiment worked - you can start Xorg in a rootless bubblewrap sandbox there as well.  Also, same ownership of /dev/ttyN exists there.  Also, this was systemd.  I didn't even bother altering things so that the display manager wasn't functioning with a root Xorg running - I just switched to a different tty, logged in there, and ran startx with ~/.xserverrc written to call bwrap around Xorg.

I've determined that there cases in which rootless sandboxing Xorg with bubblewrap only works without --unshare-pid (and thus --unshare-all as well).  In these cases, if --unshare-pid is used, the xf86OpenConsole: Cannot open virtual console... message we've sometimes seen shows up.  I don't know what distinguishes the cases - perhaps what modules/extensions are loaded by Xorg?
I don't know if that podman has the same ability as bubblewrap to unshare specific namespaces while sharing others.

You can podman-image-mount it somewhere, then bind that as the root in the bubblewrap sandbox.

Good to know! That allows further possibilities.

Do you think sysVinit or runit systems would all allow Xorg to run without cap_sys_tty_config?

I am not sure how systemd/logind do the magic, so I don't have an idea for SysVinit and runit.
Maybe the magic is just:

tty owned by logged in user
user in groups video, render and input
something more?

I directly edited /etc/group and /etc/gshadow on my CoreOs install so that the user is in video, input and render groups

Please also check if the device files in /dev/dri belong to video and render, and if the device files in /dev/input belong to group input. Otherwise the groups would not help to allow access.

Am I doing it right?:
x11docker --vt=7 --backend=podman --xc=podman --desktop x11docker/fluxbox
which gives me the error Did not find the possibility to provide a display. even though the user has the x11docker/xserver image.

Currently x11docker denies to run Xog in rootless podman because it is not supported/would fail.
To disable the check, look at https://github.com/mviereck/x11docker/blob/master/x11docker#L3338-L3343
      --xorg|--weston|--weston-wayland)
        [ "$Xcrootless" = "yes" ] && {
          $Message "${1:-} cannot claim a new virtual terminal (option --vt) with rootless X container (option --xc)."
          Return=1
        }
      ;;

If you disable line Return=1, x11docker will run Xorg in rootless podman. Option --printcheck would show you those check messages.

I've determined that there cases in which rootless sandboxing Xorg with bubblewrap only works without --unshare-pid
I don't know if that podman has the same ability as bubblewrap to unshare specific namespaces while sharing others.

Good find!
podman sets up all possible namespaces by default, but they can be disabled manually.
To add this to the podman command used by x11docker to run x11docker/xserver, look at https://github.com/mviereck/x11docker/blob/master/x11docker#L4533-L4540
    --xorg)
      Xcontainercommand="$Xcontainercommand $Xc_capdrop"
      Xcontainercommand="$Xcontainercommand $Xc_user"
      Xcontainercommand="$Xcontainercommand $Xc_containerx"
      Xcontainercommand="$Xcontainercommand $Xc_hostx"
      Xcontainercommand="$Xcontainercommand $Xc_gpu"
      Xcontainercommand="$Xcontainercommand $Xc_console"
    ;;

You can add a line:
      Xcontainercommand="$Xcontainercommand --pid=host"

I don't know what distinguishes the cases - perhaps what modules/extensions are loaded by Xorg?

I doubt that, but have no great idea either.
Maybe some sort of authentication - if the pid namespace is the one from host, the system believes that Xorg is a local process and allows some access that it would deny otherwise. But I don't know, just guessing.

lease also check if the device files in /dev/dri belong to video and render, and if the device files in /dev/input belong to group input. Otherwise the groups would not help to allow access. Normally the kernel adds the devices to these groups.

/dev/dri/card0 is in group video and /dev/dri/renderD128 is in group render.  Everything in /dev/input is in group input.

Currently x11docker denies to run Xorg in rootless podman because it is not supported/would fail.
To disable the check, look at https://github.com/mviereck/x11docker/blob/master/x11docker#L3338-L3343

It looks like other parts of the script are denying the possibility.  After making the above change, I am getting warnings: Option --xc with rootful container backend needs to be started with root privileges.  root needed to claim a tty.  Fallback: Disabling option --xc  and Option --xc not possible: - without image x11docker/xserver - with runtime kata-runtime - in MS Windows Fallback: Setting --xc=no
In my case both root and user have x11docker/xserver images in their stores.

It looks like other parts of the script are denying the possibility.

Sorry, I should have checked it better before.
I have added a new option --experimental that allows me to add some experimental code.
For now it allows Xorg in rootless containers and also adds --pid=host to Xorg containers.
We can add further experimental code this way.
Yet you should be able to run rootless --xc=podman --xorg without an x11docker error but with Xorg errors only.

Still similar problems.  Log file attached...
fluxbox-x11docker.log

On Alpine, where --unshare-all (which includes --unshare-pid) works when rootless bwraping Xorg, the Xorg version is newer than on Debian where it doesn't work to --unshare-pid.  Also, Debian had loaded one extra extension: SELinux.  But using Xorg's -extension arg to remove that had no impact on the inability to --unshare-pid.
The Xorg versions are:
Debian (10.12): Xorg is 1.20.4, with pixman 0.36.0
Alpine: Xorg is <IP_ADDRESS>, with pixman 0.40.0
I have a feeling that the Alpine developers changed something (hence that last extra '.4') and compiled Xorg themselves, considering the other ways Alpine differs from almost every other distro by relying on busybox and musl.  What that might mean is that if you want rootless capability in more places, it might be worth the effort to create an Alpine version of x11docker/xserver.  That's just a theory.  At the very least it might enable --unshare-pid.

As a first quick attempt I've build an x11docker/xserver image based on debian bookworm instead of debian bullseye. Debian bookworm has the same Xorg and pixman versions as alpine.
The startup in rootless podman failed, but shows different and unexpected errors. So some sort of progress!
The key error message is (EE) open /dev/dri/card0: Permission denied although container and host user are in groups video and render.
I'll give an alpine based image a try. It might have the nice side effect of creating a smaller image overall.

Did you try it without unsharing pid?

Did you try it without unsharing pid?

No; using less namespacing would only ease things, not break them.
Yet I run some tests with an alpine based image, but I am running into the same error message (EE) open /dev/dri/card0: Permission denied.

No; using less namespacing would only ease things, not break them.

True.  But do we know yet whether the problem is due to the pid namespace, groups, or even if there is something in /dev that should have been mounted?  I know the error messages say certain things, but I wonder if Xorg tries several different things and would take the first success, if none succeed it only reports the final failure.  Meaning that a possible successful route might not involve fixing what that error messages implies is the culprit.

But do we know yet whether the problem is due to the pid namespace, groups, or even if there is something in /dev that should have been mounted?

Basically we can compare with a successful startup of Xorg in rootful podman. The Xorg log in rootless podman should be identical once all issues are solved.

But do we know yet whether the problem is due to the pid namespace, groups, or even if there is something in /dev that should have been mounted?

The list of shared devices is valid for rootful podman, so it must be valid for rootless podman as well.
As far as I can see, the only essential difference is user namespacing.
podman has an option --userns, but this one does not take =host. It takes an argument --keep-id that maps the container user to the same uid as on host, but changes the uid of container root. (By default and without option --user, the container root uid 0 maps to host user uid.)

Currently I assume your attempt with bubblewrap is the way to go.

Maybe so.  I'd still like to unshare pid everywhere.  Maybe bubblewrap with an Alpine chroot on Debian is worth a try to see where the problem is.  Also, I like your idea of bundling several window servers together: Xorg, Xephyr, Wayland, XWayland, Xpra, etc. all in the same chroot.  Maybe as a flatpak, which uses bubblewrap.

On Alpine, even though I can bwrap Xorg with --unshare-pid, I get a strange behavior I don't on Debian.  The bwrapped Xorg doesn't prevent keyboard input from reaching the tty it was launched from.  In other words: from tty1, launch Xorg in a sandbox (either on tty1 or some other, it doesn't matter) and start some window manager (fluxbox in my case).  Start up xterm in the window manager desktop.  Type the date command and note its output.  Exit the xterm by killing its window and exit the window manager through its root menu exit choice.  The X screen will disappear and you'll be back at the original tty, but you will see the date command has been entered into the tty, and see that its output is more recent than the date output you saw in xterm.  All input to Xorg was seen there (by the xterm) AND also sent to the original tty and buffered until the end of the Xorg session.  This doesn't happen without the sandbox.  I haven't determined what part of the sandbox is responsible, but it is not due to --unshare-pid.

All input to Xorg was seen there (by the xterm) AND also sent to the original tty and buffered until the end of the Xorg session.

That's really crazy. On the first glance I cannot see how this could happen, and why using namespacing would make a difference here.

I'd still like to unshare pid everywhere.

In general that should be possible. x11docker/xserver containers all run with all namespaces enabled, and with all capabilities dropped. Only for Xorg x11docker sets a few privileges more:

userns not possible as seen above
device files in /dev/dri and /dev/input needed
current /dev/ttyN, also specified in Xorg command with argument vtN`
groups video and render (I did not set input). (For a setuid Xorg the groups are likely not needed at all)
capabilities SYS_TTY_CONFIG (mandatory for Xorg), also DAC_OVERRIDE and KILL (due to x11docker needs)
file /run/udev/data
currently x11docker also shares /var/run/dbus, but likely not needed

In the --debug output of x11docker you can check this in the shown X container command.
 But once Xorg takes a keypress, the buffer should be empty. Maybe an issue on kernel level? Or somewhere between Xorg and the kernel?

The buffering at the tty is the normal buffering when keypresses occur during a previous command execution.  In this case, that previous command is my startx-like script launching Xorg and the window manager.  If I background that with &, the buffering disappears and the keypresses get simultaneously processed both within Xorg and at the original tty.  It is the dual direction of input that is weird, not the buffering.  To make it even weirder: once Xorg and the window manager are started, if I use the host-F# key sequence (this being a virtualbox VM) to switch ttys, I don't see the switch occur because the Xorg screen stays visible, but it does occur as any subsequent keyboard input is directed to the tty I switched to (as well as Xorg) instead of the original tty.  I think that the bwrap sandbox is preventing some interaction between Xorg and the kernel that is supposed to tell the kernel not to direct any input to whatever tty is currently attached to the seat - but only on Alpine.

I have lately been trying to sandbox weston-launch, and found that, like Xorg, it does not like unsharing the pid namespace.  In addition, it does not like unsharing the cgroups namespace, either.  This is on Debian.
However, non-top-level weston (backed by either Xorg or wayland) and Xwayland sandbox very well with all namespaces unshared.  They also do not lose much performance when sandboxed.  A fully sandboxed weston/Xwayland combination is faster than an unsandboxed Xephyr or Xpra, at least in my benchmarks.  I am using hardinfo -b "GPU Drawing" -m benchmark.so -a to benchmark.

Sorry that I did not respond for a long time!
I am still quite interested in your experimental setups.
I will try out some setups with bubblewrap with containers created but not started by podman. This might take some time until I really do so, though.

Sorry that I did not respond for a long time!
I am still quite interested in your experimental setups.

Ok, but should I keep posting to this issue?
My goal is a portable sandboxed desktop, including sandboxed display server, that is agnostic about window managers, and that can run on minimal distros that are more secure than typical (because immutable like CoreOS, small attack surface like Alpine, etc.).  The target audience is single-login desktop metal users, not Docker users.  Although there is overlap,  I would guess that most of your audience cares more about the ability to run GUI apps from containers without lowering the security on those containers, and not so much about whether the display server (X or Wayland) is itself in a sandbox.
So, there is no need to apologize about not responding timely!  You have already paid more attention to this issue than I anticipated, and I am grateful.

Ok, but should I keep posting to this issue?

Yes, please!

Although there is overlap, I would guess that most of your audience cares more about the ability to run GUI apps from containers without lowering the security on those containers, and not so much about whether the display server (X or Wayland) is itself in a sandbox.

You are right. Running X or Wayland in container adds some isolation, but still needs so much privileges that only small improvement of security is possible.
Some people like X servers in container to keep the host clean.

My goal is a portable sandboxed desktop, including sandboxed display server, that is agnostic about window managers, and that can run on minimal distros that are more secure than typical

I am not sure what you mean with "agnostic of window managers".

I am not sure what you mean with "agnostic of window managers".

No part of the sandbox setup or underlying distro depends on the user's choice of window manager and server pair.  The user should be able to choose dwm on Xorg or KDE on Wayland, or pretty much anything in between, and switch easily without installing anything in the underlying distro (especially if it is immutable). This should be an easy thing to accomplish: package WM or DE as flatpak or docker/podman container, as you do.  I add it as a goal to rule out any choices that are specific to some window managers or desktop environments but not others, even though I don't anticipate any.
I like what Fedora did with Silverblue and Kinoite, but I think they missed an opportunity by not moving the whole DE and display server into one or more flatpaks or podman containers.   The user has to choose the DE they want at installation, cannot easily switch, and the  OS and DE upgrade together  - so these are a counterexample to the agnostic goal above.  But, Fedora also has CoreOS without any WM/DE - so that was my initial target, CoreOS with WM/DEs and servers in flatpaks or podman containers.

$ cat ~/.xserverrc
#!/bin/sh
exec /usr/bin/bwrap --dev-bind / / --cap-drop ALL --unshare-all -- /usr/libexec/Xorg -nolisten tcp "$@"

I was experimenting to see if it's possible to run wayland contained with bwrap in a similar way
Initially trying this in a VT with non-root user:
bwrap --dev-bind / / --bind ~/bwhome /home/$USER --cap-drop ALL --unshare-all -- /bin/bash

In this bash environment I can see the home folder is the host contents of ~/bwhome, and real home is hidden, and I can confirm there is no network capabilities (see ip addr only has loopback and can't ping internet). But if changing /bin/bash to e.g. /bin/startplasma-wayland, or just running startplasma-wayland from the bwrapped bash, the resulting session seems to have full access to original home folder, and all capabilities including internet etc.

@digitalsignalperson
I have had considerable success recently running labwc within a bwrap sandbox, with clients and desktop components (including the taskbar I use with it, sfwbar) individually in separate bwrap sandboxes.  Labwc does not require dbus, and I don't use a single session dbus with it.  If some client needs dbus, I place that client in a bwrap sandbox along with its own session dbus (using dbus-run-session).  I gave up on KDE/plasma because the interaction between desktop components is too complex for me to figure out how to effectively securitize components.  I imagine GNOME would be similar, although I have not tried it.  The labwc project focuses instead on building a simple wayland compositor in the openbox style, meaning very simple configuration and very few runtime interdependencies.  It is much easier to tame with respect to sandboxing and other kinds of securitizing (I use AppArmor as well).
Very recently, I have been communicating with labwc's developers about how to get even finer grained security through limiting parts of the wayland protocol within different sandboxes: see https://github.com/labwc/labwc/discussions/1002.  So that certain sandboxed apps cannot spy on the clipboard, or take screenshots, or know what other windows exist, while other utilities can.

@jonleivent thanks for the links, I hadn't heard of labwc. It's surprising how dbus gives access to everything, and the list of wayland/wlroots protocols that can allow for monitoring/messing with many things (as described in the labwc discussion & with wl_monitor.py).
Do you have examples/snippets of your bwrap scripts you could share? E.g. how to run a sandboxed labwc, or how to run another client with dbus-run-session in a sandbox.
For isolating Xorg in general, some ideas here might be interesting/relevant:

Qubes-lite With KVM and Wayland https://roscidus.com/blog/blog/2021/03/07/qubes-lite-with-kvm-and-wayland/
Isolating Xwayland in a VM https://roscidus.com/blog/blog/2021/10/30/xwayland/

the proxy can also be used even without VMs, avoiding the need for any X11 support in Wayland compositors at all

https://github.com/talex5/wayland-proxy-virtwl accepts Wayland connections from client applications and proxies the messages to a host compositor

@digitalsignalperson

Do you have examples/snippets of your bwrap scripts you could share? E.g. how to run a sandboxed labwc, or how to run another client with dbus-run-session in a sandbox.

For labwc, the only devs that need to be bound into the bwrap sandbox are /dev/dri and /dev/input (at least on my old intel-only laptop).  You will also need to grant write to XDG_RUNTIME_DIR so that labwc can create its socket there.  It also needs read access to /var/cache/fontconfig.  Also, it needs a writable ~/.cache - and you can either do that with a tmpfs or have its own cache persistently kept somewhere and --binded in.
Something like:
bwrap --ro-bind / / --dev /dev --proc /proc --dev-bind /dev/dri /dev/dir --dev-bind /dev/input /dev/input \
--bind $XDG_RUNTIME_DIR $XDG_RUNTIME_DIR --tmpfs $HOME/.cache \
--tmpfs /tmp --tmpfs /var/tmp --tmpfs /var/cache --ro-bind /var/cache/fontconfig /var/cache/fontconfig \
--unshare-all --new-session -- labwc

It's certainly possible to block out much more from the sandbox, but I haven't investigated that yet.  I do have a labwc AppArmor profile that probably makes blocking out other things from the sandbox unnecessary.  I also have a seccomp bpf for it , which is slightly more permissive than the seccomp bpf I use for everything else.  I haven't tuned those tightly yet either.
Of course, if you want to be able to launch apps from labwc's menu, you'll need to allow that somehow.  I do that with a background process running external to the sandbox that listens on a named pipe for requests to launch apps, and then only launches apps that are in a specific write-protected folder just for it.  Which are all scripts that filter arguments passed in and start other sandboxes.  The named pipe is then --ro-binded into the bwrap sandbox (and others, like the one I use for the sfwbar taskbar).  The Exec rules in my labwc menu config file write to that named pipe.  Think of this as a safe alternative to dbus, where you control the entire protocol.  You can also configure your ~/.local/share/applications/*.desktop files so that their Execs write to that named pipe, so that other launchers pick those up.
The dbus-run-session is easy.  Take atril, the pdf reader.  It doesn't run without a session dbus.  So I put it into a bwrap sandbox as bwrap ....  -- dbus-run-session atril.  You can tighten this up by limiting what each such dbus can do using its config file.
Interesting links.  I once tried a multi-VM configuration, and found it was a huge consumer of resources.  I don't have the hardware necessary to use that effectively.  Also, probably my security requirements are likely lower than a typical Qubes-phile.

@jonleivent thanks for all those details. I notice with that bwrap command binding $XDG_RUNTIME_DIR, it's possible to escape the sandbox through dbus, even if labwc doesn't use dbus. I ran that command with -- labwc -s kitty to start in a sandboxed terminal, then I ran d-feet and launched an unrestricted terminal with org.kde.krunner (ofc mileage may vary depending on what dbus services are available).
If I don't bind XDG_RUNTIME_DIR, it can't start labwc (or any other wayland compositor I've tried). labwc in particular:
00:00:00.000 [ERROR] [libseat] [libseat/backend/logind.c:642] Could not check if session was active: No such device or address
00:00:00.000 [ERROR] [libseat] [libseat/libseat.c:79] No backend was able to open a seat
00:00:00.000 [ERROR] [backend/session/session.c:83] Unable to create seat: Function not implemented
00:00:00.000 [ERROR] [backend/session/session.c:248] Failed to load session backend
00:00:00.000 [ERROR] [backend/backend.c:86] Failed to start a session
00:00:00.000 [ERROR] [backend/backend.c:357] Failed to start a DRM session
00:00:00.000 [ERROR] [../labwc-0.6.4/src/server.c:248] unable to create backend

If I do bind XDG_RUNTIME_DIR, monitoring dbus-monitor --system I can see stuff related to org.freedesktop.login1.Session & SetType "wayland", so I'm thinking dbus may be involved in getting the seat. Trying the same with Hyprland I see that same indicator in dbus-monitor, and also the stdout says [libseat] [libseat/libseat.c:73] Seat opened with backend 'logind'.
Ideas for approaches to get a seat without binding XDG_RUNTIME_DIR or without unrestricted dbus access:

use seatd and bind /run/seatd.sock into the sandbox
proxy/filter the dbus and only allow a limited list of destinations like org.freedesktop.login1.Session (I saw bubblejail uses XDG D-Bus Proxy for similar filtering capability)

For X11, I don't have any luck with the bwrap commands. I can use the same command as above for labwc, but use startx: First I get "Cannot open virtual console 1 (Permission denied)". If I open that up with chmod, I get "Switching VT failed". If I try startx with a different VT I'm on on like  -- vt3 (ensuring no getty running on it, and given access to it) I get
xf86EnableIO: failed to enable I/O ports 0000-03ff (Operation not permitted)
vesa: Refusing to run, Framebuffer or dri device present
(EE)
Fatal server error:
(EE) no screens found(EE)
(EE)

Of course, if you want to be able to launch apps from labwc's menu, you'll need to allow that somehow. I do that with a background process running external to the sandbox that listens on a named pipe for requests to launch apps, and then only launches apps that are in a specific write-protected folder just for it. Which are all scripts that filter arguments passed in and start other sandboxes. The named pipe is then --ro-binded into the bwrap sandbox (and others, like the one I use for the sfwbar taskbar). The Exec rules in my labwc menu config file write to that named pipe. Think of this as a safe alternative to dbus, where you control the entire protocol. You can also configure your ~/.local/share/applications/*.desktop files so that their Execs write to that named pipe, so that other launchers pick those up.

I'm curious about this. Do I read this as you aren't running apps that are binded into the sandbox in /bin, but purposefully running apps outside the sandbox?

@digitalsignalperson

I notice with that bwrap command binding $XDG_RUNTIME_DIR, it's possible to escape the sandbox through dbus, even if labwc doesn't use dbus.

That doesn't happen in my case because I don't run a session dbus.  In other words, I have patched the login startup for that user so it doesn't auto-launch the session dbus.  I did this by masking the systemd unit for that user, so that no systemd services run for that user.  There is nothing in that user's $XDG_RUNTIME_DIR initially.
You can always bind something other than $XDG_RUNTIME_DIR in to the sandbox as $XDG_RUNTIME_DIR, like:
--bind /tmp/my-wayland-socket-dir $XDG_RUMTIME_DIR  where you create /tmp/my-wayland-socket-dir before.  Then bind the socket that gets created there into the client sandboxes.

I'm curious about this. Do I read this as you aren't running apps that are binded into the sandbox in /bin, but purposefully running apps outside the sandbox?

All of my apps, with just one exception (a terminal), run in their own private sandboxes.  Obviously, you can't start a bwrap sandbox directly from within an existing bwrap sandbox.  Instead, you need to have something outside any sandbox that can start new sandboxes - a background process (daemon).  When I raise labwc's primary menu, I have entries on to to for instance run firefox.  I want that to run in its own sandbox which has entirely different permissions than the one that labwc runs in.  In fact, labwc's sandbox has no network connection, so firefox wouldn't run there.  What I have labwc's menu do is instead of directly exec'ing firefox, it writes a command to run firefox out on a named pipe that the daemon is monitoring.  The daemon reads that, and launches a script that runs firefox in its own sandbox.  The scripts that the daemon is allowed to run (17 so far) are very restricted, so that no sandbox can use them to escape in a bad way (such as run arbitrary code outside any sandbox).  I may convert this later to using a unix socket instead of a named pipe (if for instance I need bidirectional communication, or the ability to pass fds), but for now the named pipe works well and is very easy to set up.

@jonleivent thanks for explaining & sharing some details of your unique setup. It would be interesting to play with it in a VM and poke around a working demo or sounds like good material for a blog post too.

Obviously, you can't start a bwrap sandbox directly from within an existing bwrap sandbox

Is nested bwrap not possible (PR landed here https://github.com/containers/bubblewrap/issues/164) or is that a design decision for your setup?
Still curious about getting the seat backend part for wayland to work, and also getting sandboxed x11 to work. I'm on an arch linux setup. Reading above about this working on alpine, I managed to get a working POC in a VM:
lxc launch images:alpine/3.18 alp --vm -c limits.cpu=4 -c limits.memory=4GiB --console=vga -c security.secureboot=false

# from another terminal set root password
lxc exec alp -- /bin/ash
passwd

# log in as root in the vga console
setup-xorg-base
apk add openbox xterm font-terminus
apk add bubblewrap
adduser person
addgroup person input
addgroup person video
exit

# login as person

# run X without sandbox - works
startx /usr/bin/openbox-session

# with sandbox - also works
bwrap --ro-bind / / --dev /dev --proc /proc --dev-bind /dev/dri /dev/dir --dev-bind /dev/input /dev/input \
--tmpfs /tmp --tmpfs /var/tmp --tmpfs /var/cache --ro-bind /var/cache/fontconfig /var/cache/fontconfig \
--tmpfs $HOME --unshare-all --new-session -- startx /usr/bin/openbox-session

Did you also get it to work with debian? I cannot
lxc launch images:debian/buster bust --vm -c limits.cpu=4 -c limits.memory=4GiB --console=vga -c security.secureboot=false

# from another terminal set root password
lxc exec bust -- /bin/bash
passwd

# log in as root in the vga console
apt install openbox xinit
apt install xterm
apt install bubblewrap
adduser person
addgroup person input
addgroup person video
addgroup person sudo
exit

# login as person

# run X without sandbox - works
startx /usr/bin/openbox-session

bwrap --ro-bind / / --dev /dev --proc /proc --dev-bind /dev/dri /dev/dir --dev-bind /dev/input /dev/input \
--tmpfs /tmp --tmpfs /var/tmp --tmpfs /var/cache --ro-bind /var/cache/fontconfig /var/cache/fontconfig \
--tmpfs $HOME --unshare-all --new-session -- startx /usr/bin/openbox-session
# cannot open /dev/tty0
# debian is weird, startx seems to be non-blocking

sudo chown person:tty /dev/tty2
bwrap --ro-bind / / --dev /dev --proc /proc  --dev-bind /dev/tty2 /dev/tty2 --dev-bind /dev/dri /dev/dir --dev-bind /dev/input /dev/input \
--tmpfs /tmp --tmpfs /var/tmp --tmpfs /var/cache --ro-bind /var/cache/fontconfig /var/cache/fontconfig \
--tmpfs $HOME --unshare-all --new-session -- startx /usr/bin/openbox-session -- vt2
# xf86EnableIOPorts: failed to set IOPL for I/O (Operation not permitted)

@digitalsignalperson

Is nested bwrap not possible (PR landed here https://github.com/containers/bubblewrap/issues/164) or is that a design decision for your setup?

Even if it is possible, the inner bwrap can't get more capabilities than the outer one.  As I mentioned, my labwc sandbox has no network connection.  If I started firefox in a nested bwrap inside that, it wouldn't have a network connection either.
Also I disable unprivileged (actually, all) user namespaces, which makes it impossible to nest because bwrap always sets no-new-privileges, and running bwrap without unprivileged user namespaces needs setuid, which is disabled under no-new-privileges.
And, something to remember about sandboxes.  They prevent things from getting out, not in.  With nested sandboxes, the inner apps are not protected from the more outer ones.  The outer ones can do things like use the magic links in /proc/PID/fd of an inner app.  Or ptrace it (actually, I have Yama turned up so ptrace doesn't work except for root - but if I didn't...!).
I abandoned my alpine setup a while ago.  It was flaky in strange ways.  The tty console always received input even if that input was intended for X11 when I had X11 bwrapped.  I would start X11 in a bwrap and have fluxbox running, then start an xterm in that and type into the xterm.  When I exited X11, I would see that input I typed in the tty console.  Alpine is just not like other nixes.   I'm on Debian 12, and will probably stay there for a while.
I have also been entirely focused on wayland because my experiments with using nested X11 servers (like Xephyr and Xpra) to isolate keyboard/mouse/clipboard/etc from apps was disappointing.  It worked, but was difficult to manage and lost graphics performance.


[NOMERGE] OpenZeppelin Test Environment

Provisional PR - do not merge.
Use OpenZeppelin Test Environment instead of Truffle to run tests.
OpenZeppelin Test Environment has much faster boot up time, and additionally we could replace Mocha with Jest so we can run tests in parallel which should cut down the total time to run tests (which is at around 13 mins today).
However, its code coverage support is still being developed and it depends on a really old fork of ganache-core and solidity-coverage. I believe this is going to be addressed in Q2 of 2020 though. Until then, we will continue to use Truffle.

Closing for now, if we were to switch, we'd probably move to Hardhat.


Infinite loop during installation

After setting up my ci.yaml here - https://github.com/fr3fou/methor/blob/main/.github/workflows/ci.yaml and running the CI, I encounter an infinite loop (?) during the setup phase, causing the CI to time out

https://github.com/fr3fou/methor/runs/1379017245
The output is just non-stop "0"

import { evalExp, expression } from "./math.ts";

async function main() {
  const p = expression();
  while (true) {
    const input = await prompt("> ");
    console.log(evalExp(p(input)[0][0]));
  }
}

async function prompt(message: string) {
  const buf = new Uint8Array(1024);
  await Deno.stdout.write(new TextEncoder().encode(message));
  const n = <number> await Deno.stdin.read(buf);
  return new TextDecoder().decode(buf.subarray(0, n)).trim();
}

main();

This is because your code has an infinite loop.
The process never exits, and finally caused a timeout.
This is not caused by setup-deno.

Thanks! My bad, I was looking at the wrong step


[Storage] Add back support for importing HandleItem

A previous change removed the generated HandleItem from export but to be safe this change adds back support for importing the class but without it showing up in intellisense. This was done by adding it to a __getattr__ in the main file-share __init__ file.
NOTE: This currently breaks API View generation

/azp run python - storage - ci

/azp run python - storage - ci

API change check
APIView has identified API level changes in this PR and created following API reviews.
azure-storage-file-share


Getting last item created

Hi, great job with the library, I got a question:
Is there a way to obtain the inserted data on upsert?, I'm trying to create a library on top of Nano-SQL to make 'models' and 'relations' handling easier, something like this:
Person.create({ name: 'alice', age: 21 }).then(person => {
  // person: Person { id: 1, name: 'alice'... }
  person.
    .pets()
    .create({ name: 'Max', nickname: 'Chew Barka' }).then(pet => {
      // pet: Pet { id: 1, name: 'Max'... }
      console.log('Added ' + pet.name + ' to ' person.name + ' pet list')
    })
})

I would like to accomplish such task with something like this:
class Person {
  static async create (attributes) {
    return new Promise((res, rej) => {
      nSQL('users')
        .query('upsert', attributes)
        .exec()
        .then((result, db) => {
          // Grab the last inserted record and inflate it
          res(result) // [ { msg: '1 row(s) inserted' } ]

          // Ideally something like this should happen:
          // Return an instance of this class with the newly created data
          res(new this(result))
        })
  }
}

The problem I have is that on upsert I get the result mentioned above ([ { msg: '1 row(s) inserted' } ]): and I find it difficult to retrieve the newly created model. I could simply instance the model using the passed 'attributes' but this would skip data created by the database such as the id and defaulted attributes not passed.
Thanks in advice, keep up the good work!

Hey thanks Jesuso, glad you're getting some use out of the lib!
I plan to support databases like SQLite, MySQL and Cassandra in the future so the current internals emulate the processes of those systems where it can.  For example when you perform an insert in MySQL it simply returns that the command succeeded and nothing else.
If this feature is implemented once a MySQL driver is written the feature will have to be removed when using MySQL and other data stores, creating a complicated web of compatibility as more datastores are added.
NanoSQL is intended to be a universal abstraction layer that removes the complexity of the underlying system.  For example the current internals produce identical behavior regardless of if you're using the in memory store, local storage, level DB or indexed DB even though these storage systems all have very different methods.
Something I think that might be doable is a static function called something like "generateRow" that takes the row being inserted and runs the sanitisation / default behavior and returns a copy of the row that will be identical to the original row after it's inserted.  This would really only work if you're not using an auto increment primary key since UUIDs and timeIDs can be generated before they're inserted.
Use case might be something like this:
        function insertRecord(originalRow) {
            return new Promise((res, rej) => {
                const genRow = NanoSQLInstance.generateRow(originalRow, "targetTable");
                nSQL("targetTable").query("upsert", genRow).exec().then(() => {
                    res(genRow);
                }).catch(rej);
            });
        }

Let me know what you think.

Thanks for the response.
I traced back where the promise data was coming from and came to the next lines in your code (see below), and noted an object that only contained the result "msg" was being returned, altho the data I required was availabe right there, so I modified the code a little bit and this works for me. This wouldn't break previous implementations and offers what I need, let me know if this wouldn't work on every database, if it does, maybe it would be a nice feature, to have such data when available.
// db-query.ts:620
private _tableChanged(updatedRowPKs: string[], describe: string, callBack: QueryCallBack): void {
  ...
  t._db._store._readArray(table._name, updatedRowPKs, (rows) => {
    callBack([{msg: updatedRowPKs.length + " row(s) " + describe}], describe, rows, updatedRowPKs);
    // Changed for this:
    callBack([{ msg: updatedRowPKs.length + " row(s) " + describe, rows, updatedRowPKs }], describe, rows, updatedRowPKs);
  });
  ...
}

Thanks.

#16


Implement alternative number encodings

As I proposed on Reddit
This includes two new encoding options

FullGammaEncoding: uses the same algorithm as the default encoding but also applies it for integers.
BoundedGammaEncoding: the optimization I proposed in my comment.

There is a todo in the code to flip the negative numbers so they follow the performance curve of the positive numbers, closer to zero uses fewer bits.
Currently, negative numbers will always use N + 1 bits.
This is part of the reason I made i64 remain with the previous encoding for now.
Please tell me what you think.

Hello!
Thanks for spending time thoroughly investigating a possible improvement to bitcode :heart:
We definitely want to add a way for users to invoke Elias Gamma encoding outside of lengths and enum variants. However, we're skeptical that a crate-wide setting is viable. At the very least, it isn't viable for the protocols in our use-case. Some of our integers are randomly generated, meaning Elias Gamma encoding would be inefficient. Back when we used bincode, the crate-wide scope of number encoding was a severe annoyance for us.
Therefore, our current plan is to allow switching/hinting encodings at the type/field level. Three possible implementations immediately come to mind, two of which would even work within Serde. Ideally it would be a hint as opposed to a guarantee (we don't want to stabilize particular optimizations
P.S.: Don't worry about the failing test in the CI!

However, we're skeptical that a crate-wide setting is viable. At the very least, it isn't viable for the protocols in our use-case. Some of our integers are randomly generated, meaning Elias Gamma encoding would be inefficient.

If those some is less than half of the occurrences, you'd still benefit from the BoundedGammaEncoding. The numbers using less than half the bits would use Elias gamma, and those that use more would still use the normal representation with a single bit in front.
Besides, it is not really crate-wide as I've implemented. You can choose it per-root type. Even more, we can encode the encoding itself in the message so that some messages might use the more aggressive Elias gamma and not others.

Therefore, our current plan is to allow switching/hinting encodings at the type/field level. Three possible implementations immediately come to mind, two of which would even work within Serde. Ideally it would be a hint as opposed to a guarantee (we don't want to stabilize particular optimizations).

I haven't worked a lot with Serde from the data-format implementation side, so I don't know how this would be done. If you could give me some hints (pun intended), I might look into it for this PR, it was really fun to do.

I'll work on optimizing negative numbers first.

I think you're technically correct from a size perspective, but note that anything other than directly copying the integer bits is going to incur a performance penalty on all integers in addition to a size penalty on worst-case integers. It would really help to be selective about which integers are treated as gamma, delta, etc.

Indeed, I'm talking just about smaller size, not faster {en,de}coding. In fact, it is about 20% slower on my machine.

We haven't decided which approach is best and we don't want to send anyone down a rabbit hole of implementation for something we ultimately decide against.

Don't worry. I'm doing this for the learning experience and the challenge of compact encoding. It's been years since I truly worked on bitwise optimizations, so this has been the highlight of my weekend (and it was raining rivers so I could not go out and touch grass).
Although I see that you guys are targeting more of a balance between compact encoding and speed, not just the most compact notation for a single scenario. On that note, and specially considering what you said "anything other than directly copying the integer bits is going to incur a performance penalty", I think an optimization on the direction of speed would be to flag numbers that are uniformly random to be byte aligned in the encoding, which would be even faster to decode at the expense of up to 7 bits.

The approaches I was thinking of were:

struct BitcodeHint<T, const HINT: &str>(T) e.g. BincodeHint<u16, "gamma,delta">(7777). Major downside is that const generic strings are not yet stable.
struct UserNewType(#[bitcode(gamma, delta)] u16). Major downside is that it isn't compatible with serde.
struct UserNewType(#[serde(with = "bitcode::small_int")] u16). Major downside is that we must guarantee the existence of the named module so we wouldn't be comfortable exporting specific encodings like gamma or delta.

I would like the last one the best if it could be a function that returns a tuple of (SerializerFn, DeserializerFn), but sadly it can´t.

Just to re-iterate, we haven't decided if any of these are actually viable (and we may think of a better idea), so please don't spend time PR'ing any of them. If you have an opinion/idea, we're happy to hear it though.

What about doing something similar as serde_with? They added a macro on top of Serde's derive macros like so:
#[serde_as]
#[derive(Deserialize, Serialize)]
struct Foo {
    // Serialize with Display, deserialize with FromStr
    #[serde_as(as = "DisplayFromStr")]
    bar: u8,
}

We could have this:
#[bitcode::optimization] // Name TBD
#[derive(Deserialize, Serialize)]
struct Foo {
    // Optimize for that range with the most likely value becoming the most compact
    #[bitcode(likely_rage=50..=100, most_likely_value=60)]
    bar: u8,
}

And, finally, here's an awful hack that probably works right now: impl {Des, S}erialize for your number new-type to serialize it as a Vec<()> with length equal to the number's value. Lengths are currently optimized with gamma and the ()'s wont take any space at all.

This is horribly amazing in ways I can't explain. I hate that I love it.

We are working on derive macros that will allow something like this:
#[derive(Encode, Decode)]
struct Foo {
    a: u8,
    #[bitcode_hint(bounded_gamma)]
    b: i32,
    #[bitcode_hint(gamma)]
    #[bitcode(with_serde)]
    c: glam::IVec2, // might apply gamma to x and y
}

#[derive(Encode, Decode)]
enum Bar {
    #[bitcode_hint(weight = 100)]
    A, // might serialize as 1
    #[bitcode_hint(weight = 10)]
    B(u8), // might serialize as 00
    // Default weight is 1.
    C(String), // might serialize as 01
}

#[bitcode(with_serde)] will call Serialize or Deserialize.
Any #[bitcode_hint(..)] may result in changes to the output.

With version 0.3.0, we add a set of derive macros for fine-grained customization including with respect to numerical encoding. It would be, for example, possible to implement bounded gamma with a hint like #[bitcode_hint(bounded_gamma)].

Specifically to add a new integer encoding you have to implement Encoding. If you override zigzag to true it will make small negative numbers into small positive numbers. Then to allow #[bitcode_hint(your_encoding)] add it to here and here in the derive macro.

Closing as stale and superseded by #[bitcode_hint(gamma)]; see past comments!


Enable Default PSSCript Rules

Pull Request (PR) description
I have noticed that the default rules for PSScriptAnalyzer were not enabled, although a bunch of them are part of IncludeRules array.
Adding following line in analyzersettings.psd1 would enable them:
IncludeDefaultRules = $true
This Pull Request (PR) fixes the following issues
Task list

[x] Added an entry under the Unreleased section of the change log in the CHANGELOG.md.
Entry should say what was changed, and how that affects users (if applicable).
[ ] Resource documentation added/updated in README.md.
[ ] Resource parameter descriptions added/updated in README.md, schema.mof
and comment-based help.
[ ] Comment-based help added/updated.
[ ] Localization strings added/updated in all localization files as appropriate.
[ ] Examples appropriately added/updated.
[ ] Unit tests added/updated. See DSC Resource Testing Guidelines.
[ ] Integration tests added/updated (where possible). See DSC Resource Testing Guidelines.
[ ] New/changed code adheres to DSC Resource Style Guidelines and Best Practices.

This change is

Codecov Report

Merging #419 into dev will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##              dev     #419   +/-   ##
=======================================
  Coverage   43.33%   43.33%
=======================================
  Files          40       40
  Lines        4938     4938
=======================================
  Hits         2140     2140
  Misses       2798     2798

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 60159b9...28c889e. Read the comment docs.


Raise helpful error when you try to send something that's not an email struct

Going to close this for now. So far I haven't actually had any issues with this


The plugin fluttertoast could not be built due to the issue above.

e: C:\XXX\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\fluttertoast-7.0.2\android\src\main\kotlin\io\github\ponnamkarthik\toast\fluttertoast\FlutterToastPlugin.kt: (4, 28): Unresolved reference: Non
Null
Plugin Version
fluttertoast: ^7.0.2
Flutter Doctor result
Doctor summary (to see all details, run flutter doctor -v):
[√] Flutter (Channel beta, 1.21.0-9.1.pre, on Microsoft Windows [Version 10.0.19041.450], locale en-IN)
[√] Android toolchain - develop for Android devices (Android SDK version 29.0.3)
[√] Android Studio (version 4.0)
[√] Connected device (1 available)
I can run the application from android studio and debug the app. But while i want to release apk it will give me this error.
when  i run flutter build apk it gives me this error.
Thanks.

same error bro

Fixed in 7.1.0


directory structure problem downloading data with ggd-run.sh, can't do a fresh install

Running into the same error with fresh installations on two different systems (one CentOS, other Ubuntu). Installation with --nodata works fine, but fails when trying to get data with:
bcbio_nextgen.py upgrade --cores 12 --data --genomes GRCh37 --aligners bwa --aligners star

End of the error string is:
  File "/home/sdt5z/bcbio/local/share/bcbio/anaconda/lib/python2.7/subprocess.py", line 574, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['bash', '/nv/vol46/uvabx/bcbio/local/share/bcbio/genomes/Hsapiens/GRCh37/txtmp/ggd-run.sh']' returned non-zero exit status 1

When I try running that script by itself:
$ bash ggd-run.sh
--2016-12-02 10:04:45--<EMAIL_ADDRESS>           => `-'
Resolving ftp.broadinstitute.org... <IP_ADDRESS>
Connecting to ftp.broadinstitute.org|<IP_ADDRESS>|:21... connected.
Logging in as gsapubftp-anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /bundle/2.8/b37 ...
No such directory `bundle/2.8/b37'.

gzip: stdin: unexpected end of file

Looks like ftp.broadinstitute.org/bundle/2.8/b37/ doesn't exist - that is, there's no 2.8/, it's just<EMAIL_ADDRESS>When I edit this script to remove the broken path:
#<EMAIL_ADDRESS><EMAIL_ADDRESS>
and re-run it, the data downloads successfully. But, I still can't run the bcbio_nextgen.py upgrade ... command successfully, because it apparently overwrites the file I just edited.
Thanks for the help!

if chapmanb/cloudbiolinux#235 fixes this, there are other .yaml files in chapmanb/cloudbiolinux that'll need updating. hopefully this works.

Glad to see chapmanb/cloudbiolinux#235 was the issue. Once this is merged, would simply running the update script again do the trick? Or will I need to remove some tmp directories? (1st time trying bcbio, thanks).

Stephen;
Thanks again for the report and the fix. It looks like the GATK FTP directory structure changed, causing these recipes to break. I updated all of the remaining recipes and to fix you'd need to do:
rm -rf tmpbcbio-install
bcbio_nextgen.py upgrade --cores 12 --data --genomes GRCh37 --aligners bwa --aligners star

that will remove the cached cloudbiolinux and get an update from GitHub with the fixed recipes. Hope this gets installation working smoothly for you. Let us know if you have any other problems at all.


add welcomepage

I can't merger you request because you don't have gulpfile.js change, I can't complie your scss file to the public folder.

And also the backgroud.jpg, please don't let it in the uploads/avatars folder which let people update their avatar.It's unreasonable


[Chore]: Give workstation access to connect to k8s clusters

Here's what happens when we attempt to login to the acceptance cluster
❯ gcloud container clusters get-credentials --zone europe-west1-b --project cf-on-k8s-wg acceptance
Fetching cluster endpoint and auth data.
ERROR: (gcloud.container.clusters.get-credentials) ResponseError: code=403, message=Request had insufficient authentication scopes.

If you are in a compute engine VM, it is likely that the specified scopes during VM creation are not enough to run this command.
See https://cloud.google.com/compute/docs/access/service-accounts#accesscopesiam for more information about access scopes.
See https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances#changeserviceaccountandscopes for how to update access scopes of the VM.

Let's figure out what access scopes are missing from our workstations and plug the hole.

I think you just need to authenticate with gcloud first.

Works, closing


add configurable request completion

Currently Oryx requires users to buffer the entire contents of request in memory if they use the fetch handler to run their requests.  I'd like to configure the HttpCompletionOption parameter to SendAsync so that certain requests can be 'completed' when their headers arrive (like file transfers), so that I don't have to have that much memory on my machines, and so that my requests are actionable earlier.
Therefore, I've added an option to the HttpRequest record and associated modification functions.  I didn't think this belonged on the context because to me it's a very per-request kind of setting. Some requests are perfectly fine waiting for the whole content.

Thanks, this is really interesting. Didn't know about this option. Found some more info at https://www.stevejgordon.co.uk/using-httpcompletionoption-responseheadersread-to-improve-httpclient-performance-dotnet

FYI: @einarmo


Display satis logs in admin panel

Satis gives great feedback when the indexation of a repository fails, unfortunately the indexation of 'all' repositories fail when this happens.
Appending the output of the Satis build job to any kind of log file makes debugging these kind of errors allot easier, but it still requires you to SSH onto the server where Satis is running to look into the logs.
I'm running a cron every minute to keep things up to date, therefore I don't directly see the Satis output when something goes wrong.
My suggestion is to keep logs by default for all kind of application errors and secondly make them visible in the management interface.
ie. something like control-center/serviced does utilising Elasticsearch and Kibana.
I don't think we need a full stack Elasticsearch environment to keep logs, Satis doesn't report that much. A simple rotating file setup and some kind of frontend parser would suffice. (Maybe something like kassner/log-parser?)
I'm open for suggestions!
EDIT: This would partially depend on #54.

Might be helpful...
https://github.com/EasyCorp/easy-log-handler


{{deps}} moustaches should work in runtime.env: keys

https://github.com/teaxyz/pantry/actions/runs/4581371758/jobs/8090857190
For now, I'll just export this from ca-certs, so git will work.

There's plenty of places where the {{moustaches}} aren't fully hydrated. It mostly depends on what data the function has to hand.
I think here the information about what dependencies a specific package has is lost. However we could just pass all the packages in the fully hydrated dependency graph. It's more than needed but that does include the deps.

Since we don't hydrate multiple versions currently (right? just conflict them out?), all deps should have a unique identity for a given dep.
Possibly we could hydrate this envvar using $SSL_CERT_FILE, but we'd have to know that would populate first...

Since we don't hydrate multiple versions currently (right?)

Yes this is correct. So far we haven't needed more granularity than that.

Possibly we could hydrate this envvar using $SSL_CERT_FILE, but we'd have to know that would populate first...

I added env.FOO to the moustaches recently so that should work. Though I recall humming and harring about whether this was a good idea. So maybe I didn't and only added env.HOME.

It seems versatile, but we should aspire to never need arbitrary env vars. Unless/until we want optional build features; then we're recreating emerge.

indeed. This was my reasoning and you spelling it out makes me think that yeah it's a bad call. Let me go check what I did.

teaxyz/teash

That sounds like the use case I needed. Transitive sounds like you'd be doing it wrong anyway; should be explicit.

my bad, I was misreading the output, let me see why it doesn't work


Question: Custom logger

Hi, I need to log in JSON format so i've created a custom logger but am having some issues
export const gatewayConfig = defineConfig({
  ...
  logging: createJsonLogger("customLogger"),
  ...
});

1. Still some plain log lines
This works as expected but i still get some plain log lines as they don't seem to use the logger interface.
[2024-09-24T12:39:25.492Z] INFO   Loaded config
[2024-09-24T12:39:26.474Z] INFO  Worker graphql-hive/platform#2 Serving supergraph from config
[2024-09-24T12:39:26.477Z] INFO  Worker graphql-hive/platform#1 Serving supergraph from config
[2024-09-24T12:39:26.485Z] INFO  Worker graphql-hive/platform#2 Listening on http://localhost:4000
[2024-09-24T12:39:26.488Z] INFO  Worker graphql-hive/platform#1 Listening on http://localhost:4000

2. Worker thread
I like the worker id (Worker graphql-hive/platform#2 above) and i would like to log it in my custom logger. But worker id is not in the context object AFAICS. Is there a way i can access it in my GatewayPlugin?
gateway v1.0.4

This works as expected but i still get some plain log lines as they don't seem to use the logger interface.

What do you mean by plain log lines? Can you explain it a bit more? What do you expect, and what do you get instead?

Is there a way i can access it in my GatewayPlugin?

It is not part of the runtime. It is CLI that creates workers. We get those IDs from process.worker.id. So you can use the same.

Closing the issue due to the lack of response. Feel free to keep us updated, we can reopen the issue later.


change url address to https address in .gitmodules

Purpose
Currently, due to ssh url specifications in the .gitmodules file, users cannot clone pace without having to set up SSH keys.
Cloning pace should be accessible and the .gitmodules file should be modified with  https urls.
Code changes:
The urls specified in .gitmodules have been changed from ssh to https address
Requirements changes:
N/A
Infrastructure changes:
N/A
Checklist
Before submitting this PR, please make sure:

[x] You have followed the coding standards guidelines established at Code Review Checklist.
[x] Docstrings and type hints are added to new and updated routines, as appropriate
[x] All relevant documentation has been updated or added (e.g. README, CONTRIBUTING docs)
[x] For each public change and fix in pace-util, HISTORY has been updated
[x] Unit tests are added or updated for non-stencil code changes

Additionally, if this PR contains code authored by new contributors:

[x] The names of all the new contributors have been added to CONTRIBUTORS.md

Relates to #63