problem with pnp usb mic?

I get a whole bunch of errors when i try to run visualization.py. here's the copy pasta:
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5047:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM sysdefault
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5047:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM sysdefault
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5047:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM front
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5047:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM surround21
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5047:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM surround21
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5047:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM surround40
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5047:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM surround41
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5047:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM surround50
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5047:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM surround51
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5047:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM surround71
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5047:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM iec958
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5047:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM spdif
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5047:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM spdif
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.hdmi
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.hdmi
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.modem
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.modem
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline
ALSA lib pcm_hw.c:1822:(_snd_pcm_hw_open) Invalid value for card
ALSA lib pcm_hw.c:1822:(_snd_pcm_hw_open) Invalid value for card
ALSA lib pcm_hw.c:1822:(_snd_pcm_hw_open) Invalid value for card
ALSA lib pcm_hw.c:1822:(_snd_pcm_hw_open) Invalid value for card
ALSA lib pcm_hw.c:1822:(_snd_pcm_hw_open) Invalid value for card
ALSA lib pcm_hw.c:1822:(_snd_pcm_hw_open) Invalid value for card
ALSA lib confmisc.c:1281:(snd_func_refer) Unable to find definition 'defaults.bluealsa.device'
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5036:(snd_config_expand) Args evaluate error: No such file or directory
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM bluealsa
ALSA lib confmisc.c:1281:(snd_func_refer) Unable to find definition 'defaults.bluealsa.device'
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5036:(snd_config_expand) Args evaluate error: No such file or directory
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM bluealsa
ALSA lib pcm_oss.c:377:(_snd_pcm_oss_open) Unknown field port
ALSA lib pcm_oss.c:377:(_snd_pcm_oss_open) Unknown field port
ALSA lib pulse.c:243:(pulse_connect) PulseAudio: Unable to connect: Connection refused
ALSA lib pulse.c:243:(pulse_connect) PulseAudio: Unable to connect: Connection refused
ALSA lib pcm_a52.c:823:(_snd_pcm_a52_open) a52 is only for playback
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5047:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM iec958:{AES0 0x6 AES1 0x82 AES2 0x0 AES3 0x2  CARD 0}
ALSA lib pcm_hw.c:1822:(_snd_pcm_hw_open) Invalid value for card
ALSA lib pcm_hw.c:1822:(_snd_pcm_hw_open) Invalid value for card
ALSA lib pcm_hw.c:1822:(_snd_pcm_hw_open) Invalid value for card
ALSA lib pcm_hw.c:1822:(_snd_pcm_hw_open) Invalid value for card
ALSA lib pcm_usb_stream.c:486:(_snd_pcm_usb_stream_open) Invalid type for card
ALSA lib pcm_usb_stream.c:486:(_snd_pcm_usb_stream_open) Invalid type for card
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5047:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM dmix
FPS 42 / 50
FPS 48 / 50
FPS 30 / 50
FPS 39 / 50
FPS 40 / 50

followed some config steps given in the Scott Lawson repo which seems to have removed some errors but I still get several ALSA related errors. copy pasta:
ALSA lib pcm_dmix.c:1108:(snd_pcm_dmix_open) unable to open slave
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.hdmi
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.hdmi
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.modem
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.modem
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline
ALSA lib confmisc.c:1281:(snd_func_refer) Unable to find definition 'defaults.bluealsa.device'
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5036:(snd_config_expand) Args evaluate error: No such file or directory
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM bluealsa
ALSA lib confmisc.c:1281:(snd_func_refer) Unable to find definition 'defaults.bluealsa.device'
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5036:(snd_config_expand) Args evaluate error: No such file or directory
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM bluealsa
ALSA lib pcm_oss.c:377:(_snd_pcm_oss_open) Unknown field port
ALSA lib pcm_oss.c:377:(_snd_pcm_oss_open) Unknown field port
ALSA lib pulse.c:243:(pulse_connect) PulseAudio: Unable to connect: Connection refused
ALSA lib pulse.c:243:(pulse_connect) PulseAudio: Unable to connect: Connection refused
ALSA lib pcm_a52.c:823:(_snd_pcm_a52_open) a52 is only for playback
ALSA lib pcm_usb_stream.c:486:(_snd_pcm_usb_stream_open) Invalid type for card
ALSA lib pcm_usb_stream.c:486:(_snd_pcm_usb_stream_open) Invalid type for card
ALSA lib pcm_dmix.c:1108:(snd_pcm_dmix_open) unable to open slave

All the "unknown" lines are just warnings. The install.py script is supposed to disable those so if you ran that script and it didn't then I might need to look at why. But they're just warnings, they shouldn't affect functionality.
Are you running the script as sudo? The pulseaudio connection error could mean that something is blocking it. Are you running anything else on that Pi? Is this a fresh install or were you already running something on the Pi previously?
It's been a while since I ran this project and I don't currently have a Pi available to test but I'll do my best to give guidance. Just wanted to let you know of my limitations on testing at the moment.

Since I wrote the install script, Python2 packages were deprecated so those package errors are expected because they don't exist anymore. I initially put them in support legacy installations. I need to remove them at some point so you can ignore those errors for now.
As for the fortran compiler, try installing it manually:
sudo apt install gfortran -y
Then install scipy again:
sudo pip3 install scipy==1.4.1
One note about scipy is that when I wrote the installer, latest scipy version was 1.5.x and it didn't work with a lot of other stuff. So the installer has version 1.4.1 hard-coded. You could try removing the version and trying with the latest version. If you leave it blank it should auto install version 1.8.1.

@naztronaut ok cool. I have done this. should I rerun install.py after I complete these steps? or can I go straight to running visualization.py? Previously when I had tried running visualization.py (with scroll) I would get (seemingly) some numpy errors:
RuntimeError: module compiled against API version 0xf but this version of numpy is 0xd
Traceback (most recent call last):
  File "/home/pi/dancyPi-audio-reactive-led/python/visualization.py", line 5, in <module>
    from scipy.ndimage.filters import gaussian_filter1d
  File "/usr/local/lib/python3.9/dist-packages/scipy/ndimage/__init__.py", line 151, in <module>
    from ._filters import *  # noqa: F401 F403
  File "/usr/local/lib/python3.9/dist-packages/scipy/ndimage/_filters.py", line 37, in <module>
    from . import _nd_image
ImportError: numpy.core.multiarray failed to import

This time when i ran sudo pip3 install scipy==1.4.1 i get some more errors:

      gfortran:f77: scipy/sparse/linalg/eigen/arpack/ARPACK/UTIL/smout.f
      gfortran:f77: scipy/sparse/linalg/eigen/arpack/ARPACK/UTIL/cvout.f
      gfortran:f77: scipy/sparse/linalg/eigen/arpack/ARPACK/UTIL/dmout.f
      gfortran:f77: scipy/sparse/linalg/eigen/arpack/ARPACK/UTIL/svout.f
      gfortran:f77: scipy/sparse/linalg/eigen/arpack/ARPACK/UTIL/zvout.f
      gfortran:f77: /tmp/pip-install-ot5hrja8/scipy_c1012e98dccf485eb9c10d6a1056d701/scipy/_build_utils/src/wrap_dummy_g77_abi.f
      gfortran:f77: scipy/sparse/linalg/eigen/arpack/ARPACK/UTIL/dvout.f
      gfortran:f77: scipy/sparse/linalg/eigen/arpack/ARPACK/UTIL/icnteq.f
      gfortran:f77: scipy/sparse/linalg/eigen/arpack/ARPACK/UTIL/icopy.f
      error: Command "/usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops -Iscipy/sparse/linalg/eigen/arpack/ARPACK/SRC -I/tmp/pip-build-env-1vmaqd5m/overlay/lib/python3.9/site-packages/numpy/core/include -c -c scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/dsaitr.f -o build/temp.linux-armv7l-cpython-39/scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/dsaitr.o" failed with exit status 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for scipy
Failed to build scipy
ERROR: Could not build wheels for scipy, which is required to install pyproject.toml-based projects

This failure follows a whole bunch of warnings and a few errors which tend to look like this:
Error: Rank mismatch between actual argument at (1) and actual argument at (2) (scalar and rank-1)
      scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/znaitr.f:737:39:

        383 |             call dvout (logfil, 1, rnorm, ndigit,
            |                                   2
      ......
        737 |                 call dvout (logfil, 2, rtemp, ndigit,
            |                                       1
      Error: Rank mismatch between actual argument at (1) and actual argument at (2) (scalar and rank-1)
      gfortran:f77: scipy/sparse/linalg/eigen/arpack/ARPACK/UTIL/iset.f
      stat.h:8:27:

      Warning: Unused variable ‘t2’ declared at (1) [-Wunused-variable]
      stat.h:8:31:

      Warning: Unused variable ‘t3’ declared at (1) [-Wunused-variable]
      stat.h:8:35:

You don't need to run  install.py again, the rest of the things were installed so they should be fine.
Looks like Scipy is still not installing correctly. Strange. That's probably what's causing the Numpy errors too. You could try upgrading pip and reinstalling scipy:
sudo pip3 install --upgrade pip
sudo pip3 uninstall scipy
sudo pip3 install scipy
The last command will install the latest version of scipy, hopefully the errors I ran into on 1.5.x is resolved.
Not sure why it keeps failing on scipy. If you keep running into issues with installing it, try other versions, you can find a list of the releases here: https://github.com/scipy/scipy/releases - the latest is 1.8.1 so if that doesn't work, 1.8.0 and go backwards a few versions. If they all fail, then it might be something with fortran which I know very little about.

Ok, thanks for the help. I have tried installing older versions of scipy which seem to fail if the version tries to build a whl file... this starts when I go backwards in versions at 1.4.1. I can install the latest version but that gives me the problems with numpy. not sure what's going on. I might try and get in touch with the folks that develop scipy. Thanks for the help though!

@naztronaut I got it working! I am using scipy 1.8.1 and I guess that installs numpy of a certain version? I simply installed the latest version of numpy (1.22.4) and the LEDs sprang to life! thanks for the help. u tha best!

Awesome! Congrats on getting it working. Have fun with it :)


Clientside Callback (JS) appears to result in different behavior from seemingly equivalent Dash/Server Callback (Python)

Python 3.11.4, Dash 2.14.1, Dash-Ag-Grid 2.4.0
I'm posting here as I cannot seem to reproduce this behavior with a more or less equivalent set-up using dash-core-components so my initial thought is it might be related to ag-grid somehow.
The basic gist is this: I have a dcc.Store to handle my state and am pulling from it to update my grid's rowData. In my current example, I'm using a callback (server side) attached to cellValueChanged to update my state, then using the state to update rowData. [it's a bit circular for this trivial example, but in my actual use case I'm trying to draw/patch all my tables from a single source of truth in state where some computations might need to happen on cellValueChange before sending new data to the tables]. I'm experimenting with the update to rowData happening as either a client side callback or a server side one. However, i've stumbled into a situation where it seems as though they behave differently and in an unexpected fashion.
Now, please bear with me as it's a slightly weird scenario, but essentially, if I end up calling PreventUpdate from my cellValueChanged callback, when my rowData update callback is on the client, I still see updates to my state/localStorage (unexpected) whereas when it lives on the server, i do not (as expected).
The reason I felt this was worth to create an issue is that, with the only route to update state/localStorage being blocked by a PreventUpdate, I found it strange that the existence of this clientside callback let the state/localStorage still update.
Hopefully some code will help:
Here's a minimal example.
from dash import (
    dcc,
    html,
    Dash,
    callback,
    Input,
    Output,
    State,
    clientside_callback,
    no_update,
)
from dash.exceptions import PreventUpdate
import dash_ag_grid as dag

app = Dash(__name__)

columnDefs = [
    {
        "field": "direction",
        "editable": True,
    },
    {
        "field": "color",
        "editable": True,
    },
]

layout = html.Div(
    [
        html.Div(
            [
                dag.AgGrid(
                    id="grid",
                    rowData=[],
                    columnDefs=columnDefs,
                    style={"height": "200px"},
                )
            ]
        ),
        html.Div(id="output", children=["output here"]),
        dcc.Store(
            "state",
            data=[{"direction": "North", "color": "orange"}],
            storage_type="local",
        ),
        dcc.Checklist(id="prevent-toggle", options=["Prevent State Update"]),
    ]
)

app.layout = layout

@callback(
    output=Output(
        "state",
        "data",
    ),
    inputs=[Input("grid", "cellValueChanged"), State("prevent-toggle", "value")],
    prevent_initial_call=True,
)
def on_grid_cell_change(change_data, do_prevent_update):
    if do_prevent_update:
        print(f"Preventing Update Of: {change_data}")
        raise PreventUpdate
        # return no_update
    else:
        print(f"Updating:  {change_data}")
        return [change_data["data"]]

# keep this callback in to see expected behavior
# @callback(
#     output=Output("grid", "rowData"),
#     inputs=Input("state", "data"),
# )
# def add_rows_to_grid(row_data):
#     return row_data

# keep this callback in to see the error (when checking preventUpdate)
clientside_callback(
    """
    function addRowsToGrid(row_data) {
        console.log(row_data)
        return Object.values(row_data);
    }
    """,
    Output("grid", "rowData"),
    Input("state", "data"),
)

clientside_callback(
    """
    function printState(row_data) {
        return `STATE: ${JSON.stringify(row_data)}`;
    }
    """,
    Output("output", "children"),
    Input("state", "data"),
)

if __name__ == "__main__":
    app.run(debug=True)

This first screen cap is the "unexpected" behavior using the clientside callback. You'll see how making changes in the table updates the Local Storage, as well as a separate clientside callback printing the storage values to a screen. However, when I enable preventUpdate, you'll see Local Storage still updates, however the clientside callback printing to screen doesn't. This just seems very odd to me, as 1) i expect that state should not be updated here and 2) since it is anyway, why is only one of the callbacks "seeing" it?
https://github.com/plotly/dash-ag-grid/assets/19380579/846f3953-46b6-48c0-9a9a-cf03e9cf6245
This second screen cap is the "expected" behavior using the serverside callback - the state does not update when preventUpdate is in effect.
https://github.com/plotly/dash-ag-grid/assets/19380579/f69a5249-a00a-4660-a7ae-90fbd2f114c9
Hopefully enough of that is ... somewhat clear?

Hello @jbeyer16 ,
I think this may be due to how you are setting the rowData with the store. In this fashion, it looks like you have actually linked them to be the same.
Try using JSON.parse(JSON.stringify(data)) as what you are returning to the rowData.

@BSd3v ooooooh great observation. thanks for pointing that out. indeed it seems they are referencing the same object, creating a copy as you suggest seems to now work as expected. Thanks for the second set of eyes there.
Indeed in my actual use case I have getRowId set as it did appear as the better way to go about things!
I appreciate the help here!

You may have accidentally found a way to solve the issue for rowData persistence. XD


Remove .xcodesamplecode.plist, which prevents editing markdown from Xcode

This is kind of cool, but it prevents editing the markdown from Xcode (but not any other text editor).
Checklist

[ ] I've added at least one test that validates that my change is working, if appropriate
[ ] I've followed the code style of the rest of the project
[x] I've read the Contribution Guidelines
[ ] I've updated the documentation if necessary

@swift-ci Please test

@swift-ci please test


About data

Thanks for release you code.  Could you please release your data?

Hi here is the link for related datasets.


Separate frontend from server and harden/clean up server code

@nihonjinrxs This is looking great, what's left to be done in your mind before this moves out of draft? I'm looking to make some improvements on the backend with the data loading flow and it would probably make sense to start on top of this work. Should we go ahead and merge it?

Give me a bit to take a look. I'll let you know before tomorrow how this is best to handle, and we can go from there.


fix sonatype integration, failed with changes in sonatype migration

fixed by following guide for account migration:
"To configure a publisher’s plugin authentication you would need to update your plugin settings to use a user token instead of the Nexus UI username and password login.


configure which files to gzip, defaulting to js, css & svg

thx!

@LevelbossMike @duizendnegen
This introduces an issue with the ember-deploy-s3 extension. It is hardcoded within the uploader to expect js or css extensions. Won't have time to make a fork for a few days, but wanted to document that is happening in case anyone else is wondering why their svg's aren't displaying in browsers.

@tylr
thanks for the feedback. I can recommend you to update your deploy.json to include
"gzipExtensions": ["js", "css"]

@tylr
Thx for catching this issue.
I just fixed this in ember-deploy-s3 and published version 0.0.3 on npm.
ember-deploy-s3 should now mark the files gzipped by ember-deploy as gzip content on s3. If you don't pass a gzipExtensions property in deploy.json the fallback is to use ember-deploy's default ['js', 'css', 'svg'].
Sorry for the inconvenience caused. Please report back any issues you are encountering.


node_modules not included in site deploy

The npm install runs successfully during the build, however, when deploying the site the folder is not there.
What am I missing?

Why do you need the node_modules folder in your deployment? There should be only development dependencies in there.

We have some production dependencies.  Scripts that run on request.

My guess ist that you need to add the node_modules folder to your VS Project, but without any additional informations about your setup I cannot really help.
Also, having node modules as a production dependecy is more of an edge case I guess.


User Profile: update username 'once' as part of registration

The User api now returns a read-only property called "registration_completed" which is  false when the user first logs in.
"registration_completed"=true implies that the username can be updated.
"registration_completed"=false implies that the username cannot be updated.
username can be updated only once in the registration workflow. On doing so, registration_completed is automatically set to true internally. Subsequent updates to username is not allowed.
During (HTTP PATCH) updation , if the username already exists, then a HTTP 409 Conflict is returned.
"write" operation on registration_completed  by HTTP PATCH is not supported.

@pmuir

I saw the newly added registration_completed attribute and we're already using it in the getting started page. However, I added the check for the 409 status.
That said, I was able to update using my existing username without any errors. I'm not able to change my username to something different now that an update has taken place. For testing purposes, it would be helpful to update the username more than once.
Fixed by:
https://github.com/fabric8io/fabric8-ui/pull/813

:tada: This issue has been resolved in version 4.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Chart save pop-up now automatically goes to song data directory

amazing

can yoy make this toggleable please?


[camera] Add camera test module map

Create a test header for https://github.com/flutter/plugins/pull/4426.
Create camera.Test explicit clang module. Include new modulemap and umbrella header to support this.

Closing, this is included in https://github.com/flutter/plugins/pull/4426.


Implemented masternodebroadcast

This set of rpc commands provide an ability to create masternode broadcast messages offline and relay them from online node later (messages expire in ~1 hour).
Available commands:

create-alias  - Create single remote masternode broadcast message by assigned alias configured in masternode.conf
create-all    - Create remote masternode broadcast messages for all masternodes configured in masternode.conf
decode        - Decode masternode broadcast message
relay         - Relay masternode broadcast message to the network

Merged, thanks!

you guys are hardcore :)

@schinzelh Can we start voting on these before merging? I wasn't done with the code review and I want to be much more careful about merging after 12.1

@evan82 agree on PR's voting but I would also like to see no more direct commit pushing (when there is no PR at all) even in dev branches :stuck_out_tongue:

Let's move discussion to #744 :smile:

@evan82 sorry, my bad - i was in a merging frenzy :-)
I agree on voting on PRs before merging, and agree on @UdjinM6 that we should not directly commit to upstream without PR - it makes review of codes changes much easier IMHO.


WIP Container events2

Mainly want to see how badly drone blows up.

closing in favor of a new branch


Drop unennecessary assignments from usage example

Keyword arguments work with some_call(foo: "value", bar: 123) and currently only for default arguments. This just creates a couple of unused local variables and works because the value of an assignment expression is its right hand side.

Thanks!


Add jakirkham as a maintainer

No need to wait for the CIs

Thanks.

Added one for autoconf too.


By default PulsarSource consume encrypted message

Motivation
Right now, default CryptoFailureAction configuration is FAIL in Consumer which retries to consume message which is not applicable for pulsar-function because pulsar-function will never have CryptoReader configured. so, function should consume encrypted message and forward to sink along with EncryptionContext.

retest this please


Auto apply change

Hi, thanks for this tool.
Is there a way to apply all changes without validation ? That way the tool could be run as a cron or a travis job easily.

Hi @ovv, noted, it is a fair request. At the moment you can use yes  via yes | dothub to accept all prompts if working on unix.


CloudWatch Events: add SQS Event Target

Duplicate of https://github.com/awslabs/aws-cdk/issues/1786


Is filter size depends on scale factor?

I wonder how filter size is depending on scale factor?
Seems filter size = factor*4 can you elaborate on this?
https://github.com/trevor-m/tensorflow-bicubic-downsample/blob/master/bicubic_downsample.py#L13

If you print the kernel after https://github.com/trevor-m/tensorflow-bicubic-downsample/blob/master/bicubic_downsample.py#L18 you can see that anything beyond a filter size of 8 for factor =2 will just have zeros, so there is no point in having a larger filter.
Filter size = 8: [-0.01171875 -0.03515625  0.11328125  0.43359375  0.43359375  0.11328125  -0.03515625 -0.01171875] Filter size = 16: [ 0.          0.          0.          0.          0.          0.
0.          0.          0.          0.          0.          0.
-0.01171875 -0.03515625  0.11328125  0.43359375  0.43359375  0.11328125
-0.03515625 -0.01171875  0.          0.          0.          0.
0.          0.          0.          0.          0.          0.
0.          0.        ]`
The same phenomenon happens with factor=4 when comparing filter size 16 (44) with larger filters.
I'm sure there is a better mathematical explanation for this, but I found the factor4 relationship to work for these common cases.


Helper for rendering  from Slayer::Form

It would be awesome if we could simple_form with Slayer::Form somehow.

We can:

class PostForm < Slayer::Form
  attribute :title, String
  attribute :body, String
end

<% form = PostForm.new(title: "Foo") %>
<%= simple_form_for(form, as: :post, url: post_path) do |f| %>
  <%= f.input :title %>
  <%= f.input :body, as: :text %>
  <%= f.button :submit %>
<% end %>

I meant more like:
<%= simple_form_for(form, as: :post, url: post_path) do |f| %>
  <%= form.render_fields(f) %>
<% end %>


plugins not recognized when pip isn't installed

Environment information

OS: linux
Python Version: Python 3.11.8
ape and plugin versions:
ape 0.7.13
ape-alchemy 0.7.1

$ ape --version
0.7.13

$ ape plugins list
  alchemy      0.7.1.dev11+gad25ef2.d20240323
  etherscan    0.7.2

Contents of your ape-config.yaml (NOTE: do not post anything private like RPC urls or secrets!):

$ cat ape-config.yaml
cat: ape-config.yaml: No such file or directory

What went wrong?
couldn't use alchemy in my script. upon further inspection, it wasn't showing up in plugins or networks. i realized this is because of my non-standard installation with uv. since I don't have pip installed, it fails trying to query which plugins are installed.
here are some terminal printouts:
>> ape networks list
ethereum  (default)
├── goerli
│   └── geth  (default)
├── local  (default)
│   ├── geth
│   └── test  (default)
├── mainnet
│   └── geth  (default)
└── sepolia
    └── geth  (default)
>> uv pip install pip
Resolved 1 package in 25ms
Installed 1 package in 104ms
 + pip==24.0
>> ape networks list
ethereum  (default)
├── goerli
│   ├── alchemy
│   └── geth  (default)
├── local  (default)
│   ├── geth
│   └── test  (default)
├── mainnet
│   ├── alchemy
│   └── geth  (default)
└── sepolia
    ├── alchemy
    └── geth  (default)

trying to use a script that uses alchemy without pip installed fails:
>> uv pip uninstall pip
Uninstalled 1 package in 29ms
 - pip==24.0
>> python steth.py
Traceback (most recent call last):
  File "/code/testape/steth.py", line 19, in <module>
    context = networks.parse_network_choice('ethereum:mainnet:alchemy')
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/testape/.venv/lib/python3.11/site-packages/ape/managers/networks.py", line 528, in parse_network_choice
    provider = self.get_provider_from_choice(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/testape/.venv/lib/python3.11/site-packages/ape/managers/networks.py", line 488, in get_provider_from_choice
    return network.get_provider(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/code/testape/.venv/lib/python3.11/site-packages/ape/api/networks.py", line 1042, in get_provider
    raise ProviderNotFoundError(
ape.exceptions.ProviderNotFoundError: No provider named 'alchemy' in network 'mainnet' in ecosystem 'ethereum'. Options:
geth

How can it be fixed?
there's probably a more generic way to detect installed packages than going through pip. but it may be messy.

@antazoey was experimenting a bit with using importlib for plugin detection, but yeah I don't think the install portion of ape will work without pip (but maybe can just detect uv and use uv pip instead in that case)

Yeah, can you try on main branch? I think you should be able to at least list the plugins now as of https://github.com/ApeWorX/ape/pull/1996

yeah, I'm thinking if at least you can use ape and just ape plugins install is broken, that's probably not as huge a deal for @wakamex since he should know how to install plugins himself manually via uv

so a re-install that just points to my local ape-alchemy repo (that i'm still working on!) and installs the latest requirements:
>> uv venv .venv -p 3.10
>> source .venv/bin/activate
>> uv pip install -e ../ape-alchemy
>> ape plugins list
Traceback (most recent call last):
  File "/code/testape2/.venv/bin/ape", line 5, in <module>
    from ape._cli import cli
  File "/code/testape2/.venv/lib/python3.10/site-packages/ape/__init__.py", line 10, in <module>
    from ape.managers.project import ProjectManager as Project
  File "/code/testape2/.venv/lib/python3.10/site-packages/ape/managers/__init__.py", line 23, in <module>
    ManagerAccessMixin.config_manager = ConfigManager(
  File "/code/testape2/.venv/lib/python3.10/site-packages/pydantic/main.py", line 164, in __init__
    __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__)
  File "/code/testape2/.venv/lib/python3.10/site-packages/ape/managers/config.py", line 134, in load_configs
    return cm.load()
  File "/code/testape2/.venv/lib/python3.10/site-packages/ape/managers/config.py", line 277, in load
    _ = self._plugin_configs
  File "/code/testape2/.venv/lib/python3.10/site-packages/ape/managers/config.py", line 205, in _plugin_configs
    valid_ecosystems = dict(self.plugin_manager.ecosystems)
  File "/code/testape2/.venv/lib/python3.10/site-packages/ape/plugins/__init__.py", line 138, in __getattr__
    self._register_plugins()
  File "/code/testape2/.venv/lib/python3.10/site-packages/ape/plugins/__init__.py", line 193, in _register_plugins
    module = importlib.import_module(module_name)
  File "/usr/lib64/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/code/testape2/.venv/lib/python3.10/site-packages/ape_test/__init__.py", line 10, in <module>
    from ape_test.provider import EthTesterProviderConfig, LocalProvider
  File "/code/testape2/.venv/lib/python3.10/site-packages/ape_test/provider.py", line 7, in <module>
    from eth.exceptions import HeaderNotFound
  File "/code/testape2/.venv/lib/python3.10/site-packages/eth/__init__.py", line 1, in <module>
    import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'

installing setuptools solves this error:
>> uv pip install setuptools
Resolved 1 package in 1ms
Installed 1 package in 32ms
 + setuptools==69.2.0
>> ape plugins list
Installed Plugins
  alchemy    0.7.1.dev11+gad25ef2.d20240411

add setuptools as a dependency? if you add pip as a dependency then uv should install it...


Points at infinity.

For milagro-crypto the points at infinity over FP and FP2 are:
https://github.com/status-im/nim-milagro-crypto/blob/290f927865f9e575920dca5f415c58b554dbe92e/src/milagro_crypto/generated/ecp_BLS381.c#L165-L178
https://github.com/status-im/nim-milagro-crypto/blob/290f927865f9e575920dca5f415c58b554dbe92e/src/milagro_crypto/generated/ecp2_BLS381.c#L41-L49
So for FP:

Edwards curve: infinite at (0, 1, 1)
Montgomery curve: infinite at (0, 0, 0)
Weierstrass curve: infinite at (0, 1, 0)

and FP2: infinite at (0, 1, 0)
The research implementation of BLS uses Z1 and Z2 infinites defined in Py-Ecc:
https://github.com/ethereum/py_ecc/blob/master/py_ecc/optimized_bn128/optimized_curve.py#L39-L42
# Point at infinity over FQ
Z1 = (FQ.one(), FQ.one(), FQ.zero())
# Point at infinity for twisted curve over FQ2
Z2 = (FQ2.one(), FQ2.one(), FQ2.zero())

https://github.com/ethereum/beacon_chain/blob/eea52999a578fbd29751330a6f2bb27e60c67f7f/beacon_chain/utils/bls.py
from py_ecc.optimized_bn128 import (  # NOQA
    G1,
    G2,
    Z1,
    Z2,

...

def aggregate_sigs(sigs):
    o = Z2
    for s in sigs:
        o = add(o, decompress_G2(s))
    return compress_G2(o)

def aggregate_pubs(pubs):
    o = Z1
    for p in pubs:
        o = add(o, decompress_G1(p))
    return compress_G1(o)

So I'm not sure what curve type Py-ECC is using.
Furthermore BN128 is different from BLS12-381, at best it's a naming problem, otherwise we can't use the proof of concept test_bls as reference

Issue solved, basically the beacon chain is mocking BLS12-381 use with BN128 which was already implemented in Python as it is needed for a precompile.
See Sharding call #0 on Vitalik's answers: https://notes.ethereum.org/s/r1wW6TZrQ#BLS-Signatures


Taxon constraint: GO:0035658 | Mon1-Ccz1 complex

Please provide as much information as you can:
GO:0035658 | Mon1-Ccz1 complex
never in Schizosaccharomyces

Supporting evidence if available (e.g PMID):

see https://github.com/geneontology/go-annotation/issues/4266

This one was done a while back, forgot to close


Information is unnecessary and misleading

In "By using the steps for invoking GraphTopologyList, you can invoke GraphTopologySet to set a graph topology. "
it confuses the user. Mentioning GraphTopologyList is misleading. Don't mention the previous method. Just say "like we did before you can invoke...".

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 8c6611e1-23a3-48b0-d7e9-b7034bc3be9e
Version Independent ID: b05e4e17-70e7-59eb-3c75-d54efea6aa75
Content: Get started with Live Video Analytics on IoT Edge - Azure - Live Video Analytics on IoT Edge
Content Source: articles/media-services/live-video-analytics-edge/get-started-detect-motion-emit-events-quickstart.md
Service: media-services
Sub-service: lva-edge
GitHub Login: @Juliako
Microsoft Alias: juliako

@tiagonmas Thanks a lot for bringing this to our notice, our team will review the feedback and update the document accordingly.

@tiagonmas Thanks again for the feedback! I have assigned the issue to the content author to review further and update the document as appropriate.

changes are in. This will be updated by tomorrow. Good to close this issue.

changes are in. This will be updated by tomorrow. Good to close this issue.


Implement the transaction() contextmanager on deleteMessage

This is a proposal to #9

Thank you for your contributions.


LEXI-791: Fix colors not displaying in Lexicon site

See LEXI-791:

Review existing colors
Split dark and secondary sections
Add new colors (Dark D1 & D2)

Currently, colors variables and constants are not well named (e.g. main). This should be change later.

Thanks @drakonux !


clang doesn't print colored diagnostics when invoked from ninja

The reason for this is that ninja sets subprocess stdout/stderr to a pipe (Subprocess::Start(), subprocess.cc), and clang checks if StandardErrHasColors() (tools/clang/lib/Driver/Tools.cpp), which is false if !isatty(2) (lib/Support/Unix/Process.inc).
I looked around a bit, and the way to do this seems to be to call fork_pty() to run the child in a pseudo terminal. I don't know if this would impact subprocess creation time, and if opening ~4000 pseudo ttys (a chrome build at -j10000) is considered good form.
(It's possible to force clang to always emit color escaped codes using "-Xclang -fcolor-diagnostics", but that's pretty hacky. make doesn't seem to override stderr on unix as far as I can tell, child_execute_job() in job.c)

You have summarized everything (the problem, workaround, and possible problem with fix) exactly as I think about it.  (I have done the forkpty thing in another project.
I kinda wonder if we should make gyp know to add this flag when using clang.  (That is, by virtue of the fact that it's a specific command it belongs in the generator program to know what to do.)
One other thing that makes me nervous about the magic-pty-fix option is that if a program shells out to an editor (e.g. something runs "svn up" for some reason and then that hits a conflict), we don't want the subprocess to think that they're getting a real terminal.

Hm, I'm not sure if gyp is in a good position to do this. It sounds like these flags would need to be passed at build time if isatty(stderr) in the ninja process (else the control characters would appear on build bots, too); gyp time is too early.
For now, I'm running ninja without the dup2(pipe, 2) locally. That's not ideal (some missing newlines on warning output), but it works better than the current behavior for me.

That's a good point about gyp-time vs run-time.  Hmm.  I wonder if Ninja could expose some $isatty variable at runtime.  Feels slippery.
(FWIW, the reason for the pipe is that otherwise, output from parallel commands can be interleaved.)

Make doesn't have the pipe, and I can't remember ever being annoyed by interleaved commands. Another reason for the pipe is to print a newline after the "progress line" before writing stderr tho, and I can't think of a good way to make that work without writing a newline after every command. Now that I think about this, I think blaze manages to print colored diagnostics from clang while not printing every command. (time passes) I talked to someone, blaze just passes -fcolor-diagnostics, lame.
So, to sum up:

pseudo-ttys might not be fast enough, but it's not even worth trying because that could also confuse some commands like svn
not dup'ing 2 means warnings are printed right after the progress bar, which is pretty ugly
gyp can't force colors

I think I've heard people asking for ninja to respect $CC, and you said "it's just nobody has implemented this yet". To support this, ninja would have to grow the concept of "compiler" anyway. If it did that, maybe checking for cc == clang and passing in 'force colors' isn't so bad? But now that I wrote this, ninja could just have the general concept of "try to get env var FOO from the env" in its build rules to get CC support, and then it wouldn't have to know about colors.)
With env support, I could set CFLAGS=-fcolor-diagnostics…or clang could check for env("SOME_PARENT_IN_TTY") and I could set that…that'd be workable, but it wouldn't Just Work.
All in all, I probably like your $isatty suggestion best, if we could agree on a name that the clang guys would be comfortable supporting. Then all kinds of build systems could use that if they wanted. (In theory, it could build break steps that grep for warnings, so maybe there would have to be a flag to disable setting this variable…but build steps like that are questionable anyhow.)

IMHO, without having pipe won't scale well for muti-core parallel compilation. Currently, we have 12 thread processor and we need 16 or more parallel jobs to hide I/O latencies.. In the situation like this, commands output might get intermixed and breaks ANSI sequences.
Having $isatty might introduce another complexity to Ninja and would cause hard-to-track problem like "My job won't complete when it invoked from cron-job!". So i feel Ninja shouldn't have multiple ways to invoke a command and current one; invoke program and buffer its output via pipe is good for multi-core era.
PS.
Fortunately, clang has machine-readable diagnostics as well as TTY-based diags. So I think someone could implement "external colorizer for Ninja log" for clang. Integrating to emacs or other notification system like Growl is a plus :)

Sounds like there's no good solution here.
For chrome, I'll try adding a CLANG_FORCE_COLOR_DIAGNOSTICS gyp define that defaults to 1 and is set to 0 on the bots (which adds -fcolor-diagnostics).

I'd also be ok with extending Ninja to obey environment variables.
I.e. make it so "$ENV_foobar" expands to $foobar from the environment.
(All the other machinery is already in place such that Ninja will rebuild the world if you change such a variable between builds.)

For chrome, colors need to Just Work. Env vars don't really help with that, so let's WontFix this issue.
(I ran a small test, and my system – OS X 10.7 – runs out of pseudo ttys after allocating 125.)

Wrong button, sorry.

We noticed that forcing colors on for clang confuses programs that parse its output, such as emacs and vim. The current plan is to let ninja strip ansi escapes from subprocesses if it's not writing to a terminal. I'll prepare a patch for that.

That landed in https://github.com/martine/ninja/commit/2ef3a546476987d6565141337c6309e7ae60f9fd

Many years later, I'm confused about how this is supposed to work... because it's not working on my configuration. My CMake compile options are like so:
set_property(DIRECTORY APPEND PROPERTY COMPILE_OPTIONS $<$<COMPILE_LANGUAGE:C>:-Wimplicit-int -Wimplicit-function-declaration -Wno-gnu-zero-variadic-macro-arguments -std=c11>  -Xclang -fcolor-diagnostics)
No colored output is available unfortunately (MacOS, iTerm2 with zsh). And I've checked the output, the flags do get added to the compile options at runtime. If I switch Ninja with Make, all is well.

Try -fdiagnostics-color. Are you running ninja in a terminal?

Sorry, I didn't specify I'm using clang. That flag is for gcc.

So as I understand it, Clang detects if it is outputting to a terminal and enables colours automatically in that case. Ninja captures Clang's output so Clang things it isn't outputting to a terminal.
So you have to force Clang to output colours with -fdiagnostics-color=always (or -fcolor-diagnostics for GCC - yeay consistency). But then that means that when Ninja is outputting to something that isn't a terminal (e.g. Emacs) that program gets confused by the colour codes.
And your "solution" was to have Ninja strip the colour codes if it is not outputting to a terminal.
So Ninja knows if it is meant to output colours. It can "tell" Clang whether it should output colours (by pretending to be a terminal). But instead it throws that information away and forces the user to set -fdiagnostics-color manually?
That seems like a bad solution. Why can't Ninja just do something like this:
if (output_is_terminal) {
   use_pseudo_terminal = true;
}

Clang uses istty() on Unix and GetConsoleMode() on Windows to determine if the output is a console.
So you can just create the subprocess in a pseudo-terminal instead (openpty() on Linux, not sure about Windows).
The only downside I can see is that you can't get stdout and stderr separately from openpty().  There is a question about doing that on StackOverflow but the only solution uses a dirty LD_PRELOAD hack. However it seems like Ninja doesn't distinguish between them anyway.

I did find this mentioned in the wiki:

Ninja could use pseudo ttys to work around this further, but different systems have different limits on the number of ptys available. I also worry that commands like svn up would assume they can interact with the user.

Yeah, using a pseudo tty is overkill when the only thing you want is color codes. See https://bugs.llvm.org/show_bug.cgi?id=23609 and https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80271 for a possible solution.

For CMake users, CMake 3.24 (which looks like it releases on June 15th?) will have CMAKE_COLOR_DIAGNOSTICS. Here's their issue thread where the feature was discussed.
Here's a link to a comment in the thread by Craig about how the feature might be best used due to some IDEs not supporting ansi escape coloring- the environment variable allows per-user config.

Hi there,
Even with the latest CMake and ninja release I don't get colorised output when building with -DCMAKE_COLOR_DIAGNOSTICS=On.
$ cmake --version
cmake version 3.30.3

CMake suite maintained and supported by Kitware (kitware.com/cmake).
$ ninja --version
1.12.1

$ gcc --version
gcc (GCC) 14.2.1 20240805
Copyright (C) 2024 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

If I set the CLICOLOR_FORCE=1  in the environment running the CMake build then I do get output. Just sharing for anyone else that stumbles on this.
FYI: This doesn't cause CMake itself to colorize the output like it does in Make. So the [x%] build target outputs won't be colorised. But the build-tool/compiler output will be.

In what kind of terminal simulator are you running ninja?

Tmux within suckless terminal. I'm also using a custom terminfo definition that adapts the combines the sequences supported by tmux and st. To be clear color does work when I run CMake/Ninja into a tty but if I put a pager in front of it like cat then color is suppressed unless I add that CLICOLOR_FORCE option.
Example for with and without:

In both cases the -fdiagnostics-color=always option is passed to gcc. I'd wager ninja is probably stripping it out but I haven't investigated that far yet :thinking:.

Yes, see https://github.com/ninja-build/ninja/wiki/FAQ#why-does-my-program-with-colored-output-not-have-color-under-ninja

I think that wiki page should also mention you can prevent ninja escaping color output by setting that env variable.

Ninja itself will detect if it's writing to a terminal and escape color codes from commands if it isn't.

Also I don't really follow the reasoning for this. Its one thing to explicitly pass a flag to tools to enable color in environments that ordinarily wouldn't have it but needing to pass another different flag to ninja to prevent it overriding that choice is a bit odd. I guess there's a uniformity aspect to the output but really I think ninja should defer to the tools it runs instead of deciding for them. This might make more sense if output from ninja was colored. Like task descriptions. But as of now I don't think that's supported.

The compiler would behave in the same way when invoked directly, that's the reasoning ;)

When invoked directly the compiler would print color... unless I'm misunderstanding. If I just run the command that ninja is doing directly and pipe to cat it still has color.

On the hander if I stop setting the CMake option that passes the compiler option that enables color but do still supply the CLI_COLOR force env variable then I no longer get color regardless of if I'm writing to a terminal or not.

I think what I dislike about the current behaviour is it's an override of an override. If gcc respected CLICOLOR_FORCE I'd disagree but adding support is still missing and the interim you need to both set CLI_COLOR and pass compiler options to get color working with ninja :disappointed:.

I mean without -fdiagnostics-color=always the compiler will not print color codes when it gets piped into cat. So ninja wants to mimic that, but it's kinda hard since the compiler is ALWAYS piped when ninja executes it. And ninja doesn't (and IMHO shouldn't) know/parse the flags passed to the compiler.
You're right that -fdiagnostics-color is a non standard way and it would be better if GCC used the same environment variable instead. You can raise your support for that here: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80271

You're right that -fdiagnostics-color is a non standard way and it would be better if GCC used the same environment variable instead. You can raise your support for that here: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80271

Environment variables are significantly more nonstandard than -fdiagnostics-color, but perhaps you meant that GCC should add support for --color={always|auto|never}?

Environment variables are significantly more nonstandard [...]

Huh why? There's a standard set of environment variables and there's a standard set of command line flags.

Note that CLICOLOR_FORCE is a nonstarter since setting it will break the ls program on some operating systems, [...]

Can you elaborate?


Concurrency bug in DefaultContractResolver

Just this week a client of mine stumbled on a concurrency bug in the application which I traced back to JSON.NET. The application uses JSON.NET v7.0.1 but after reviewing the code, my conclusion is that the bug still exists in v9.0.1.
Below I'll describe why I conclude the problem is in JSON.NET.
Analysis of the encountered bug in the application
The application runs the following code:
private static readonly Dictionary<string, Type> knownQueryTypes =
    Bootstrapper.GetQueryTypes().ToDictionary(type => type.FullName, type => type);

private static readonly JsonSerializerSettings jsonSettings = new JsonSerializerSettings
{
    TypeNameHandling = TypeNameHandling.Auto,
};

[OperationContract]
[FaultContract(typeof(ValidationError))]
public string JsonExecute(string queryTypeName, string json)
{
    Type queryType = knownQueryTypes[queryTypeName];

    dynamic query = JsonConvert.DeserializeObject(json, queryType, jsonSettings);

    Type resultType = GetQueryResultType(queryType);
    Type queryHandlerType = typeof(IQueryHandler<,>).MakeGenericType(queryType, resultType);
    dynamic queryHandler = Bootstrapper.Container.GetInstance(queryHandlerType);

    object result = queryHandler.Handle(query);

    return this.jsonSerializerService.SerializeObject(result);
}

In short, the JsonExecute method is a WCF service call that accepts a query message, serialized as json where the message type is supplied as queryTypeName. Internally, the method converts the queryTypeName to the actual Type using a dictionary that is initialized during application start-up and after that is only read from.
Based on this query Type, a IQueryHandler<TQuery, TResult> implementation (a query handler) is requested from the use DI container, after which this handler is invoked using C# dynamic.
Things to note in this code:

The method is completely synchronous in nature. No async.
The knownQueryTypes dictionary is never changed after creation.
The only two places where currency issues could arrise are JsonConvert.DeserializeObject and the call to Container.GetInstance.

After this web service has ran for several years, this week the following exception ocurred:

Microsoft.CSharp.RuntimeBinder.RuntimeBinderException: The best overloaded method match for 'MyApp.BusinessLayer.CrossCuttingConcerns.PermissionQueryHandlerDecorator<MyApp.Contract.Queries.GetAnalyseAanvraagViewByIdQuery, MyApp.Contract.DTOs.AnalyseAanvraagView>.Handle(MyApp.Contract.Queries.GetAnalyseAanvraagViewByIdQuery)' has some invalid arguments

With the following stack trace
at CallSite.Target(Closure , CallSite , Object , Object )
at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet](CallSite site, T0 arg0, T1 arg1)
at MyApp.ServiceLayer.QueryService.JsonExecute(String queryTypeName, String json)

In the log we saw that the
Ftom the call was for the following request:
JsonExecute("MyApp.Contract.Queries.GetAnalyseAanvraagViewByIdQuery", "{ json payload }");

During that day all following requests for this particular query failed with the same exception. After the service was restarted the problem disapeared. We haven't seen it.
After close inspection of the code I came to the conclusion that the call to JsonConvert.DeserializeObject returned an instance of the incorrect type, because:

The call to knownQueryTypes[queryTypeName] can't return the incorrect type, since the dictionary is never changed after creation.
Since the queryType is correct, the correct query handler must be requested from the container.
The container returns the correct handler type, because the C# dynamic binder reports that it found the a method with the signature Handle(GetAnalyseAanvraagViewByIdQuery), but that it doesn't match.

This means that the only possible reason for the C# binder to report that the method signature doesn't match (and overload resolution fails) is because the actual type of the query instance (created by JsonConvert.DeserializeObject) is different from the requested queryType. Also note that this application runs unchanged in production for quite some time and used a lot.
Analysis of the JSON.NET code base
After coming to the conclusion that the problem must be in JSON.NET, I started looking at the code base of JSON.NET 7.0.1 and I found the following code that might explain why things go wrong (note the following code is extract from v7.0.1 using Reflector):
public virtual JsonContract ResolveContract(Type type)
{
    JsonContract contract;
    if (type == null)
    {
        throw new ArgumentNullException("type");
    }
    DefaultContractResolverState state = this.GetState();
    ResolverContractKey key = new ResolverContractKey(base.GetType(), type);
    Dictionary<ResolverContractKey, JsonContract> contractCache = state.ContractCache;
    if ((contractCache == null) || !contractCache.TryGetValue(key, out contract))
    {
        contract = this.CreateContract(type);
        lock (TypeContractCacheLock)
        {
            contractCache = state.ContractCache;
            Dictionary<ResolverContractKey, JsonContract> dictionary2 = (contractCache != null)
				? new Dictionary<ResolverContractKey, JsonContract>(contractCache)
				: new Dictionary<ResolverContractKey, JsonContract>();
            dictionary2[key] = contract;
            state.ContractCache = dictionary2;
        }
    }
    return contract;
}

After analyzing this code we can quickly spot a concurrency bug: contractCache.TryGetValue is used outside the lock. Dictionary<TKey, TValue> can not safely be used with one concurrent writer. MSDN clearly states:

A Dictionary<TKey, TValue> can support multiple readers concurrently, as long as the collection is not modified.

In other words, there is a concurrency bug in the JsonContract cache, and that might explain problem described above. If the DefaultContractResolver resolves a JsonContract for the wrong type, deserialization can fail silently, since that type can usually be created successfully, but it will be incorrectly initialized. In this case all query objects have default constructors. Deserialization would likely fail if the query class was immutable with a non-default constructor.
A possible fix for this problem is to replace the Dictionary<ResolverContractKey, JsonContract> with an old fashioned HashTable, since:

Hashtable is thread safe for use by multiple reader threads and a single writing thread.

Do note that the latest code contains the same bug. This means that even v9.0.1 is affected.

HashTable is not safe for read/writer at the same time. We should update the documentation...

@davkean I am pretty sure HashTable is meant to be safe for this. Take a look at the code for HashTable. You'll see that the code goes into great length to ensure it supports multiple readers and a single writer concurrently. You'll see spinwait loops in there for this.

It does attempt to go to great lengths to attempt to this - unfortunately, it wasn't successful and still had lots of threading issues (when I'm next connected to the VPN, I'll look up the bugs we had around this if you'd like). In the end, this is why Dictionary wasn't made safe for a writer and reader at the same time - lock free data structures are hard.

@davkean I think it is really important that you make this publically that there are issues with HashTable and what they are. There are OSS projects that depend on this documented behavior of HashTable.

But in case HashTable with one writer and multiple concurrent readers is out of the question, the best solution would be to use 'immutable' Dictionary<T, V> and never update the instance but replace its reference with the reference of a new copy that includes the new value.

Would be great to know the bugs. Our usage in Serilog is pretty forgiving (I don't think we care if we see stale reads, and we don't enumerate or have hard requirements for a correct Count), so as long as the bugs aren't state-corrupting we'll probably be fine.
Seems a bit late to change the docs here though; perhaps adding the known-issues list would be more useful?

After analyzing this code we can quickly spot a concurrency bug: contractCache.TryGetValue is used outside the lock. Dictionary

You are incorrect. state.ContractCache is never modified. When state.ContractCache needs to change then a new dictionary is created on a local variable, the new value is added to it, and only then is it assigned to state.ContractCache. It is thread safe.

It is but you are incorrect that it's not thread safe.

Crap! You are right. I missed the newing of the dictionaries. You are already reference swapping.

This doesn't invalidate the analysis of the reported problem though. I suspect there to still be a concurrency problem somewhere in JSON.NET (at least in v7). I advised my client to upgrade to v9 and to add an extra guard clause to make it very clear when DeserializeObject returns a different type than the supplied one. In case this happens, I will post a new  bug report.

@davkean, do you have any public announcement about HashTable and the possible problems with it? It would really help the whole community and libraries like Serilog that depend on hashtable, quite extensively.


Fix markdownlint warnings, and highlight release-branch distinction.

I added some sample command to clone a release branch in case it's not clear.

Yep I forgot about that, or rather, I had remembered it as an issue, not a PR. I'll rebase this at some point since I think it still adds useful data

OK I rebased to keep the useful bits.


Qusetion about raw dataset

Thank you for great your work!
I have a question about your datasets.
I get the RoadScene dataset from here, and then there are two types of thermal images(cropinfrared and infrared).
What is correct infrared image for making the test images using this code..?
To my understanding, I use the corp infrared images for making the test dataset. Is it right?
Thank you. I'm waiting your reply.

In addition, Train/Test data are not clear.. How can I get the splitted RoadScene datasets to reproduce your paper. Could you provide the pretrained weight for testing your model...?

@wdhudiekou I'm waiting your reply.


Disable deprecated track 1 SDKs from building

These are generating CG alerts unnecessary and are both deprecated in
favor of Azure Monitor Query SDK.

/check-enforcer override


SMO version 150 documentation is missing

Submitted to site feedback by @jeffshumphreys
Is your feature request related to a problem? Please describe.
I am trying to use SMO scripter to extract table DDL from source tables. I am looking for documentation on SMO v 150, which is the minimum version required for connecting to Azure Managed instances
Describe the solution you'd like
I'd like some documentation on the ServerConnection that explains if it is possible in v150 to use Azure Active Directory Authentication to connection to Azure from SMO.
Describe alternatives you've considered
My alternative is to use .NET Sql connections (which support AAD) and try and describe objects through dictionary objects.
Additional context
Add any other context or screenshots about the feature request here.

@ryanmajidi , Could you please provide the document you are referring to?

@ryanmajidi , Could you please provide the document you are referring to?

@ryanmajidi , I have not heard back from you on this issue. I will go ahead and close the issue for now. Please feel free to re-open the issue if you need further assistance.


pypy3.7 failing install - invalid version spec

example build
Investigate and resolve.

This is occurring because the Ubuntu 18.04 agent does not have pypy3.7 on it. Easiest way to resolve this is to bump the tests-weekly to ubuntu 20.04. I'll submit a PR for that and invite discussion.


Strip NULL characters when normalizing paths

(see related issue https://github.com/wagtail/wagtail/issues/4496)
After migrating a Wagtail-based site from MySQL to Postgres, we noticed that malicious requests to the site that included percent-encoded Unicode NULLs (%00) raised a ValueError exception that we hadn't seen when using MySQL: A string literal cannot contain NUL (0x00) characters. This may relate to psycopg2's decision to raise an exception in these situations, as discussed here: https://github.com/psycopg/psycopg2/issues/420
While newer versions of Django appear to provide some field validation that addresses these null chars, it doesn't look like Wagtail's redirect middleware is making use of those validators, and so it seemed reasonable to clean these chars in the context of normalizing the paths before looking in the db for corresponding redirects – especially since a quick investigation on the internet suggests that U+0000 in URLs can be used as a means of attack, and also since RFC 3986 says:

Note, however, that the "%00" percent-encoding (NUL) may require
special handling and should be rejected if the application is not
expecting to receive raw data within a component.

Thanks for considering this patch.
cc: @chosak

Seems like this is what is needed -- but should the issue be elevated to psycopg or Django's db backend? Postgres doesn't allow \0 in general.

@benjaoming per the psycopg2 issue mentioned by @acrewdson above, the psycopg2 code was modified here in the 2.7 release to deliberately raise an exception when a null byte is sent.
See this Django ticket and this django-developers thread for discussion about how to handle this in Django; consensus solution was creation of the ProhibitNullCharactersValidator and its addition to CharField and TextField.
For our purposes here in Wagtail, we can't leverage that validator because we're not using a form when we test the request path against redirects.

Merged in 882f8f3; thanks @acrewdson.

Revisiting this fix to backport it to earlier Wagtail versions, and I'm not sure if this will catch all versions of this attack:

are there other ways of inserting null bytes other than %00 (e.g. sending an actual 0 byte as part of the HTTP request)?
can the null byte be passed somewhere other than the querystring?

It looks like a more robust fix would be to add
if '\0' in path:
    return None

to the top of _get_redirect in middleware.py. However, I can't reproduce the original issue to test this locally. Has anyone been able to reproduce this on ./manage.py runserver, or does it only occur when running on a 'real' webserver?

@gasman I was able to repro it locally with Django's runserver, using the type of request that we saw, and that others reported, involving the null character being percent-encoded in the querystring. It's a good point that the character could be inserted in other contexts, not just the querystring, so it might be worth revisiting this as you suggest.

I can reliably produce the A string literal cannot contain NUL (0x00) characters error using runserver following these steps:

wagtail start mysite && cd mysite
Edit mysite/settings/base.py to change default database to one usingdjango.db.backends.postgresql_psycopg2. (You'll also need to pip install psycopg2 into your environment, if you don't have it already.)
./manage.py migrate
./manage.py runserver
Either of these will produce the 500:$ curl -I 'http://localhost:8000/foo%00'
$ curl -I 'http://localhost:8000/foo/?bar=%00'

This generates 500s with all of these combinations:

Python 2.7/Django 1.11/Wagtail 1.13
Python 3.6/Django 1.11/Wagtail 1.13
Python 3.6/Django 1.11/Wagtail 2.0

but not with this:

Python 3.6/Django 2.0/Wagtail 2.0

So perhaps there's some fix in Django 2 that helps this somehow? Looking at the changelog, nothing obvious jumps out except maybe this.
@gasman your idea of a more robust checking of the entire path seems like a good one.


Broken relocation in Pop'n_Music_Portable (Japanese)

For some reason the Image base parameter is 0000000 when I load the Eboot.bin binary. This is not fatal, but I think it causes Ghidra to assume that a lot of functions reference the data (when, in fact, they reference the code) => the analyzer creates the data in the middle of the code segments so when the time comes to disassemble the region, it fails. I ended up with lots of partially disassembled funcs which no one can work with. You have to manually clear the data, then disassemble again, then recreate the function again.
Setting up the base to 0x08804000 doesn't help as now it breaks the calls:

I say something is wrong with the way someone handles relocation. My initial guess is that I should check if this plugin is to blame.
I'd like to know what can be done with this.

Sounds like a bug in the plugin. I'll check it.

Confirmed this is unsupported case where relocation info is not in the section header but in the program header. Hopefully it will be easy to add if format and processing is the same.