BUG: stats.gaussian_kde: segmentation fault

Describe your issue.
I am trying to use gaussian_kde() to fit a 2D KDE model on a dataset with shape (47000, 2). However, I keep getting segmentation fault on a cluster with 40 CPUs and 300+GB memory. I tried to increase the stack limit using ulimit -s, but it didn't help.
Reproducing Code Example
import numpy as np
import pandas as pd
from scipy.stats import gaussian_kde

df = pd.read_csv('https://gist.githubusercontent.com/xiaohk/925115eda0c5ed2471dc00be5b3d6046/raw/5adc38dce004303b4c413aa9797c53bd99addfe7/test-data-2d.csv')
data = np.stack([df['x'], df['y']], axis=1)
kde_model = gaussian_kde(data, bw_method='silverman')

Error message
Segmentation fault

==== Below is the gdb result ====

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
__memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:145
145     ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S: No such file or directory.
(gdb) backtrace
#0  __memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:145
#1  0x00007fa0587f276c in f2py_rout.flapack_dpotrf () from /nvmescratch/jay/miniconda3/envs/rapids/lib/python3.9/site-packages/scipy/linalg/_flapack.cpython-39-x86_64-linux-gnu.so
#2  0x0000555555688fa7 in _PyObject_MakeTpCall (tstate=0x55555590ddd0, callable=0x7fa058a3e780, args=<optimized out>, nargs=<optimized out>, keywords=<optimized out>) at /usr/local/src/conda/python-3.9.13/Objects/call.c:191
#3  0x000055555568507f in _PyObject_VectorcallTstate (kwnames=0x7fa058a1b5c0, nargsf=<optimized out>, args=<optimized out>, callable=0x7fa058a3e780, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:116
#4  _PyObject_VectorcallTstate (kwnames=0x7fa058a1b5c0, nargsf=<optimized out>, args=<optimized out>, callable=0x7fa058a3e780, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:103
#5  PyObject_Vectorcall (kwnames=0x7fa058a1b5c0, nargsf=<optimized out>, args=<optimized out>, callable=0x7fa058a3e780) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:127
#6  call_function (kwnames=0x7fa058a1b5c0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5077
#7  _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3537
#8  0x000055555567ea17 in _PyEval_EvalFrame (throwflag=0, f=0x7ffae9624640, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40
#9  _PyEval_EvalCode (tstate=<optimized out>, _co=0x7fa0586e22f0, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>, kwargs=<optimized out>, kwcount=<optimized out>,
    kwstep=<optimized out>, defs=<optimized out>, defcount=<optimized out>, kwdefs=<optimized out>, closure=<optimized out>, name=<optimized out>, qualname=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4329
#10 0x0000555555690ff9 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.9.13/Objects/call.c:396
#11 0x0000555555680bcb in _PyObject_VectorcallTstate (kwnames=0x7fa0586e0a90, nargsf=<optimized out>, args=<optimized out>, callable=0x7fa0586ddf70, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:118
#12 PyObject_Vectorcall (kwnames=0x7fa0586e0a90, nargsf=<optimized out>, args=<optimized out>, callable=0x7fa0586ddf70) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:127
#13 call_function (kwnames=0x7fa0586e0a90, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5077
#14 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3537
#15 0x000055555567ea17 in _PyEval_EvalFrame (throwflag=0, f=0x7ffabd1f0d60, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40
#16 _PyEval_EvalCode (tstate=<optimized out>, _co=0x7fa0586e23a0, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>, kwargs=<optimized out>, kwcount=<optimized out>,
    kwstep=<optimized out>, defs=<optimized out>, defcount=<optimized out>, kwdefs=<optimized out>, closure=<optimized out>, name=<optimized out>, qualname=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4329
#17 0x0000555555690ff9 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.9.13/Objects/call.c:396
--Type <RET> for more, q to quit, c to continue without paging--
#18 0x0000555555684636 in _PyObject_VectorcallTstate (kwnames=0x0 <numba.dynamic.globals.7ffd46043b50>, nargsf=<optimized out>, args=0x55555c4f0868, callable=0x7fa0586e6040, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:118
#19 PyObject_Vectorcall (kwnames=0x0 <numba.dynamic.globals.7ffd46043b50>, nargsf=<optimized out>, args=0x55555c4f0868, callable=0x7fa0586e6040) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:127
#20 call_function (kwnames=0x0 <numba.dynamic.globals.7ffd46043b50>, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5077
#21 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3489
#22 0x0000555555691303 in _PyEval_EvalFrame (throwflag=0, f=0x55555c4f06e0, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40
#23 function_code_fastcall (tstate=0x55555590ddd0, co=<optimized out>, args=<optimized out>, nargs=<optimized out>, globals=0x7fa0566b2100) at /usr/local/src/conda/python-3.9.13/Objects/call.c:330
#24 0x0000555555680063 in _PyObject_VectorcallTstate (kwnames=0x0 <numba.dynamic.globals.7ffd46043b50>, nargsf=<optimized out>, args=0x55555c350548, callable=0x7fa0566bd0d0, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:118
#25 PyObject_Vectorcall (kwnames=0x0 <numba.dynamic.globals.7ffd46043b50>, nargsf=<optimized out>, args=0x55555c350548, callable=0x7fa0566bd0d0) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:127
#26 call_function (kwnames=0x0 <numba.dynamic.globals.7ffd46043b50>, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5077
#27 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3506
#28 0x000055555567ea17 in _PyEval_EvalFrame (throwflag=0, f=0x55555c3503b0, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40
#29 _PyEval_EvalCode (tstate=<optimized out>, _co=0x7fa0566bc240, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>, kwargs=<optimized out>, kwcount=<optimized out>, kwstep=<optimized out>,
    defs=<optimized out>, defcount=<optimized out>, kwdefs=<optimized out>, closure=<optimized out>, name=<optimized out>, qualname=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4329
#30 0x00005555556a0328 in _PyFunction_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, stack=0x5555576d29f8, func=0x7fa0566bd040) at /usr/local/src/conda/python-3.9.13/Objects/call.c:396
#31 _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x5555576d29f8, callable=0x7fa0566bd040, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:118
#32 method_vectorcall (method=<optimized out>, args=0x5555576d2a00, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.9.13/Objects/classobject.c:53
#33 0x0000555555680bcb in _PyObject_VectorcallTstate (kwnames=0x7fa05673d280, nargsf=<optimized out>, args=<optimized out>, callable=0x7ffd4606ddc0, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:118
#34 PyObject_Vectorcall (kwnames=0x7fa05673d280, nargsf=<optimized out>, args=<optimized out>, callable=0x7ffd4606ddc0) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:127
#35 call_function (kwnames=0x7fa05673d280, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5077
#36 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3537
#37 0x000055555567ea17 in _PyEval_EvalFrame (throwflag=0, f=0x5555576d2870, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40
#38 _PyEval_EvalCode (tstate=<optimized out>, _co=0x7fa0566b7500, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>, kwargs=<optimized out>, kwcount=<optimized out>, kwstep=<optimized out>,
    defs=<optimized out>, defcount=<optimized out>, kwdefs=<optimized out>, closure=<optimized out>, name=<optimized out>, qualname=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4329
#39 0x00005555556886e0 in _PyFunction_Vectorcall (kwnames=0x0 <numba.dynamic.globals.7ffd46043b50>, nargsf=<optimized out>, stack=<optimized out>, func=0x7fa0566b5af0) at /usr/local/src/conda/python-3.9.13/Objects/call.c:396
#40 _PyObject_FastCallDictTstate (tstate=0x55555590ddd0, callable=0x7fa0566b5af0, args=<optimized out>, nargsf=<optimized out>, kwargs=<optimized out>) at /usr/local/src/conda/python-3.9.13/Objects/call.c:118
#41 0x000055555569ced8 in _PyObject_Call_Prepend (kwargs=0x0 <numba.dynamic.globals.7ffd46043b50>, args=0x7ffae80c9ee0, obj=<optimized out>, callable=0x7fa0566b5af0, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Objects/call.c:489
#42 slot_tp_init (self=<optimized out>, args=<optimized out>, kwds=0x0 <numba.dynamic.globals.7ffd46043b50>) at /usr/local/src/conda/python-3.9.13/Objects/typeobject.c:6971
#43 0x0000555555688fbe in type_call (kwds=<optimized out>, args=0x7ffae80c9ee0, type=<optimized out>) at /usr/local/src/conda/python-3.9.13/Objects/typeobject.c:1028
#44 _PyObject_MakeTpCall (tstate=0x55555590ddd0, callable=0x5555564f61e0, args=<optimized out>, nargs=<optimized out>, keywords=<optimized out>) at /usr/local/src/conda/python-3.9.13/Objects/call.c:191
#45 0x00005555556844ae in _PyObject_VectorcallTstate (kwnames=0x0 <numba.dynamic.globals.7ffd46043b50>, nargsf=<optimized out>, args=0x55555596b4a0, callable=<optimized out>, tstate=<optimized out>)
    at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:116
#46 _PyObject_VectorcallTstate (kwnames=0x0 <numba.dynamic.globals.7ffd46043b50>, nargsf=<optimized out>, args=0x55555596b4a0, callable=<optimized out>, tstate=<optimized out>) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:103
#47 PyObject_Vectorcall (kwnames=0x0 <numba.dynamic.globals.7ffd46043b50>, nargsf=<optimized out>, args=0x55555596b4a0, callable=<optimized out>) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:127
#48 call_function (kwnames=0x0 <numba.dynamic.globals.7ffd46043b50>, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5077
#49 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3520
#50 0x000055555567ea17 in _PyEval_EvalFrame (throwflag=0, f=0x55555596b330, tstate=0x55555590ddd0) at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40
#51 _PyEval_EvalCode (tstate=<optimized out>, _co=0x7ffda3a04870, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>, kwargs=<optimized out>, kwcount=<optimized out>, kwstep=<optimized out>,
    defs=<optimized out>, defcount=<optimized out>, kwdefs=<optimized out>, closure=<optimized out>, name=<optimized out>, qualname=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4329
#52 0x000055555567e6d7 in _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>, kwargs=0x0 <numba.dynamic.globals.7ffd46043b50>, kwcount=0, kwstep=2,
    defs=0x0 <numba.dynamic.globals.7ffd46043b50>, defcount=0, kwdefs=0x0 <numba.dynamic.globals.7ffd46043b50>, closure=0x0 <numba.dynamic.globals.7ffd46043b50>, name=0x0 <numba.dynamic.globals.7ffd46043b50>, qualname=0x0 <numba.dynamic.globals.7ffd46043b50>)
    at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4361
#53 0x000055555567e689 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0 <numba.dynamic.globals.7ffd46043b50>, defcount=0,
    kwdefs=0x0 <numba.dynamic.globals.7ffd46043b50>, closure=0x0 <numba.dynamic.globals.7ffd46043b50>) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4377
#54 0x0000555555739e3b in PyEval_EvalCode (co=co@entry=0x7ffda3a04870, globals=globals@entry=0x7ffda39f8d80, locals=locals@entry=0x7ffda39f8d80) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:828
#55 0x00005555557684a9 in run_eval_code_obj (tstate=0x55555590ddd0, co=0x7ffda3a04870, globals=0x7ffda39f8d80, locals=0x7ffda39f8d80) at /usr/local/src/conda/python-3.9.13/Python/pythonrun.c:1221
#56 0x0000555555764694 in run_mod (mod=<optimized out>, filename=<optimized out>, globals=0x7ffda39f8d80, locals=0x7ffda39f8d80, flags=<optimized out>, arena=<optimized out>) at /usr/local/src/conda/python-3.9.13/Python/pythonrun.c:1242
#57 0x00005555555e96d2 in pyrun_file (fp=0x5555559735e0, filename=0x7ffda3981b70, start=<optimized out>, globals=0x7ffda39f8d80, locals=0x7ffda39f8d80, closeit=1, flags=0x7fffffffd568) at /usr/local/src/conda/python-3.9.13/Python/pythonrun.c:1140
#58 0x000055555575e1f2 in pyrun_simple_file (flags=0x7fffffffd568, closeit=1, filename=0x7ffda3981b70, fp=0x5555559735e0) at /usr/local/src/conda/python-3.9.13/Python/pythonrun.c:450
#59 PyRun_SimpleFileExFlags (fp=0x5555559735e0, filename=<optimized out>, closeit=1, flags=0x7fffffffd568) at /usr/local/src/conda/python-3.9.13/Python/pythonrun.c:483
--Type <RET> for more, q to quit, c to continue without paging--
#60 0x000055555575b533 in pymain_run_file (cf=0x7fffffffd568, config=0x55555590e760) at /usr/local/src/conda/python-3.9.13/Modules/main.c:377
#61 pymain_run_python (exitcode=0x7fffffffd560) at /usr/local/src/conda/python-3.9.13/Modules/main.c:602
#62 Py_RunMain () at /usr/local/src/conda/python-3.9.13/Modules/main.c:681
#63 0x000055555572db79 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at /usr/local/src/conda/python-3.9.13/Modules/main.c:1101
#64 0x00007ffda293fc87 in __libc_start_main (main=0x55555572db30 <main>, argc=2, argv=0x7fffffffd768, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffd758) at ../csu/libc-start.c:310
#65 0x000055555572da81 in ?? () at /usr/local/src/conda/python-3.9.13/Python/getplatform.c:9
#66 0x00007fffffffd758 in ?? ()
#67 0x000000000000001c in ?? ()
#68 0x0000000000000002 in ?? ()
#69 0x00007fffffffdaab in ?? ()
#70 0x00007fffffffdade in ?? ()
#71 0x0000000000000000 in ?? ()

SciPy/NumPy/Python version information
1.9.3 1.23.4 sys.version_info(major=3, minor=9, micro=13, releaselevel='final', serial=0)

Note that the order of the axes is different between gaussian_kde() and sklearn. You need the transpose (2, 47000) array instead.
dataset : array_like
    Datapoints to estimate from. In case of univariate data this is a 1-D
    array, otherwise a 2-D array with shape (# of dims, # of data).

I can reproduce the segmentation fault with the main development branch, but as @rkern points out, the data is almost certainly being passed in with the wrong shape.  Still, it would be nice if the code didn't crash Python, so I think we can leave this open until we have a fix, or we know that it actually can't be fixed (e.g. if the process is being killed by the OS because it is low on memory).

Thank you @rkern and @WarrenWeckesser! Yup, after fixing the data shape, it only takes 0.1 second to fit the model 🤩
I think it might be helpful to give a warning too if the passed shape is "very unusual" (like only 2 rows and 14k dimensions 😄). I imagine many users would assume that scipy expects shape (n, k) like sklearn.

Issuing a warning whenever n_features > n_data would probably be a reasonable rule of thumb.

If the number of observations is less than the number of dimensions, won't this always result in a (theoretically, if not numerically) singular covariance? gaussian_kde really doesn't work at all with singular covariance matrices right now, and that is tracked by gh-10205.
So until gh-10205 is resolved, should we raise an error when n_features > n_data?

@xiaohk Can you tell me which of the following lines, if any, causes a segfault on your cluster?
import numpy as np
import pandas as pd
from scipy.stats import gaussian_kde
from scipy import linalg

df = pd.read_csv('https://gist.githubusercontent.com/xiaohk/925115eda0c5ed2471dc00be5b3d6046/raw/5adc38dce004303b4c413aa9797c53bd99addfe7/test-data-2d.csv')
data = np.stack([df['x'], df['y']], axis=1)

weights = [0.5, 0.5]
temp = np.cov(data, rowvar=1, bias=False, aweights=weights)
_data_covariance = np.atleast_2d(temp)
_data_cho_cov = linalg.cholesky(_data_covariance, lower=True)

The traceback shows it segfaulting in flapack_dpotrf, the Cholesky decomposition routine.

Ok. Then will gh-17446 resolve the scipy.stats part of this issue?
Otherwise, it seems that this needs a scipy.linalg label, no?

@mdhaber
The segfault happens on this line:
_data_cho_cov = linalg.cholesky(_data_covariance, lower=True)

Yes, after I change data to np.random.rand(47000, 2), the program still crashes with a segfault after running for 25 minutes.

We had to pin scipy<1.9.2 to fix a segfault happening in tests.
Could reproduce a different fault reliably by running pytest through valgrind and simply importing gaussian_kde:
from scipy.stats import gaussian_kde

print("hi")

This would produce:
________________________________________________________ ERROR collecting tests/unit_tests/gui/test_gui_load.py _________________________________________________________
tests/unit_tests/gui/test_gui_load.py:13: in <module>
    from ert.gui.gert_main import GUILogHandler, _start_window, run_gui
src/ert/gui/gert_main.py:16: in <module>
    from ert.gui.simulation import SimulationPanel
src/ert/gui/simulation/__init__.py:1: in <module>
    from .simulation_panel import SimulationPanel
src/ert/gui/simulation/simulation_panel.py:27: in <module>
    from .run_dialog import RunDialog
src/ert/gui/simulation/run_dialog.py:34: in <module>
    from ert.gui.tools.plot.plot_tool import PlotTool
src/ert/gui/tools/plot/__init__.py:4: in <module>
    from .plot_tool import PlotTool
src/ert/gui/tools/plot/plot_tool.py:4: in <module>
    from .plot_window import PlotWindow
src/ert/gui/tools/plot/plot_window.py:20: in <module>
    from ert.gui.plottery.plots.gaussian_kde import GaussianKDEPlot
src/ert/gui/plottery/plots/gaussian_kde.py:2: in <module>
    from scipy.stats import gaussian_kde
../envs/ert/lib64/python3.8/site-packages/scipy/stats/__init__.py:467: in <module>
    from ._stats_py import *
../envs/ert/lib64/python3.8/site-packages/scipy/stats/_stats_py.py:46: in <module>
    from . import distributions
../envs/ert/lib64/python3.8/site-packages/scipy/stats/distributions.py:10: in <module>
    from . import _continuous_distns
../envs/ert/lib64/python3.8/site-packages/scipy/stats/_continuous_distns.py:31: in <module>
    import scipy.stats._boost as _boost
../envs/ert/lib64/python3.8/site-packages/scipy/stats/_boost/__init__.py:1: in <module>
    from scipy.stats._boost.beta_ufunc import (
E   SystemError: initialization of beta_ufunc raised unreported exception

Hope this helps

Thanks. That import issue has been reported in https://github.com/scipy/scipy/issues/17368.


circleci refactor + content

Description
(Please describe the changes in this PR for reviewers, motivation, rationale - mandatory)

Upstream PR
The following PR deals with the following generated content for the following versions:

3.10:
3.11: 3.11
3.12:
devel:

/generate

/generate

/generate

/generate

/generate

/generate

/generate

/generate

/generate

/commit-generated


Create data package for examples and tests?

keep toy-data.R script for "ideal/good" inputs/outputs
easier to reference what exactly the "bad" inputs contain when writing tests

Set up for tests and examples is non-trivial because you need concordance tables, panel map, data;
Might be worth creating a separate data package to house this? -- a good template anyway for distributing the panel map artefact?
Discussed in R Packages (2e):

https://r-pkgs.org/man.html#contents
https://r-pkgs.org/data.html#introduction


#17473 fix simple condition build

Q
A

Is bugfix?
✔

New feature?
❌

Breaks BC?
❌

Tests pass?
✔

Fixed issues
#17473

Merged. Thank you!


Version 1.0.0

Rename package to @metarhia/common and update all the mentions in
README.md.
Update devDependencies.
Use "files" field in package.json instead of .npmignore.
Publish lib/unicode-categories.js to npm instead of generating it on
install since it was published inside dist/ anyway.

Refs: #138

Landed in https://github.com/metarhia/common/commit/2a04a4946c43d668b8e4cffb4e8fb9aef0626930.


sitepicker does not reflect changes made to initialSites by other components

Thank you for reporting an issue, suggesting an enhancement, or asking a question. We appreciate your feedback - to help the team understand your
needs please complete the below template to ensure we have the details to help. Thanks!
Please check out the documentation to see if your question is already addressed there. This will help us ensure our documentation is up to date.
Category
[x ] Enhancement
[x ] Bug
[x ] Question
Version
Please specify what version of the library you are using: [ latest  ]
hey alex, i have another 'issue?'. likely my musunderstanding as i interpreted initialsites to be analagous to 'selectedsites'.
I have two sitepickers on my component. When the user selects a site in the First one , i want the second one to be set to what the user selected in  the first one (he can change it later if he wants). I couldn't get it to work so as a test , I just added a button to the component to set the value of the first picker to an arbitrary site. That does not work either.  Any thoughts on this?
I'm happy to work on it if you can point me in the right direction.
Test component is shown below. I can send to github if you like.
Thanks
Russell
Expected / Desired Behavior / Question
other code should be able to change the selected sites after the initial display
Observed Behavior
other code cant  change the selected sites after the initial display
Steps to Reproduce
import styles from './HelloWorld.module.scss';
import { escape } from '@microsoft/sp-lodash-subset';
import { SitePicker } from "@pnp/spfx-controls-react/lib/SitePicker";
import * as React from 'react';
import { BaseComponentContext } from '@microsoft/sp-component-base';
export interface IHelloWorldProps {
context: BaseComponentContext;
}
interface IHelloWorldState {
siteUrl: string;
siteId: string;
siteTitle: string;
captureToSiteId: string;
captureToSiteUrl: string;
captureToSiteTitle: string;
}
export default class HelloWorld extends React.Component<IHelloWorldProps, IHelloWorldState> {
public constructor(props: IHelloWorldProps) {
super(props);
this.state = { siteId: null, siteTitle: null, siteUrl: null, captureToSiteId: null, captureToSiteTitle: null, captureToSiteUrl: null };
}
public render(): React.ReactElement {
console.log(this.state);
return (

<SitePicker
        initialSites={[{ id: this.state.siteId, url: decodeURIComponent(this.state.siteUrl), title: this.state.siteTitle }]}
        context={this.props.context}
        label={'Select site to capture'}
        mode={'site'}
        multiSelect={false}
        onChange={(sites) => {
          debugger;
          console.log('setting Item');
          if (this.state.captureToSiteId === this.state.siteId) {
            this.setState((temp) => ({ ...temp, siteUrl: sites[0].url, captureToSiteUrl: sites[0].url, siteId: sites[0].id, captureToSiteId: sites[0].id, siteTitle: sites[0].title, captureToSiteTitle: sites[0].title }));
          } else {
            this.setState((temp) => ({ ...temp, siteUrl: sites[0].url, siteId: sites[0].id, siteTitle: sites[0].title }));
          }
          console.log(this.state);
        }}
        placeholder={'Select sites'}
        searchPlaceholder={'Filter sites'} />
      <button onClick={
        (e) => {
          debugger;
          this.setState((temp) => ({
            ...temp,
            siteId: "f232b610-7177-4f15-8dd3-59662f47a1f2",
            siteTitle: "All Company",
            siteUrl: "https://russellwgove.sharepoint.com/sites/AllCompany.14022107136.tvymheuq",
            captureToSiteId: "f232b610-7177-4f15-8dd3-59662f47a1f2",
            captureToSiteTitle: "All Company",
            captureToSiteUrl: "https://russellwgove.sharepoint.com/sites/AllCompany.14022107136.tvymheuq"
          }
          ));
        }}>
      </button>
      <SitePicker
        initialSites={[{ id: this.state.captureToSiteId, url: decodeURIComponent(this.state.captureToSiteUrl), title: this.state.captureToSiteTitle }]}
        context={this.props.context}
        label={'Select site to save captured Audit Items to'}
        mode={'site'}
        multiSelect={false}
        onChange={(sites) => {
          this.setState((temp) => ({ ...temp, captureToSiteId: sites[0].id, captureToSiteUrl: sites[0].url, captureToSiteTitle: sites[0].title }));
          console.log(this.state);
        }}
        placeholder={'Select sites'}
        searchPlaceholder={'Filter sites'} />
    </div>
  </div>
);

}
}

Hi @russgove,
Currently SitePicker works in "uncontrolled" mode only. You can provide the initial sites, but after that the selection is completely controlled by the control itself.

So if i changed it to update based on changes made to initialSites. Can I make a PR for this?

or submitted to add selected sites mproperty

This issues was fixed in my last pr


Update old link in the readme

Thanks, there are probably more, if no one else gets to them I will when I move to the 0.17.0 release of solid.


Parsing public.url attachments in note BLOB

Currently, mac_apt does not extract public.url attachments in the note BLOB of Apple Notes.
When creating a note from a web page, a note containing a public.url attachment is created.
Since the Notes app allows the user to choose which note to use, a note may contain multiple public.url attachments.
Each public.url attachment in the note BLOB contains an UUID that can be used to look up the URL.
This will only extract one public.url attachment from the note BLOB:
e.g. High Sierra
SELECT n.Z_PK, n.ZNOTE as note_id, n.ZDATA as data,
c3.ZFILESIZE,
c4.ZFILENAME, c4.ZIDENTIFIER as att_uuid, c3.ZURLSTRING as att_url,
c1.ZTITLE1 as title, c1.ZSNIPPET as snippet, c1.ZIDENTIFIER as noteID,
c1.ZCREATIONDATE1 as created, c1.ZLASTVIEWEDMODIFICATIONDATE, c1.ZMODIFICATIONDATE1 as modified,
c2.ZACCOUNT3, c2.ZTITLE2 as folderName, c2.ZIDENTIFIER as folderID,
c5.ZNAME as acc_name, c5.ZIDENTIFIER as acc_identifier, c5.ZACCOUNTTYPE
FROM ZICNOTEDATA as n
LEFT JOIN ZICCLOUDSYNCINGOBJECT as c1 ON c1.ZNOTEDATA = n.Z_PK
LEFT JOIN ZICCLOUDSYNCINGOBJECT as c2 ON c2.Z_PK = c1.ZFOLDER
LEFT JOIN ZICCLOUDSYNCINGOBJECT as c3 ON c3.ZNOTE= n.ZNOTE
LEFT JOIN ZICCLOUDSYNCINGOBJECT as c4 ON c4.ZATTACHMENT1= c3.Z_PK
LEFT JOIN ZICCLOUDSYNCINGOBJECT as c5 ON c5.Z_PK = c1.ZACCOUNT2
ORDER BY note_id

Can you share a sample of a notes database having this? You can email me if you don't want it shared publicly.

It is a large database.
If you view a web page in Safari on iPhone, you can create a note from the web page which will include a public.url attachment. If you choose the same note, more public.url attachments will be added to the same note.
I did find this which describes the Apple note BLOB format in detail (including public.url attachments) and includes a parser written in Ruby (MIT License):
Revisiting Apple Notes (2): Easy Embedded Objects
https://ciofecaforensics.com/2020/01/13/apple-notes-revisited-easy-embedded-objects/
Revisiting Apple Notes (1): Improved Note Parsing
https://ciofecaforensics.com/2020/01/10/apple-notes-revisited/
More embedded objects:
https://ciofecaforensics.com/2020/01/20/apple-notes-revisited-galleries/
https://ciofecaforensics.com/2020/01/14/apple-notes-revisited-embedded-tables/
MIT License, written in Ruby:
https://github.com/threeplanetssoftware/apple_cloud_notes_parser
ISC License, written in Python:
https://github.com/mildsunrise/protobuf-inspector

Well, I don't have an iphone. So, if you could generate some data and share it, I could look into it.

Here are CSV files with a subset of the data. The BLOB columns are encoded using the SQLite hex() function:
https://www.dropbox.com/s/qmmqqs0r2irkxmk/ZICCLOUDSYNCINGOBJECT.csv?dl=0
https://www.dropbox.com/s/8syzrl7bqxtt0t9/ZICNOTEDATA.csv?dl=0

I accidentally ended up at this repository while looking for other information about MacOS and was going to offer a suggestion on this issue, but then I saw @renesugar already found everything I had to offer :). I eyeballed plugins/notes.py and can confirm that issue issue is that the protobuf itself isn't being read, you are just taking one attachment. That issue shouldn't be limited to URLs since modern Notes functionality allows embedding many attachments and attachments themselves can have children (such as thumbnails). If you don't read the protobuf and then also deal with the mergable objects from ZICCLOUDSYNCINGOBJECT, you won't get all the modern attachments.
You can recreate URLs on a Mac as well, if you don't have an iOS device:

Open Safari
Click the Action button in the upper right of the window, next to the tabs icon
Click on Notes, to embed this in a new note

If needed, I can provide some specific URL BLOBs, but this link Rene already provided specifically deals with URLs about half way through.
I hope this helps!

Will take a look as time permits, it would really help if someone could just share sample databases with this info.

I ended up using a library that parses the various Protobuf objects (including URLs, tables, etc.) embedded in Apple Notes here:
https://github.com/renesugar/readnotes/blob/master/notes2html.py
The library was sufficient for extracting data to migrate notes; for forensics, there is more data to extract like the thumbnails mentioned.
The CSV files that I provided contain examples of Protobuf BLOBs containing URLs.

Revisited this today. The base query is fine, it's just missing a few columns.
In https://github.com/ydkhatri/mac_apt/commit/6bebfff09b6564845e1f0f88105fec905f58f019 I've now fixed the query to read the embed type, url, summary and title. This gets table-data and URLs. I haven't looked into other types.
This does not attempt to read the protobuf and reconstruct the original note view yet. It just gives you the text content of everything in the note, which mostly works to see what's in it, except for tables, where you don't know which piece of the data was in which row/col.

Closing this as public.url attachments are now displayed.


Updated version to CosmosDB_2022-11-15

Contributing to the Azure SDK
Please see our CONTRIBUTING.md if you are not familiar with contributing to this repository or have questions.
For specific information about pull request etiquette and best practices, see this section.

@kshittiz7 we have moved to Track 2 already, so please just upgrade the Track 2 sdk instead of Track 1


Replace command line tests with golden file tests

Replace command line tests with golden file tests

Codecov Report

Merging #61 (3812791) into main (da37027) will increase coverage by 0.37%.
The diff coverage is 50.00%.

@@            Coverage Diff             @@
##             main      #61      +/-   ##
==========================================
+ Coverage   91.18%   91.55%   +0.37%
==========================================
  Files          15       15
  Lines         805      805
==========================================
+ Hits          734      737       +3
+ Misses         71       68       -3

Impacted Files
Coverage Δ

flux_local/tool/diff.py
91.07% <50.00%> (ø)

flux_local/tool/get.py
91.04% <0.00%> (ø)

flux_local/tool/build.py
100.00% <0.00%> (+2.00%)
:arrow_up:

flux_local/tool/selector.py
94.87% <0.00%> (+5.12%)
:arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Improving loggable extension querying performance

The issue is that the query got by the getLogEntriesQuery method don’t use the correct MySQL index therefore the performance is very low when the ext_log_entries has a several millions of rows.
The problem is that the “object_id” field in ext_log_entries table is a varchar(64) type field.
The table has the next indexes:

PRIMARY (id)
log_class_lookup_idx (object_class)
log_date_lookup_idx (logged_at)
log_user_lookup_idx (username)
log_version_lookup_idx (object_id, object_class, version)

getLogEntriesQuery method queries entries by objectId, objectClass and ordered by version. The right index should be log_version_lookup_idx but mysql always use log_class_lookup_idx even thought you force the log_version_lookup_idx index. This happens because objectId is stored as a varchar field and then is queried using an integer value. As an example:
where log.objectId = 1 AND log.objectClass = “AppBundle\Entity\FooClass” ORDER BY log.version DESC -> this uses log_class_lookup_idx index
where log.objectId = “1” AND log.objectClass = “AppBundle\Entity\FooClass” ORDER BY log.version DESC -> this uses log_version_lookup_idx index
In my case, I have 35 million rows in ext_log_entries table. When I perform a query with log_class_lookup_idx index, it takes several minutes!!! (a lot of time). But the same query with the log_version_lookup_idx takes a few milliseconds (great).
I propose, change the line 57 and 89 on Gedmo/Loggable/Entity/Repository/LogEntryRepository.php
Original:         $objectId = $wrapped->getIdentifier();
To:               $objectId = (string) $wrapped->getIdentifier();
By this way, Mysql will use log_version_lookup_idx index and it will improve performance in a big way.
Thanks a lot.

you should PR to https://github.com/Atlantic18/DoctrineExtensions
regarding mapping, you can extend the loggable mapped supperclass instead and modify the annotation for objectId to be integer instead. Although, loggable is not that usable, check a specific impl audit bundle

Hi,
How can I PR to https://github.com/Atlantic18/DoctrineExtensions from my current fork?


Add base62 encoder/decoder

Base62 encoding code and unit tests.

Why would someone use base62 instead of base64?

it does not have / and it seems that it could be more convenient when used in URLs that use / to separate stuff.


Logging conventions etc

@benjaminbollen fixed this. Apparently slices aren't hashable so die if you try to use them as a key for a map.


Terminal: some tokens not light theme aware?

Refs: https://github.com/Microsoft/vscode/issues/19071
I am using the light theme and the results from npm are sometimes shown in black, which looks bad in this theme:

It's like that because npm is saying print yellow on black background. Looks better on a dark terminal:

Nothing we can do other than tweak the colors, which then may cause more contrast issues for yellow on light.


Fixes validation of input bytes in the Base64 decoder

Motivation:
In the current implementation of Base64 decoder an invalid character ½ (\u00BD) is treated as =. Also the character \u007F leads to ArrayIndexOutOfBoundsException.
Modification:
Explicitly checks that all input bytes are ASCII characters (greater than zero). Fix decodabet tables.
Result:
Correctly validation input bytes in Base64 decoder.

@njhill thanx! I also fixed an ArrayIndexOutOfBoundsException and squashed commits.

@netty-bot test this pleas

@netty-bot test this please

@normanmaurer test fixed

@netty-bot test this please

@normanmaurer rebased

@netty-bot test this please


fix: bundle-data metadata leak because of bundlestats was not clean

Master Issue: #17087
Motivation
Explain here the context, and why you're making that change. What is the problem you're trying to solve.
Modifications
Describe the modifications you've done.
Verifying this change

[ ] Make sure that the change passes the CI checks.

(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API: (yes / no)
The schema: (yes / no / don't know)
The default values of configurations: (yes / no)
The wire protocol: (yes / no)
The rest endpoints: (yes / no)
The admin cli options: (yes / no)
Anything that affects deployment: (yes / no / don't know)

Documentation
Check the box below or label this PR directly.
Need to update docs?

[ ] doc-required
(Your PR needs to update docs and you will update later)

[x] doc-not-needed
(Please explain why)

[ ] doc
(Your PR contains doc changes)

[ ] doc-complete
(Docs have been already added)

Hi, @leizhiyuan
Would you like to push a PR to branch-2.9 to fix this problem?

Can you help open a new PR to branch-2.10? There are a lot conflict when cherry-pick directly. @leizhiyuan

Can you help open a new PR to branch-2.10? There are a lot conflict when cherry-pick directly. @leizhiyuan

ok

release/2.10.3 moved to https://github.com/apache/pulsar/pull/17463

Hi @leizhiyuan
Could you help cherry-pick this PR to branch-2.9? thanks

[fix][broker]bundle-data metadata leak because of bundlestats was not clean #17316

https://github.com/apache/pulsar/pull/17316
I have pushed before, please take a look

[fix][broker]bundle-data metadata leak because of bundlestats was not clean #17316

#17316
I have pushed before, please take a look

seem the test failed, could you please fix this test? Thanks.

@leizhiyuan  hi, I move this PR to release/2.9.5, if you have any questions, please ping me. thanks.

Cherry-pick PRs failed and stale. Close them and label this PR with release/2.10.3.

As discussed on the mailing list https://lists.apache.org/thread/w4jzk27qhtosgsz7l9bmhf1t7o9mxjhp, there is no plan to release 2.9.6, so I am going to remove the release/2.9.6 label


Skip type resolution in isDelegate

During profiling I noticed that PatchDelegate.isDelegate() consumes some time during compilation. I noticed that the type resolution of the annotation is responsible for that. Even if I'm not sure if that this increases the compilation time I added an additional check that returns early.
This causes some interesting side effects as it broke some unrelated test cases. After checking out why, I think that the unsuccessful type resolution (due to the unknown ckeckerframework annotation) stops compilation to early and hides the underlying problem. It seems like the this parameter have to be set as Receiver instead of an Argument in eclipse. Otherwise it fails with Only the first formal parameter may be declared explicitly as 'this'.
I think I fixed that but couldn't test it in a real world scenario.

Nice job!

I think we should have seen that the constructor body was empty...


KuzuGraph not working

System Info
Name: kuzu
Version: 0.0.10 (latest)
Name: langchain
Version: 0.0.311
Who can help?
No response
Information

[ ] The official example notebooks/scripts
[ ] My own modified scripts

Related Components

[ ] LLMs/Chat Models
[ ] Embedding Models
[ ] Prompts / Prompt Templates / Prompt Selectors
[ ] Output Parsers
[ ] Document Loaders
[ ] Vector Stores / Retrievers
[ ] Memory
[ ] Agents / Agent Executors
[ ] Tools / Toolkits
[ ] Chains
[ ] Callbacks/Tracing
[ ] Async

Reproduction
I am using the example from langchain documentation:
https://python.langchain.com/docs/use_cases/graph/graph_kuzu_qa
This line fails:
graph = KuzuGraph(db)
F```
ile ~/.local/lib/python3.10/site-packages/langchain/graphs/kuzu_graph.py:71, in KuzuGraph.refresh_schema(self)
69 for table in rel_tables:
70     current_table_schema = {"properties": [], "label": table["name"]}
---> 71     properties_text = self.conn._connection.get_rel_property_names(
72         table["name"]
73     ).split("\n")
74     for i, line in enumerate(properties_text):
75         # The first 3 lines defines src, dst and name, so we skip them
76         if i < 3:
AttributeError: 'kuzu._kuzu.Connection' object has no attribute 'get_rel_property_names'

### Expected behavior

should work like example

The method "get_rel_property_names" has been removed since kuzu 0.0.10.
Check this PR: https://github.com/kuzudb/kuzu/pull/2207
A temporary solution is to use kuzu==0.0.9
pip install kuzu==0.0.9

I've created a PR to resolve this issue: #16885

Should be fixed in https://github.com/langchain-ai/langchain/pull/16885


Fix Xcode 10.2 warnings

Is this ready to be merged and released in Result v5.0? Do you need help with anything?

Is this ready to be merged and released in Result v5.0?

This should not need major version bump.

Not on its own, but a v5 release that signals the transition to Swift 5's native Result type support would be nice! :)


feat: update react demo app with vite #76

What kind of change does this PR introduce? (check at least one)

[x] Bugfix
[ ] Feature
[ ] Code style update
[x] Refactor
[x] Build-related changes
[ ] Other, please describe:

Does this PR introduce a breaking change? (check one)

[ ] Yes
[ ] No

If yes, please describe the impact and migration path for existing applications:
The PR fulfills these requirements:

[x] It's submitted to the main branch for v2.x (or to a previous version branch)
[ ] When resolving a specific issue, it's referenced in the PR's title (e.g. fix #xxx[,#xxx], where "xxx" is the issue number)
[ ] All tests are passing
[ ] New/updated tests are included

If adding a new feature, the PR's description includes:

[ ] A convincing reason for adding this feature (to avoid wasting your time, it's best to open a suggestion issue first and wait for approval before working on it)

Other information:

Closes #76
Minor corrections on docs for dead links and missing documentation

not a priority for now - therefore closing this PR, if we get active react users we consider investing time again


ghcjs buildable version

Big fan of the library, including the api upgrades from <IP_ADDRESS>. I needed to build this via ghcjs and cpphs or thyme.h was blowing it up (linker errors, couldn't figure out which one). Regardless, this version built fine on OSX for ghcjs at the cost of the lens flag. Overall I dislike having a little custom lens as more and more people start using this lib but I do see the motivation as the build times increase. I think given stack and new-build that one time build hit is no longer so onerous.

@buckie Your PR doesn't build with ghc 7.6 of ghc 7.8:
src/Data/Thyme/Clock/TAI.hs:85:52:
    Not in scope: ‘mempty’

Versions of GHC earlier than 7.10 did not include Data.Monoid in the standard prelude. Fixing this probably involves adding something like:
#if __GLASGOW_HASKELL__ < 710
import Data.Monoid (mappend)
#endif


program can't finished normally in dist_sync mode

Description
I train resnet50 with 4  nodes by using example/image-classification/train_imagenet.py file.  At beginning, it run normally. But at the last epoch, when one of workers finished its' total iterations named n. However, the others are still waiting for update in (n+1) iter.
The difference of the number of total iterations between workers result in this bug
Configuration
the bash contenets as follows and was ran on gn2
python -u /THL5/home/daodao/cxt/mxnet0.11/tools/launch.py -n 4 --launcher ssh -H hosts-4 "python -u /THL5/home/daodao/cxt/mxnet0.11/example/image-classification/train_imagenet.py --gpus 0,1,2,3 --kv-store dist_sync "

hosts-4  contents
gn2
gn3
gn4
gn5

configuration for training
num_examples:120000
num_epochs:80

the deatils of configuration for training
 INFO:root:start with arguments Namespace(batch_size=256, benchmark=0, data_nthreads=4, data_train='/THL5/home/daodao       /imagenet/mxnet_cxt/train1200000.rec', data_val='/THL5/home/daodao/imagenet/mxnet_cxt/val40000.rec', disp_batches=1,        dtype='float32', gpus='0,1,2,3', image_shape='3,224,224', kv_store='dist_sync', load_epoch=None, lr=0.4, lr_factor=       0.1, lr_step_epochs='30,60', max_random_aspect_ratio=0.25, max_random_h=36, max_random_l=50, max_random_rotate_angle       =10, max_random_s=50, max_random_scale=1, max_random_shear_ratio=0.1, min_random_scale=1, model_prefix=None, mom=0.9       , monitor=0, network='resnet', num_classes=1000, num_epochs=80, num_examples=120000, num_layers=50, optimizer='sgd',        pad_size=0, random_crop=1, random_mirror=1, rgb_mean='123.68,116.779,103.939', test_io=0, top_k=0, wd=0.0001)

Analysis
the root of the bug
due to the numbers of data between workers are different which are partitioned using the API of mx.io.ImageRecordIter, this cause the number of iterations in each epoch between workers is different. And the total iterations during the whole trainning stage is still different.
When the least number of total iterations worker named worker1 finished the last iter n, the other workers still need to train on iter n+1,n+2... During the update of n+1 iter, kvstore must recieve the grads from worker1, And worker1 has finished the total iterations, won't push grads to kvstore, So kvstore still waiting for the push of worker1, This cause the program can't finished normally.
evidences
the number of iterations in each epoch between workers

epoch
worker0
worker1
worker2
worker3

1
1172
1173
1172
1172

2
1172
1172
1172
1172

3
1171
1172
1172
1172

4
1172
1172
1172
1172

5
1171
1172
1172
1172

6
1172
1172
1172
1171

7
1172
1173
1172
1172

8
1171
1172
1172
1172

9
1172
1172
1172
1172

10
1171
1172
1172
1172

11
1172
1172
1172
1171

12
1171
1172
1172
1172

...
...
...
...
...

log info
INFO:root:Epoch[0] Rank[3] Batch[1171] TotalIter[1171] Speed: 187.74 samples/sec
INFO:root:Epoch[0] Train-accuracy=nan
INFO:root:Epoch[0] Time cost=1585.830
INFO:root:Epoch[0] Rank[0] Batch[1171] TotalIter[1171] Speed: 187.42 samples/sec
INFO:root:Epoch[0] Train-accuracy=nan
INFO:root:Epoch[0] Time cost=1585.830
INFO:root:Epoch[0] Rank[2] Batch[1171] TotalIter[1171] Speed: 188.36 samples/sec   INFO:root:Epoch[0] Train-accuracy=nan
INFO:root:Epoch[0] Time cost=1585.818
INFO:root:Epoch[0] Rank[1] Batch[1172] TotalIter[1172] Speed: 189.53 samples/sec   INFO:root:Epoch[0] Train-accuracy=nan
INFO:root:Epoch[0] Time cost=1587.169
INFO:root:Epoch[1] Rank[3] Batch[1] TotalIter[1173] Speed: 189.76 samples/sec
INFO:root:Epoch[1] Rank[0] Batch[1] TotalIter[1173] Speed: 187.95 samples/sec
INFO:root:Epoch[1] Rank[2] Batch[1] TotalIter[1173] Speed: 189.15 samples/sec
INFO:root:Epoch[1] Rank[3] Batch[2] TotalIter[1174] Speed: 188.17 samples/sec
INFO:root:Epoch[1] Rank[1] Batch[1] TotalIter[1174] Speed: 189.03 samples/sec
INFO:root:Epoch[1] Rank[0] Batch[2] TotalIter[1174] Speed: 189.08 samples/sec
INFO:root:Epoch[1] Rank[2] Batch[2] TotalIter[1174] Speed: 188.24 samples/sec
......
INFO:root:Epoch[79] Rank[3] Batch[1151] TotalIter[93724] Speed: 189.38 samples/sec
INFO:root:Epoch[79] Rank[1] Batch[1123] TotalIter[93724] Speed: 189.85 samples/sec
INFO:root:Epoch[79] Rank[2] Batch[1139] TotalIter[93724] Speed: 189.56 samples/sec
INFO:root:Epoch[79] Rank[0] Batch[1169] TotalIter[93724] Speed: 189.05 samples/sec
INFO:root:Epoch[79] Rank[1] Batch[1124] TotalIter[93725] Speed: 189.96 samples/sec
INFO:root:Epoch[79] Rank[3] Batch[1152] TotalIter[93725] Speed: 188.91 samples/sec
INFO:root:Epoch[79] Rank[2] Batch[1140] TotalIter[93725] Speed: 190.17 samples/sec
INFO:root:Epoch[79] Rank[0] Batch[1170] TotalIter[93725] Speed: 190.06 samples/sec
INFO:root:Epoch[79] Train-accuracy=nan
INFO:root:Epoch[79] Time cost=1582.509
Killed by signal 15.^M
Killed by signal 15.^M
Killed by signal 15.^M
Killed by signal 15.^M
Killed by signal 15.^M
Killed by signal 15.^M
Killed by signal 15.^M
Killed by signal 15.^M

According to the log and table info.

We can know that in the end of epoch 79, worker0 finished the 1171-th update, then print the summary of the epoch.
And with the table info, it's clearly the others haven't finish this epoch, still waiting update in the 1172-th iteration.
the TotalIter shows each worker had execute 93725 iterations during the whole training stage, And each iteration is strictly synchronous.

ps info and gpu usage
before i kill the relative process on each node, the info of process as below:
gn2: the node of running the launch.py

gn3

gn4 and gn5 are the same as gn3
We can know those process still in waiting  stage, because the usage of gpus is 0%.
Suggestion
The root of this bug is  io.ImageRecordIter part which haven't splited the whole dataset evenly. And cause the different number of iterations in each epoch between workers. Maybe io.ImageRecordIter has more consideration of performance and efficiency.
The easiest way is:
Assumptions:
num_examples:the number of examples in the whole dataset.
num_workers: the number of workers
num_examples_each_worker: the number of examples splited to each worker.

Then
num_examples_each_worker = num_examples//num_workers + num_examples%num_workers

This is the easiest way to guarantee the same number of iterations in each epoch between workers.
Another drawback
when i set display_batchs=20 ,and haven't print Rank,TotalIterinfo, The output info is out of order, This will mistake people that is not strictly synchronized in dist_sync mode. as belows:
INFO:root:Epoch[55] Batch [80]	Speed: 138.85 samples/sec	accuracy=0.674023
INFO:root:Epoch[55] Batch [20]	Speed: 191.34 samples/sec	accuracy=0.670945
INFO:root:Epoch[55] Batch [100]	Speed: 190.75 samples/sec	accuracy=0.670312
INFO:root:Epoch[55] Batch [40]	Speed: 190.62 samples/sec	accuracy=0.664258
INFO:root:Epoch[55] Batch [40]	Speed: 190.42 samples/sec	accuracy=0.669531
INFO:root:Epoch[55] Batch [80]	Speed: 190.35 samples/sec	accuracy=0.655273
INFO:root:Epoch[55] Batch [60]	Speed: 190.31 samples/sec	accuracy=0.661133
INFO:root:Epoch[55] Batch [120]	Speed: 190.26 samples/sec	accuracy=0.668945
INFO:root:Epoch[55] Batch [100]	Speed: 190.23 samples/sec	accuracy=0.661523
INFO:root:Epoch[55] Batch [40]	Speed: 190.10 samples/sec	accuracy=0.671289
INFO:root:Epoch[55] Batch [120]	Speed: 189.92 samples/sec	accuracy=0.670117
INFO:root:Epoch[55] Batch [60]	Speed: 189.77 samples/sec	accuracy=0.668164
INFO:root:Epoch[55] Batch [60]	Speed: 189.87 samples/sec	accuracy=0.666406
INFO:root:Epoch[55] Batch [80]	Speed: 189.79 samples/sec	accuracy=0.651758
INFO:root:Epoch[55] Batch [100]	Speed: 189.63 samples/sec	accuracy=0.676758
INFO:root:Epoch[55] Batch [140]	Speed: 189.74 samples/sec	accuracy=0.677344
INFO:root:Epoch[55] Batch [120]	Speed: 189.76 samples/sec	accuracy=0.664258
INFO:root:Epoch[55] Batch [60]	Speed: 189.84 samples/sec	accuracy=0.678516
...............................
INFO:root:Epoch[119] Batch [220]	Speed: 189.30 samples/sec	accuracy=0.692578
INFO:root:Epoch[119] Batch [260]	Speed: 189.36 samples/sec	accuracy=0.697461
INFO:root:Epoch[119] Batch [140]	Speed: 189.32 samples/sec	accuracy=0.705664
INFO:root:Epoch[119] Batch [180]	Speed: 189.32 samples/sec	accuracy=0.696484
INFO:root:Epoch[119] Batch [160]	Speed: 189.26 samples/sec	accuracy=0.708984
INFO:root:Epoch[119] Batch [320]	Speed: 189.30 samples/sec	accuracy=0.704297
INFO:root:Epoch[119] Batch [300]	Speed: 189.24 samples/sec	accuracy=0.715625
INFO:root:Epoch[119] Batch [140]	Speed: 189.33 samples/sec	accuracy=0.692969
INFO:root:Epoch[119] Batch [240]	Speed: 189.33 samples/sec	accuracy=0.702148
INFO:root:Epoch[119] Batch [280]	Speed: 189.26 samples/sec	accuracy=0.688477
INFO:root:Epoch[119] Batch [160]	Speed: 189.38 samples/sec	accuracy=0.708789
INFO:root:Epoch[119] Batch [200]	Speed: 189.36 samples/sec	accuracy=0.706641
INFO:root:Epoch[119] Batch [180]	Speed: 189.39 samples/sec	accuracy=0.693750
INFO:root:Epoch[119] Batch [320]	Speed: 189.39 samples/sec	accuracy=0.695898
INFO:root:Epoch[119] Batch [340]	Speed: 189.38 samples/sec	accuracy=0.702930
INFO:root:Epoch[119] Batch [160]	Speed: 189.41 samples/sec	accuracy=0.694727
INFO:root:Epoch[119] Batch [260]	Speed: 189.39 samples/sec	accuracy=0.710156
INFO:root:Epoch[119] Batch [300]	Speed: 189.40 samples/sec	accuracy=0.709961
INFO:root:Epoch[119] Batch [180]	Speed: 189.43 samples/sec	accuracy=0.693555

@sandeep-krishnamurthy Can you add labels

Distributed
Python
Example

Can you tell me the details of setting epoch-size? @feiyuvl
I set the epoch-size behind the train_imagenet.py. it tells:
train_imagenet.py: error: unrecognized arguments: --epoch-size 320
this docs says it's better to set epoch_size explicitly in dist_sync. but haven't tell how to set.

@XiaotaoChen sorry, mxnet has removed the epoch-size arg, which only exists in old version

And then, how to solve the problem that the total iteration of each worker is different in the current version? @feiyuvl

@XiaotaoChen This doc http://newdocs.readthedocs.io is outdated, it was created by a contributor long time ago but not updated.
I'll ping the internal team to check how to resolve it

@mli has there been any updates? This issue has also caused annoyance for me.

Hello, request you to look at the above PR as an example on how we can avoid this hang


Implemented pages on profiles leaderboard.

Should now all be implemented with a page system. Also expanded list to be 10 in length and two either side of the user. Fixes #23.

That's probably a better idea. I'll inplement that.


WalEx Source

Add WalEx as a source.

Should this be named something like postgres-cdc (opposed to polling postgres would would have a separate purpose)?


solve the unstability issues with TritonPart

Needs to be clang-format'ed

At a minimum you need to format the code

Done. (I always forget to do this ....)
Thanks,
Zhiang
On Fri, Feb 17, 2023 at 3:18 PM Matt Liberty @.***>
wrote:

At a minimum you need to format the code
[image: image]
https://urldefense.com/v3/__https://user-images.githubusercontent.com/761514/219815113-9e262d5e-75c2-4340-b9dd-f20ff6802ff8.png__;!!Mih3wA!E6BawYBopm0F6Rbu8ujubCUtYlOQ-qx4KBLgFdcZXSpnJ2_1DNU-QSbC7SgfvzvK6RjnZlyOixII1M-FbU0_ohIKIIg$
—
Reply to this email directly, view it on GitHub
https://urldefense.com/v3/__https://github.com/The-OpenROAD-Project/OpenROAD/pull/2912*issuecomment-1435386510__;Iw!!Mih3wA!E6BawYBopm0F6Rbu8ujubCUtYlOQ-qx4KBLgFdcZXSpnJ2_1DNU-QSbC7SgfvzvK6RjnZlyOixII1M-FbU0_I5EQhhk$,
or unsubscribe
https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AQR3KJWAJD6KULWNMV7SMFLWYABJNANCNFSM6AAAAAAU7CPDU4__;!!Mih3wA!E6BawYBopm0F6Rbu8ujubCUtYlOQ-qx4KBLgFdcZXSpnJ2_1DNU-QSbC7SgfvzvK6RjnZlyOixII1M-FbU0_ZwnO61Q$
.
You are receiving this because you authored the thread.Message ID:
@.***>

Looks like you need to update the unit tests:

Replaced by https://github.com/The-OpenROAD-Project-private/OpenROAD/pull/1610


add legacy rep faucet button to test v1 rep migration to v2

add legacy REP faucet button, to be used to test V1 REP to V2 migration.

pulled down the PR and built it. it looks good.


Routes cleanup

Codecov Report

Merging #57 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff          @@
##           master    #57   +/-   ##
=====================================
  Coverage     100%   100%
=====================================
  Files          35     55   +20
  Lines        1079   1055   -24
  Branches      108     94   -14
=====================================
- Hits         1079   1055   -24

Impacted Files
Coverage Δ

src/middleware/authenticate-jwt.js
100% <ø> (ø)
:arrow_up:

src/app.js
100% <ø> (ø)
:arrow_up:

src/routes/dibs/delete.js
100% <100%> (ø)

src/routes/auth.js
100% <100%> (ø)
:arrow_up:

src/routes/friendships/shared.js
100% <100%> (ø)

src/routes/gifts/post.js
100% <100%> (ø)

src/routes/users/delete.js
100% <100%> (ø)

src/routes/dibs/shared.js
100% <100%> (ø)

src/routes/friendships/get.js
100% <100%> (ø)

src/routes/gifts/patch.js
100% <100%> (ø)

... and 66 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0f9477e...73968d4. Read the comment docs.


Can't build Lean on docker / Ubuntu

Problem
Can't build Lean using the docker files or even modifying the commands to match the README instructions for linux.
How to reproduce
Trying to build Lean using docker (or Ubuntu-based OS):
docker build -t quantconnect/lean:foundation -f DockerfileLeanFoundation .
docker build -t quantconnect/lean:algorithm -f Dockerfile .

Or more specifically:
mono nuget.exe restore QuantConnect.Lean.sln -NonInteractive
xbuild QuantConnect.Lean.sln

These commands fail with several errors, from nuget version (2.8) being incompatible while some of the packages (requires > 2.12) to MSBUILD: error MSBUILD0004: Too many project files specified and 5 build errors (which I'll show up next)
Ultimately, I was able to use the linux native nuget restore command, but the xbuild QuantConnect.Lean.sln still fails with:
Errors:

/root/Lean/QuantConnect.Lean.sln (default targets) ->
(Build target) ->
/root/Lean/Algorithm/QuantConnect.Algorithm.csproj (default targets) ->
/usr/lib/mono/xbuild/14.0/bin/Microsoft.CSharp.targets (CoreCompile target) ->

	QCAlgorithm.Python.cs(475,13): error CS1502: The best overloaded method match for `QuantConnect.Algorithm.QCAlgorithm.RegisterIndicator(QuantConnect.Symbol, QuantConnect.Indicators.IndicatorBase<QuantConnect.Data.Market.IBaseDataBar>, QuantConnect.Resolution?, System.Func<QuantConnect.Data.IBaseData,QuantConnect.Data.Market.IBaseDataBar>)' has some invalid arguments
	QCAlgorithm.Python.cs(475,39): error CS1503: Argument `#2' cannot convert `QuantConnect.Indicators.FilteredIdentity' expression to type `QuantConnect.Indicators.IndicatorBase<QuantConnect.Data.Market.IBaseDataBar>'
	QCAlgorithm.Python.cs(495,13): error CS1502: The best overloaded method match for `QuantConnect.Algorithm.QCAlgorithm.RegisterIndicator(QuantConnect.Symbol, QuantConnect.Indicators.IndicatorBase<QuantConnect.Data.Market.IBaseDataBar>, QuantConnect.Resolution?, System.Func<QuantConnect.Data.IBaseData,QuantConnect.Data.Market.IBaseDataBar>)' has some invalid arguments
	QCAlgorithm.Python.cs(495,39): error CS1503: Argument `#2' cannot convert `QuantConnect.Indicators.FilteredIdentity' expression to type `QuantConnect.Indicators.IndicatorBase<QuantConnect.Data.Market.IBaseDataBar>'

/root/Lean/QuantConnect.Lean.sln (default targets) ->
(Build target) ->
/root/Lean/Brokerages/QuantConnect.Brokerages.csproj (default targets) ->
/usr/lib/mono/xbuild/14.0/bin/Microsoft.CSharp.targets (CoreCompile target) ->

	Fxcm/FxcmBrokerage.cs(170,60): error CS0117: `com.fxcm.messaging.util.IConnectionManager' does not contain a definition for `APP_INFO'

	 42 Warning(s)
	 5 Error(s)

Any ideas on how to build this successfully?

@jasantunes - You have a build error which states that you are not using the RegisterIndicator method correctly.  Could you provide an example of how you are using the RegisterIndicator method? Without using docker, can you build your code without errors?

@andrewhart098 forgot to mention. I'm trying to build master.

Ok, thanks. While we work on a PR to fix this, please take a look at this solution: https://groups.google.com/forum/#!topic/lean-engine/Ul9S2lSCgq8

@jasantunes I just tested building LEAN on a fresh Ubuntu 17.10 installation and had no xbuild errors with the latest master sources following the instructions here:
https://github.com/QuantConnect/Lean/blob/master/readme.md
There was a small change required in the LEAN instructions for Linux, updated in this PR:
https://github.com/QuantConnect/Lean/pull/1386/files

@StefanoRaggi please don't close this. Let me give you a couple reasons:

The updated instructions you mention are for Ubuntu 17.10 and not the provided Dockerfile, which is based on phusion/baseimage:0.9.22 (a docker-optimized version of Ubuntu 16.04). Unfortunately, there's no phusion/baseimage for Ubuntu 17.10 and running a full fledged Ubuntu OS inside docker is not really feasible, so I couldn't confirm that it builds on Ubuntu 17.10.

Even using the instructions you mentioned, I'm still getting the same errors on phusion/baseimage:0.9.22/Ubuntu 16.04.

At best, the Dockerfile needs to be fixed/updated, which is what I'm trying to help with.
Can anyone here successfully build master on Ubuntu 16.04? And if so, could you provide the updated instructions?
Thanks!

@jasantunes Ok, thanks for the help, sorry I misunderstood the original issue: building LEAN inside a docker container.
The provided Dockerfile was actually meant to only run LEAN (the binaries are prebuilt and copied from the host machine), so it seems it is currently configured to run LEAN but not to build it.
About the updated instructions you are correct, those were only tested on Linux, without using docker.

Ah understood now @jasantunes -- this isn't a bug :) The container contains just enough to run lean not build it. Containers have to be small as possible.
You should ideally copy the LEAN executables into them and re-build; or mount a directory with the exe's built and run them from inside the container. If you're set on building in the container you can also edit the dockerfile to add a builder installs line.


Function force push from UI in Binary Ninja throws exception

Description
[Default] Traceback (most recent call last):
[Default]   File "/Users/visual/Library/Application Support/Binary Ninja/python310/site-packages/binsync/ui/control_panel.py", line 33, in <lambda>
[Default]     menu.addAction(f"Force Push {ctx_name}@{hex(ctx_addr)}", lambda : self.controller.force_push_function(ctx_addr))
[Default] AttributeError: 'BSController' object has no attribute 'force_push_function'. Did you mean: 'force_push_functions'?

Steps to reproduce the bug
Go to a function. Open the BinSync dock. Right click the function at the bottom (next to "Connected"). Click force push. Observe that nothing happens, and error is thrown in log.
Environment
Binary Ninja 4.1.5014-dev Personal (349de5a9)
macOS 14.4.1 (23E224)
Additional context
No response

@mahaloz should be easy to fix, just a typo from what I can tell :)

Fixed in https://github.com/binsync/binsync/commit/48ecdd44bacf930cebf7df7286ad4288e9bbd6e3, on version 4.0.5


Does each global environment need cross-origin Window/WindowProxy objects or just WindowProxy objects

For Location we have the situation that different accessors get different objects. See "Return the correct Location object" in https://github.com/annevk/html-cross-origin-objects. How should this work for Window/WindowProxy?

This apparently is a non-issue. I misunderstood.


Reference gitter channel in README

You really should reference the gitter channel somewhere in the README and maybe even in the docs. I accidentially saw the channel mentioned in a closed issue, otherwise wouldn't know that it exists. So make it more present, so that people join ;)

🤦♀ of course! I can't believe I haven't added that yet, thank you!


.vscode folder without workspace settings

.vscode folder without workspace settings
As soon as I open a .c file inside a workspace in VS Code, the .vscode folder is created altough I have not set any workspace settings. The files in the .vscode folder seem to belong to the Cpp runner extension.
In most cases I use VS Code as an editor only and I do not want to have the .vscode folder. So I have to delete it very often.
Can you please fix this or tell me how to change this behaviour?

No, sorry i wont change this.
This is intended and needed behaviour of the extension


Svelte 3.24.0 seems to have broken input value binding

With two input fields in the same component, changing one seems to clear the others.
See this REPL for an example: https://svelte.dev/repl/a3a78cc7744b4b4fb4a1b4a752e9a2d3?version=3.24.0
There are a couple of <input> related PRs included in svelte 3.24.0:

https://github.com/sveltejs/svelte/pull/5004
https://github.com/sveltejs/svelte/pull/5028

Yikes that's nasty, thanks for the heads up @d0x2f .  There's some delay with response due to covid situation, but catching up here, is a general fix forthcoming with svelte, or does this call into question our usage of $$props?

Edit Input.svelte, replace line 27 with this:
const { type: _omitType, color: _omitColor, value: _omitValue, ...props } = clean($$props);
Seems like it will fix the problem

Thanks, let me try this out and look for other instances

Thanks @bestguy,
To your original question, I'm not too certain on what the behaviour of the proposed svelte PR will have in this case, I'm not familiar enough with the svelte code to tell.
A svelte dev mentioned the relatively new $$restProps var, which is the same as $$props but omits any keys that are exported by the component.
So in this case, since 'value' is exported, $$restProps will omit it by default.
This seems like the right approach to me.

relatively new $$restProps var, which is the same as $$props but omits any keys that are exported by the component.

Nice! that $$props omit stuff was a pain

An update is moving to $$restProps seems to clear issue, but 3.24.0 also breaks Inputs type="select" altogether, I need to correct that prior to merge.

Hey @d0x2f , this should be corrected in<EMAIL_ADDRESS>please update/close this issue if corrects for you.
Thanks for the heads-up

Thanks so much @bestguy, it works great. The REPL is fixed now as well.
:beers:


issue #439 - added a script for detecting the country of the phone number

I have added the feature to detect the country from the given phone numbers
using pycountry and phonenumbers i have tested it with some numbers from various countries it works good.
Please accept the PR or suggest the required changes.
Thankyou
To Run it :-
configure the GUI part and run the python script named number_mask.py
** I had to upgrade the version of networkx from exactly 2.6 to >= 2.6

Are you still working on this PR @Manav958  ?
If you are, please resolve the conflicts, update your branch and answer the questions
If you are not, please close the PR


Fix inline generation bug #103

A bug was introduced that was causing top-level schema definitions to be generated as part of the inlined generation logic. This resulted in complex polymorphic classes getting over-written with simpler inline-derived versions.
This PR removes the logic that generates referenced schemas during inline generation. It relies on the fact that these schemas will undergo regular/full generation.
Closes #103

👍

And by A bug was introduced, I off-course mean: I introduced a bug
I had so many different iterations of trying to get generation from external files working I obviously broke something

👍


Harden settings service.

Especially when notification system can't be reached.
Signed-off-by: Heiko W. Rupp<EMAIL_ADDRESS>cc @karelhala

Unable to do a real review, but definitely review to give it a try once merged

Error codes and return values look good to me!


Optimize dev Dockerfile for rebuild speed and readonly filesystem, and don't embed .npmrc

Companion PR of wwWallet/wallet-ecosystem#32.
This restructures the development Dockerfile to put node_modules in a readonly filesystem and to make maximal use of the Docker build cache. This makes the installed dependencies consistently determined by the Docker image alone, independent of any runtime state held in volumes. The optimized layer order also enables using docker-compose up --build in the parent project to automatically rebuild the image on any changes to dependencies.
Unlike wallet-backend-server, does not move dist/ outside of the sources directory, since there are relative imports that expect keys/, views/ etc. to be sibling directories of dist/.
Finally, this expects .npmrc in both prod and dev to be mounted as a secret mount only at build time, instead of embedding the GitHub token into the resulting image.

Note that the GitHub Actions build will fail for the time being, because it references the wallet-ecosystem workflow at master. See here for how to make the workflow succeed in the meantime: https://github.com/wwWallet/wallet-enterprise/compare/dockerfile-secrets...tmp/dockerfile-secrets-gha?expand=1


Update PDF styling

Ribose Document Design_v5.pdf
This updates designs for:

tables, their margins and captions
figures
“NOTEs”, “EXAMPLEs” and “source”
Lists: definition lists, numbered lists, bullet lists
Definitions section
“Yellow honeycombs” now have different opacity settings, and the text area for odd/even pages has changed.
Hyperlink style

6. “Yellow honeycombs” now have different opacity settings, and the text area for odd/even pages has changed.

@ronaldtse could you attach "honeycombs" as separated images (or attach source Indesign file)? Very hard to extract them from PDF and clean from text.

What does mean "different opacity settings"? I see that they have different colors. Or something else?
The pages started from 6 don't have "honeycombs". Should I make it in PDF also? Or all pages should have "honeycombs"?

@ronaldtse 3. what is the source xml tag for 'Entry status'?

what is the source xml tag for 'Entry status'?

I think this can be ignored.

What does mean "different opacity settings"? I see that they have different colors. Or something else?
The pages started from 6 don't have "honeycombs". Should I make it in PDF also? Or all pages should have "honeycombs"?

I have asked designer for the source. Will update.

From designer:

I think we can alternate between yellow, blue and orange honeycombs for each page

Ribose Document Design_v5.pdf
For the honeycombs, let's do this.

There are two variables: placement and color.

placement 2 configurations

Every 2 pages (even/odd: 2, 3; 4, 5; ...) is a pair
We want "opposing honeycombs" at every pair page. i.e. left bottom + right top, left top + right bottom.

color: 3 configurations

yellow, blue and orange

These variables will increment per page pairs.

i.e.

Page 2 and 3: (left bottom + right top, yellow)
Page 4 and 5: (left top + right bottom, blue)
Page 6 and 7: (left bottom + right top, orange)
Page 8 and 9: (left top + right bottom, yellow)
...

Like this:

At first glance it looks not possible. Xsl-fo has 'even' and 'odd' conditional page sequence only. I'll try to investigate how to do it via apache fop intermediate format...

Done. Apache IF is too complicated for using. Page sequences will be created on-fly via XSLT. XSLT inserts 24 page sequence 50 times (1200 pages I hope will be enough, but for >1200 page will be used layout as for first page).

Updated fonts to Open Sans.
Ribose Document Design_v6.pdf

Updated fonts to Open Sans.

Ribose xslt updated.


Randomized className attribute causes react-test-renderer snapshots to fail

Issue
When using jest and react-test-renderer to snapshot test components containing MtSvgLines, the snapshot test will always fail because a new random className is generated, regardless of if a className is specified explicitly.
Example of failing test

Proposed solution
Either add a method to disable random className generation, or add support for snapshot testing using jest and react-test-renderer.

hey @MikeLuDev, the latest 0.9.0 should fix the issue. classNames should now be predictable. see https://github.com/moarwick/react-mt-svg-lines/releases/tag/v0.9.0


⚠ Bofrost.nl has degraded performance

In c65da8f, Bofrost.nl (https://www.bofrost.nl) experienced degraded performance:

HTTP code: 200
Response time: 3628 ms

Resolved: Bofrost.nl performance has improved in 840f2db.


hangs on run

I am trying to convert these files to TIFF from VSI, directory structure is as follows:
Cells/
--410c504 D0-091617 D13_Condition1.vsi
--_410c504 D0-091617 D13_Condition1_/stack1/frame_t_0.eps

Ideally I would like single channel TIFFs output (grayscale if possible). This is my output:
(python2) Cells $python -m microscoper -f .
..._Condition1.vsi:   0%|                                           | 0/1 [00:00<?, ?it/
DAPI:   0%|                                                               | 0/3 [00:00<?, ?it/s]
T:   0%|                                                                     | 0/1 [00:00<?, ?it/s]
Z:   0%|                                                                     | 0/1 [00:00<?, ?it/s]

The images open in ImageJ using the OlympusViewer but I want to batch convert and this tool might be most useful. Thanks.

Is it possible for you to send these files to me? Dropbox or Google Drive. That would help me troubleshoot the issue.

Sure - thanks for helping out. I've attached an example image to this link. It contains the vsi file and ets file acquire from the system. It is uploading now but should be available soon (~100MB or so).

I have tested with your files, and microscoper appears to be performing slowly, but it ends up generating the TIF files. It took about 54s in my computer (64G RAM, Intel Core i7-6700K @ 8x 4.2GHz). Perhaps you can try to run the tool for a little longer?
I see that the performance is affected because of the size of the images - they're about 368MB/channel. I'm assuming that it's because you have stitched multiple images into one in CellSense. Is there any reason why you would want that instead of storing individual fields separately? It would also be more efficient (in terms of performance) to process images that are not stitched.
Meanwhile, I will see if performance can be improved for your use case. :)

Thanks for getting back to me. I've tried it on my Macbook and it looks like it is hanging. I've left it for a few hours and the progress bar doesn't update and looks like I posted above. I then tried my Ubuntu 16.04 node and although it installs in my anaconda python2.7 environment, I get a segmentation fault when running the canonical python -m microscoper -f .
\OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1031176 current, 1031176 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1031176 current, 1031176 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1031176 current, 1031176 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1031176 current, 1031176 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1031176 current, 1031176 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1031176 current, 1031176 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1031176 current, 1031176 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1031176 current, 1031176 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1031176 current, 1031176 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1031176 current, 1031176 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1031176 current, 1031176 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1031176 current, 1031176 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1031176 current, 1031176 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1031176 current, 1031176 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1031176 current, 1031176 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1031176 current, 1031176 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1031176 current, 1031176 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1031176 current, 1031176 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1031176 current, 1031176 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1031176 current, 1031176 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1031176 current, 1031176 max
Traceback (most recent call last):
  File "/home/bgriffen/anaconda3/envs/py27/lib/python2.7/runpy.py", line 163, in _run_module_as_main
    mod_name, _Error)
  File "/home/bgriffen/anaconda3/envs/py27/lib/python2.7/runpy.py", line 111, in _get_module_details
    __import__(mod_name)  # Do not catch exceptions initializing package
  File "/home/bgriffen/anaconda3/envs/py27/lib/python2.7/site-packages/microscoper/__init__.py", line 1, in <module>
    from .io import run
  File "/home/bgriffen/anaconda3/envs/py27/lib/python2.7/site-packages/microscoper/io.py", line 13, in <module>
    import bioformats as bf
  File "/home/bgriffen/anaconda3/envs/py27/lib/python2.7/site-packages/bioformats/__init__.py", line 21, in <module>
    import javabridge
  File "/home/bgriffen/anaconda3/envs/py27/lib/python2.7/site-packages/javabridge/__init__.py", line 38, in <module>
    from .jutil import start_vm, kill_vm, vm, activate_awt, deactivate_awt
  File "/home/bgriffen/anaconda3/envs/py27/lib/python2.7/site-packages/javabridge/jutil.py", line 18, in <module>
    import numpy as np
  File "/home/bgriffen/.local/lib/python2.7/site-packages/numpy/__init__.py", line 142, in <module>
    from . import add_newdocs
  File "/home/bgriffen/.local/lib/python2.7/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/home/bgriffen/.local/lib/python2.7/site-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/home/bgriffen/.local/lib/python2.7/site-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/home/bgriffen/.local/lib/python2.7/site-packages/numpy/core/__init__.py", line 16, in <module>
    from . import multiarray
KeyboardInterrupt
Segmentation fault (core dumped)

Have you seen this behavior before? Also, could you provide your environment file to check my version of numpy etc. are the same? I'm on numpy 1.11.3.

numpy==1.13.3
python-bioformats==1.3.2
tifffile==0.12.1
tqdm==4.19.5

Python 3.6.4 (default, Jan  5 2018, 02:35:40)
[GCC 7.2.1 20171224] on linux

I think something might be off with your numpy installation. Could you try building numpy from source instead of installing through pip?


Updated for Swift 3

Updated SwiftBitmask for Swift 3.  A bunch of syntax has changed but Swift 3 hasn't added anything that encompasses the functionality of SwiftBitmask— so it remains a much cleaner and better alternative to the Swift-default .rawValue crappery.

Eliminated usage of the Funky library, since it (still) hasn't been updated for Swift 3.
It's worth noting that the Swift 3 syntax for the things that Funky was doing here is surprisingly similar to Funky's syntax, and most cases more concise.  It's plausible that the authors of Funky have abandoned work on their lib because Swift 3 added enough new functional-programming stuff to satisfy these types of workflows.
As a side-effect of removing Funky, SwiftBitmask is no longer dependent upon any CocoaPods, so removed all the CocoaPods stuff.
Unfortunately, leaving a Podfile without any required pods isn't an option— pod update warns that “[!] The Podfile does not contain any dependencies.” and the build fails with “…/SwiftBitmask/Pods/Target Support Files/Pods-SwiftBitmask/Pods-SwiftBitmask-resources.sh: No such file or directory”.
Auto-updated to Swift 3 syntax using Xcode 8.2.1, and made additional by-hand Swift 3 updates.
Performed Xcode's recommended setting updates.

I also have a commit ready (that depends these changes) that makes SwiftBitmask compatible with iOS, tvOS, & watchOS.  I'll be happy to submit a PR for that if/when this Swift 3 compat is merged.

Was there a problem with this PR?  I have a few other additions I'd like to contribute, but they're dependent upon Swift 3 compatibility.
Please let me know if there's anything you'd like me to do differently, or if you have any questions.  Also, if this is a case of you moving on from this library, I'd be happy to take over maintenance and upkeep of it.


Use font-awesome icons for all menu items where appropriate

Additionally set a fixed width for all dropdown menu item icons to make all the text labels aligned.
Suggestions on alternate icons to use are welcome - I chose what made the most sense to me in most cases.
fa-cloud-meatball was chosen for "How it works" because I found the name to be the most humorous 😄

It is lovely! Now I can't live without it :)
I only have one question about the "Author" one. Have you seen address-card? I don't know if it looks better or not, so I'm just suggesting, but I'm more than extremely happy to merge as-is!

I do like the address-card better as well.
I also just realized I missed the items under the "More" menu, so I will add those as well before merging.

Author icon changed to address-card and added missing icons.
I found it hard to differentiate reset ui and reset code/ui - so any suggestions here are welcome as well.

The nbsp change is just to avoid an extra space in the underline when hovering over social links, as shown below.

vs

Looks good to me


fix: allow value with slashes in URL template

Description

The REST endpoints to translate an IBC denom hash to a denom trace and the other way around were not working properly when the URL parameter contained slashes. I found the fix here.
closes: #XXXX
Commit Message / Changelog Entry
fix: allow value with slashes in URL template

see the guidelines for commit messages. (view raw markdown for examples)

Before we can merge this PR, please make sure that all the following items have been
checked off. If any of the checklist items are not applicable, please leave them but
write a little note why.

[x] Targeted PR against correct branch (see CONTRIBUTING.md).
[ ] Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
[x] Code follows the module structure standards and Go style guide.
[ ] Wrote unit and integration tests.
[ ] Updated relevant documentation (docs/) or specification (x/<module>/spec/).
[ ] Added relevant godoc comments.
[x] Provide a commit message to be used for the changelog entry in the PR description for review.
[x] Re-reviewed Files changed in the Github PR explorer.
[ ] Review Codecov Report in the comment section below once CI passes.

@crodriguezvega This should also apply to v4, v5, and v6.


Export NodePath gets buggy in release build. (Totally fine on debug build and editor)

Godot version:
Godot 3.2.2 stable
OS/device including version:
Windows 10
Issue description:
I used export var anim_player_path: NodePath and then initialized in a variable, onready var anim_player: AnimationPlayer = get_node(anim_player_path).
and many more for my state machine (based on few nodes, 3 infact)
Now, when I instance 2-3 enemies -
It works really well and as planned in editor and debug build. The moment I export a release build based on this, after I kill one enemy using queue_free() that got lots of export nodepaths. Game just crashes. Simply crashes. Then after whole day of going through this pain, checking if there was a memory leak and all that baddies. I noticed that if I hardcode the nodepaths then exporting them, it works.
Unfortunately, this happens because the second instance of the enemy just references the first instance of enemy.
I didn't find any issues about this or maybe my search was bad. I didn't try reproducing it but my enemy is really modular. So, I can try it again if you guys ask me so!
Steps to reproduce:
Minimal reproduction project:

Duplicate of https://github.com/godotengine/godot/issues/28922. This is most likely fixed in the master branch but unfortunately, it can't be fixed in 3.2.x.


Fix a recent regression related to --pipeline-creation-jobs option.

OverrideCmdBindPipeline was added recently, but a mandatory synchronization w. asynchronous operations was dropped. that causes race-conditions and bugs when using the --pipeline-creation-jobs feature.
this is fixed here.

CI gfxreconstruct build queued with queue ID 309892.

CI gfxreconstruct build # 5394 passed.


VirusTotal detect viruses in some files

VirusTotal throws me avcodec-59.dll (Malicious.moderate.ml.score)
and scrcpy.exe (Trojan.Win64.Rozena)

#1102
What is the SHA-256 of your files?

OK, these are the correct ones for scrcpy v1.24 release.
These are detected as "malicious" by only one security vendor each (not the same):

https://www.virustotal.com/gui/file/af0114e7564ed0785cc709834ad7c130e78cdc6f2bfc357b14a2239009d5cac9
https://www.virustotal.com/gui/file/5023347055e0dc62d02d6e1d557832d61834fda26da11f43ea7eb04f224ed92d

Probably false positives.


Finetuning on own dataset

Hey there!
Congratulations on the model and thanks for open sourcing!
I would like to finetune EAT on my own dataset for audio classification. Is it possible?
If so, would the best pipeline be to just classify from the latent representations of the model using a shallow head? Or to fine-tune the whole model (or last layers) on a new dataset?
If finetuning the whole model, is it possible to freeze early layers and just adjust the last layer's weights? What would be your approach?
Thank you so much for your effort!

Yes, it is definitely possible! We have fine-tuned EAT (the pre-trained version) on other classification datasets like ESC-50 and SPC-2, and have evaluated its generalizability.
Based on my experience, fine-tuning the entire model on the new dataset tends to yield better results compared to other strategies. It is feasible to only train the last few layers, but in my previous experiments, this approach didn’t perform as well as full model fine-tuning.
I hope this helps!

Thank you so much for the response, will be closing this issue and experiment !

Hello, would you have a script for finetuning on own data? I realize there are the premade scripts for specific datasets, but no open ended version

Apologies, but at the moment, we do not have a script specifically designed for open-ended finetuning. However, you can customize your own version by referencing the code we have provided for handling specific datasets.


Can't connect with airco anymore

Hi there, I'm completely new to everything so I don't know what I should do. Suddenly one of my aircons using this integration just gave me "Unavailable" on a random day. When I try to add it again it gives me this error message:
HTTPConnectionPool(host='<IP_ADDRESS>', port=51443): Max retries exceeded with url: /beaver/command/getDeviceInfo (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f503dc30cd0>: Failed to establish a new connection: [Errno 111] Connection refused'))
I am using version 0.2.

Can't seem to add a bug label, apologies

home-assistant_mitsubishi-wf-rac_2022-12-17T10-48-43.872Z.log
Here is one of the log files.

Forgot to mention that I’m already on 0.2.0, so the fix had no effect.

Ah I’ve read the other issues and realised I’m not alone with this and that you don’t have a lot of time to look into it. Apologies, I’d love to help where possible because it’s getting impossible to trust my automations. Could try contacting someone from Mitsubishi to ask what the error message means.

Even a reset hasn’t fixed this issue, like it has for other people. I even set the airco to only be controlled locally, hoping that this would fix it. Is this also what other people have set? I did have it set to out-of-house control, might try completely deleting the app once.

It seems to always happen when trying to update the airco values.
Error: something went wrong updating the airco [Airco Office] values
Error: something went wrong updating the airco [Airco Living Room] values
Could not send airco data
Maybe the update calls need a time out too?

Ah I’ve read the other issues and realised I’m not alone with this and that you don’t have a lot of time to look into it. Apologies, I’d love to help where possible because it’s getting impossible to trust my automations. Could try contacting someone from Mitsubishi to ask what the error message means.

Hi, apologies for my late reply, but contacting Mitsubishi will not help and be a good option because we reverse engineered the app and use the api in a way we "think" is a good practice. Furthermore is the error message generated by the code from library used.

Even a reset hasn’t fixed this issue, like it has for other people. I even set the airco to only be controlled locally, hoping that this would fix it. Is this also what other people have set? I did have it set to out-of-house control, might try completely deleting the app once.

It should also work in not local mode. there is no difference in that connection there.

It seems to always happen when trying to update the airco values.
Error: something went wrong updating the airco [Airco Office] values Error: something went wrong updating the airco [Airco Living Room] values Could not send airco data
Maybe the update calls need a time out too?

It looks like the connection to the airco failed. what do you see in the app if you go to the airco->settings->users (at the bottom) I think. I cannot connect to the app at the moment because I'm not at home.

It seems to always happen when trying to update the airco values.
Error: something went wrong updating the airco [Airco Office] values Error: something went wrong updating the airco [Airco Living Room] values Could not send airco data
Maybe the update calls need a time out too?

It looks like the connection to the airco failed. what do you see in the app if you go to the airco->settings->users (at the bottom) I think. I cannot connect to the app at the moment because I'm not at home.

Should I check that setting during one of the connection problems? Right now it’s all working again and it says “Number of Smartphones: 2” but I don’t quite know how to reduce that number. In home assistant it shows as having “2 accounts”
During one of the errno 111 sessions I don’t seem to be able to connect via the Mitsubishi app either, it doesn’t show any information for the affected unit.
No worries about the late reply by the way, I’m already very grateful you can help me.

The airco’s are indeed connected right now, I’ve updated to the main branch and restarted, I’ll keep you updated on whether I see an error, happens once or twice a day.

After switching to the main branch, so far so good! Running 8 hours so far without any more issues, empty log files 👍 Will keep you updated if anything pops up

Shouldn’t have jinxed it, error happened once more

Turned off “Polling for updates” in the integration settings for both units on top of switching to the main branch and I haven’t had this error in 2 days now.

A week later, polling for updates off and running main branch and it’s been flawless. No problems anymore.


Fix decreases typo

The pipeline was fixed in this commit. Could you pls rebase :slightly_smiling_face:

@bara-m ✅ done


Bug: NaN displaying in GUI

When you set an allowance for a website and backspace any numbers in the input field, the input will then display NaN and you will not be able to continue entering numbers into the field.

I put in a PR that fixes this #392

Closed with #392


Fix TO Go GET dss missing '.json'

What does this PR do?
Fix TO Go GET dss missing '.json'
Specifically, the TO client uses ".json" in its request for this. We could change the client, but the 1.1 API defined it as accepting ".json", so the Go server will be broken and pass those requests to Perl, without this fix.
Which TC components are affected by this PR?

[ ] Documentation
[ ] Grove
[ ] Traffic Analytics
[ ] Traffic Monitor
[x] Traffic Ops
[ ] Traffic Ops ORT
[ ] Traffic Portal
[ ] Traffic Router
[ ] Traffic Stats
[ ] Traffic Vault
[ ] Other _________

What is the best way to verify this PR?
Check all that apply

[ ] This PR includes tests
[ ] This PR includes documentation updates
[ ] This PR includes an update to CHANGELOG.md
[ ] This PR includes all required license headers
[ ] This PR includes a database migration (ensure that migration sequence is correct)
[ ] This PR fixes a serious security flaw. Read more: www.apache.org/security

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/2833/
Test PASSed.


Login Page Refactor

Login Page Refactor
What user problem are we solving?

Distinctive first name and last name input boxes
Validates name inputs
Password confirmation box

What solution does this PR provide?

Allows user to input first and last name separately
Checks if first and last names only contains characters
Forces user to type in password identically twice to prevent misinput

Testing Methodology

Clicked through registration page several times
Functionality is not broken

Any other considerations

@alantao912 can you attach a screen recording that walks through the various edge cases

Login Page Refactor
What user problem are we solving?

Distinctive first name and last name input boxes
Validates name inputs
Password confirmation box

What solution does this PR provide?

Allows user to input first and last name separately
Checks if first and last names only contains characters
Forces user to type in password identically twice to prevent misinput

Testing Methodology

Clicked through registration page several times
Functionality is not broken

Any other considerations

can you create a github issue and link it to the PR description so that we can track the work

@karkir0003 Demo video
https://github.com/DSGT-DLP/Deep-Learning-Playground/assets/70001632/4917d351-7ec8-47fe-b7b6-a4dcc43083bc

@karkir0003 Demo video https://github.com/DSGT-DLP/Deep-Learning-Playground/assets/70001632/4917d351-7ec8-47fe-b7b6-a4dcc43083bc

Very good!!!
Couple questions mainly on toast errors:

if I enter a first name or last name like "123" and it's invalid, we should make the toast message say something like "invalid first name/last name. please enter only letters"
for the error message about "firebase auth error: user in use, we should make this message a bit cleaner on the UI side"


WIP: [BIRD-T-41] Match page's

📝Changed routes:

/candidates

(view full size)

Commit 664bcac1f94466f7e3a7153aca54e587a14a7345 (https://bird-bvz84dbdq-hiddentity.vercel.app).