MemPool.datastore memory utilization keeps increasing when using DTables with multiple processes

I have the following setup:

Start Julia with julia --project -t1 --heap-size-hint=3G
Add 4 processes with addprocs(4; exeflags = "--heap-size-hint=3G")
Worker 1 receives a query request and then tells worker 2 to do the work

The actual query includes loading a table from a .csv file into a DTable (with a DataFrame table type). Operations include selecting columns, fetching the table into a DataFrame for adding/removing rows/columns and other processing as needed, and re-wrapping the table in a DTable to later be processed further. At the end of processing, the result is returned as a DataFrame.
The .csv file contains a table with 233930 rows and 102 columns: 1 column of InlineStrings.String15, 2 columns of InlineStrings.String1, 45 columns of Int64, and 54 columns of Float64.
The issue: I noticed that if I keep running the same query repeatedly, the MemPool.datastore on worker 2 consumes more and more memory, as determined by
remotecall_fetch(2) do
    Base.summarysize(MyPackage.Dagger.MemPool.datastore)
end

Eventually, the memory usage grows enough to cause my WSL 2 Linux OOM manager to kill worker 2, crashing my program.
Notably, I do not observe this growth in memory usage in the following scenarios:

when running everything on a single process (i.e., not calling addprocs), or
when using DataFrames exclusively (i.e., not using DTables.jl at all).

I do observe this growth in memory usage in the following additional scenarios:

when using NamedTuple as the table type for the DTables, or
when running everything on a single process, but with multiple processes available. (To clarify, my code exclusively uses worker 1 in this scenario, but it appears DTables.jl/Dagger.jl uses the other available workers. And in this case the MemPool.datastore on worker 1 (not worker 2) is what consumes more and more memory. However, I never ran into any issues with the OOM manager killing my processes.)

I'm posting this issue in DTables.jl in case there's something DTables.jl is doing that somehow causes the MemPool.jl data store to keep references around longer than expected, but of course please transfer this issue to Dagger.jl or MemPool.jl as needed.
Please let me know if there is any other information that would help with finding the root cause of this issue.

Do you have a reproducer for this one, just to help me debug it reliably?

I am working on a better reproducer, but I believe the behavior I pointed out in JuliaParallel/Dagger.jl#445 (related to JuliaParallel/Dagger.jl#438) is essentially the same I am reporting here---all those CPURAMDevices needed to be evicted when Julia closed because they were not being evicted earlier. It's just that before I wasn't running my code enough times to get to the point where the OOM manager killed one of my processes.

@jpsamaroo Here's a MWE that shows the ever-growing memory utilization. It took me running main about 120 times to get the OOM manager to kill worker 2 (I have 32 GB RAM). Let me know if you need data as well (i.e., "file.csv").
using Distributed
addprocs(5 - nprocs(); exeflags = "--heap-size-hint=3G")

@everywhere using DTables, DataFrames, CSV

@everywhere const DT = Ref{DTable}()

@everywhere mutable struct DTableCols
    key_names
    value_names
    keys
    values
end

function main()

    remotecall_fetch(query, 2)

end

@everywhere function query()

    dt1 = load_dt()
    dt2 = add_value_col!(dt1)
    dt3 = update_value_col!(dt2)
    @info "" length(dt3)
    dt4 = calc_value_cols(dt3)
    dt5 = select(dt4, [6; 12; 103:113]...; copycols = false)
    dt_agg = aggregate_dt(dt5)
    return fetch(dt_agg)

end

@everywhere function load_dt()

    isassigned(DT) && return DT[]
    file = "file.csv"
    GC.enable(false)
    dt = DTable(x -> CSV.File(x), [file]; tabletype = DataFrame)
    GC.enable(true)
    DT[] = dt
    return dt

end

@everywhere function add_value_col!(dt)

    dt_cols = create_dt_cols(dt, 1:48, 49:102)
    dt_cols.value_names = [dt_cols.value_names; "RAND"]
    dt_cols.values = (dt_cols.values..., rand(length(dt_cols.values[1])))
    return create_dt_from_cols(dt_cols; is_sorted = true)

end

@everywhere function create_dt_cols(dt, key_cols, value_cols)

    df = fetch(dt)
    key_names = names(df)[key_cols]
    value_names = names(df)[value_cols]
    keys = [df[!, i] for i in key_cols]
    values = [df[!, i] for i in value_cols]
    return DTableCols(key_names, value_names, keys, values)

end

@everywhere function create_dt_from_cols(dt_cols; is_sorted = false)

    df = DataFrame(
        (dt_cols.key_names .=> dt_cols.keys)...,
        (dt_cols.value_names .=> dt_cols.values)...;
        copycols = false,
    )
    is_sorted || sort!(df)
    return DTable(df)

end

@everywhere function update_value_col!(dt)

    dt_cols = create_dt_cols(dt, 1:48, 49:103)
    dt_cols.values = (
        dt_cols.values[1:10]...,
        rand(length(dt_cols.values[1])),
        dt_cols.values[12:end]...,
    )
    return create_dt_from_cols(dt_cols; is_sorted = true)

end

@everywhere function calc_value_cols(dt)

    newvals = Vector{Float64}[]
    for i = 1:10
        v = calc_new_value(dt, i)
        push!(newvals, v)
    end
    return append_value_cols(dt, newvals)

end

@everywhere function calc_new_value(dt, i)

    dt_cols = create_dt_cols(dt, 1:48, 49:103)
    return abs.(dt_cols.values[i])

end

@everywhere function append_value_cols(dt, newvals)

    df = fetch(dt)
    for (i, v) in enumerate(newvals)
        setproperty!(df, "NEW$i", v)
    end
    return DTable(df)

end

@everywhere function aggregate_dt(dt)

    key_names = [Symbol("6"), Symbol("12")]
    gdt = groupby(fetch(dt), key_names)
    gkeys = sort!(collect(keys(gdt)))
    key_pairs = key_names .=> invert(gkeys)
    value_names = [[Symbol("RAND")]; Symbol.("NEW", 1:10)]
    sums = fetch(reduce(+, gdt; cols = value_names))
    sorted = sortperm(invert(sums[key_names]))
    value_pairs = map(value_names) do value
        value => sums[Symbol(:result_, value)][sorted]
    end
    return DTable(DataFrame(key_pairs..., value_pairs...))

end

@everywhere invert(x) = [[x[j][i] for j = 1:length(x)] for i = 1:length(x[1])]

@everywhere function Base.reduce(f, df::DataFrames.AbstractDataFrame; cols)

    NamedTuple(col => reduce(f, df[!, col]) for col in cols)

end

@everywhere function Base.reduce(f, gdt::DataFrames.GroupedDataFrame; cols)

    gkeys = keys(gdt)
    dims = keys(gkeys[1])
    merge(
        NamedTuple(dim => getproperty.(gkeys, dim) for dim in dims),
        NamedTuple(
            Symbol(:result_, col) => [reduce(f, gdt[k]; cols = [col])[col] for k in gkeys]
            for col in cols
        ),
    )

end

One thing I remembered was that when I was benchmarking dtables.jl around release time I had a really bad time with running it in wsl2.
I would barely get to a quarter of the table size which I could run successfully on Linux due to the weird memory management wsl does
Let's keep this in mind when looking at this, Linux will behave differently for sure. I'll try to have a look at it this week

I just ran the exact same code as in #61 (which in turn is the same as the MWE above but with a call to enable_disk_caching(50, 2^10 * 20)). This time, instead of a BoundsError, I got AssertionError: Failed to migrate 183.839 MiB for ref 1624:
julia> include("mwe.jl"); for i = 1:100 (i % 10 == 0 && @show(i)); main() end
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
i = 10
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
i = 20
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
i = 30
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
ERROR: On worker 2:
AssertionError: Failed to migrate 183.839 MiB for ref 1624
Stacktrace:
  [1] #105
    @ ~/.julia/packages/MemPool/l9nLj/src/storage.jl:887
  [2] with_lock
    @ ~/.julia/packages/MemPool/l9nLj/src/lock.jl:80
  [3] #sra_migrate!#103
    @ ~/.julia/packages/MemPool/l9nLj/src/storage.jl:849
  [4] sra_migrate!
    @ ~/.julia/packages/MemPool/l9nLj/src/storage.jl:826 [inlined]
  [5] write_to_device!
    @ ~/.julia/packages/MemPool/l9nLj/src/storage.jl:817
  [6] #poolset#160
    @ ~/.julia/packages/MemPool/l9nLj/src/datastore.jl:386
  [7] #tochunk#139
    @ ~/.julia/packages/Dagger/M13n0/src/chunks.jl:267
  [8] tochunk (repeats 2 times)
    @ ~/.julia/packages/Dagger/M13n0/src/chunks.jl:259 [inlined]
  [9] #DTable#1
    @ ~/.julia/packages/DTables/BjdY2/src/table/dtable.jl:38
 [10] DTable
    @ ~/.julia/packages/DTables/BjdY2/src/table/dtable.jl:28
 [11] #create_dt_from_cols#9
    @ ~/tmp/mwe.jl:76
 [12] create_dt_from_cols
    @ ~/tmp/mwe.jl:68 [inlined]
 [13] add_value_col!
    @ ~/tmp/mwe.jl:53
 [14] query
    @ ~/tmp/mwe.jl:26
 [15] #invokelatest#2
    @ ./essentials.jl:819 [inlined]
 [16] invokelatest
    @ ./essentials.jl:816
 [17] #110
    @ ~/programs/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:285
 [18] run_work_thunk
    @ ~/programs/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:70
 [19] macro expansion
    @ ~/programs/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:285 [inlined]
 [20] #109
    @ ./task.jl:514
Stacktrace:
 [1] remotecall_fetch(::Function, ::Distributed.Worker; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Distributed ~/programs/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:465
 [2] remotecall_fetch(::Function, ::Distributed.Worker)
   @ Distributed ~/programs/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:454
 [3] #remotecall_fetch#162
   @ ~/programs/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:492 [inlined]
 [4] remotecall_fetch
   @ ~/programs/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:492 [inlined]
 [5] main
   @ ~/tmp/mwe.jl:19 [inlined]
 [6] top-level scope
   @ ./REPL[1]:1

I wonder if this is essentially the same issue as the OP, where data is being kept longer than it should. Just in this case, instead of a process getting killed, MemPool errors because we end up exceeding the 20 GB of disk space I said MemPool could use. If it is the same issue, then WSL 2 memory management shouldn't have anything to do with this.

I ran the MWE (with and without enabling disk caching) on Windows (not WSL 2).

Without disk caching: I ran main about 200 times and never ran into any memory issues.
With enable_disk_caching!(50, 2^10 * 20): I could not replicate the AssertionError: Failed to migrate, but I did always get the BoundsError mentioned in #61.

So there definitely is a difference in behavior between WSL 2 and Windows.

Without disk caching enabled:

WSL 2 still experiences the situation where the OOM manager kills my Julia process due to the ever-growing memory utilization of MemPool.datastore.
Windows seems fine.

With enable_disk_caching!(50, 2^10 * 20):

The AssertionError: Failed to migrate still occurs on both WSL 2 and Windows (though it does take many more calls to main than before to experience the error).

So, it looks like the issue is not entirely resolved yet.

@StevenWhitaker when this occurs, how much does Base.format_bytes(MemPool.GLOBAL_DEVICE[].device_size) report, and what amount of memory is reported by the OS as being used by Julia (out of the total system RAM)? If MemPool's LRU truly thinks it's running out of memory, then this will trigger (I probably should make the error more informative).

@jpsamaroo I ran the following code to grab the info you requested. Let me know if you need any other info.
julia> include("mwe.jl"); for i = 1:200
           totalmem = Base.format_bytes(Sys.total_physical_memory())
           memusage = map(procs()) do id
               remotecall_fetch(id) do
                   parse(Int, split(read(`ps -p $(getpid()) -o rss`, String), "\n")[end-1]) * 1000
               end
           end
           totalmemusage = Base.format_bytes(sum(memusage))
           worker_memusage = Base.format_bytes(memusage[2])
           device_size = remotecall_fetch(2) do
               DTables.Dagger.MemPool.GLOBAL_DEVICE[].device_size[]
           end |> Base.format_bytes # Remove `device_size` when disk caching not enabled
           @info "" i device_size worker_memusage totalmemusage totalmem
           main()
       end

Without disk caching enabled on WSL 2:

Output before worker 2 getting killed:┌ Info:
│   i = 131
│   worker_memusage = "26.100 GiB"
│   totalmemusage = "29.153 GiB"
└   totalmem = "30.994 GiB"

With enable_disk_caching!(50, 2^10 * 20) on WSL 2:

Output before the AssertionError:┌ Info:
│   i = 117
│   device_size = "19.417 GiB"
│   worker_memusage = "7.348 GiB"
│   totalmemusage = "13.088 GiB"
└   totalmem = "30.994 GiB"

After the error:┌ Info:
│   device_size = "19.590 GiB"
│   worker_memusage = "7.283 GiB"
│   totalmemusage = "13.040 GiB"
└   totalmem = "30.994 GiB"

So, it looks like the device is running out of memory, but why?

Can you try throwing in some @everywhere GC.gc() calls every iteration and see if that delays or eliminates the OOM situation? If so, then it means that we should start automatically calling the GC to reduce memory usage when we start hitting RAM limits.

I added @everywhere GC.gc() at the start of each iteration. It delayed the OOM issue until iteration 300, but the Julia process still was killed.

Any updates yet on this front?

@StevenWhitaker can you try out https://github.com/JuliaData/MemPool.jl/pull/75 and see if it delays the OOM further or eliminates it? You'll probably want to fiddle with the MemPool.MEM_RESERVED[] value to see which avoids the OOM best while minimizing overhead - it's set to 512MB to be conservative, but 2GB works on my machine better (since I run earlyoom which is more eager than the Linux kernel's OOM killer).

The error shown should be fixed by https://github.com/JuliaParallel/Dagger.jl/issues/450#issuecomment-1812894331, you need to be using Julia 1.11 with my Distributed PR at all times for reliable distributed computing.
And yes, it doesn't yet handle disk caching, but I can add support for that (we'll nicely ask the LRU to swap more stuff to disk if it can).

Oh, for some reason I thought JuliaLang/Distributed.jl#4 was only relevant if running code with multiple threads, but I'm running julia with -t1. But I will try again tomorrow with that fix.
With disk caching enabled, as far as I can tell, my issue isn't running out of RAM, it's running out of allotted disk space. So the LRU needs to delete stuff that is no longer needed, but it seems like that isn't happening.

My reproducer still fails.
With JuliaLang/Distributed.jl#4 and JuliaData/MemPool.jl#75 and the most recent nightly build of Julia:
`MemPool.MEM_RESERVED[]`: Default

$ ~/programs/julia/julia-9fc1b653c4/bin/julia --project -t1 --heap-size-hint=3G
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-DEV.1105 (2023-12-15)
 _/ |\__'_|_|_|\__'_|  |  Commit 9fc1b653c43 (0 days old master)
|__/                   |

julia> include("mwe.jl"); @show(Base.format_bytes(DTables.Dagger.MemPool.MEM_RESERVED[])); for i = 1:300 (i % 10 == 0 && @show(i)); main() end
Base.format_bytes(DTables.Dagger.MemPool.MEM_RESERVED[]) = "512.000 MiB"
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    GC error (probable corruption)
      From worker 2:    Allocations: 191540514 (Pool: 191538428; Big: 2086); GC: 3793
      From worker 2:    <?#0x7fcfcafd02e0::Memory{
      From worker 2:    !!! ERROR in jl_ -- ABORTING !!!
       for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-DEV.1105 (2023-12-15)
 _/ |\__'_|_|_|\__'_|  |  Commit 9fc1b653c43 (0 days old master)
|__/                   |

julia> include("mwe.jl"); @show(Base.format_bytes(DTables.Dagger.MemPool.MEM_RESERVED[])); for i = 1:300 (i % 10 == 0 && @show(i)); main() end
Base.format_bytes(DTables.Dagger.MemPool.MEM_RESERVED[]) = "1.863 GiB"
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    GC error (probable corruption)
      From worker 2:    Allocations: 191545919 (Pool: 191543833; Big: 2086); GC: 3751
      From worker 2:    <?#0x7f7c13df0180::Memory{
      From worker 2:    !!! ERROR in jl_ -- ABORTING !!!
       for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-DEV.1105 (2023-12-15)
 _/ |\__'_|_|_|\__'_|  |  Commit 9fc1b653c43 (0 days old master)
|__/                   |

julia> include("mwe.jl"); @show(Base.format_bytes(DTables.Dagger.MemPool.MEM_RESERVED[])); for i = 1:300 (i % 10 == 0 && @show(i)); main() end
Base.format_bytes(DTables.Dagger.MemPool.MEM_RESERVED[]) = "9.313 GiB"
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:
      From worker 2:    [5814] signal 11 (1): Segmentation fault
      From worker 2:    in expression starting at none:0
      From worker 2:    Allocations: 191516083 (Pool: 191513997; Big: 2086); GC: 6403
ERROR: ┌ Warning: Worker 2 died, rescheduling work
└ @ Dagger.Sch ~/.julia/packages/Dagger/lhyAj/src/sch/Sch.jl:546
Worker 2 terminated.ProcessExitedException┌ Warning: Worker 2 died, rescheduling work
└ @ Dagger.Sch ~/.julia/packages/Dagger/lhyAj/src/sch/Sch.jl:546

(┌ Error: Error assigning workers
│   exception =
│    ProcessExitedException(2)
│    Stacktrace:
│     [1] worker_from_id(pg::Distributed.ProcessGroup, i::Int64)
│       @ Distributed ~/tmp/distributed_fix/Distributed.jl/src/cluster.jl:1121
│     [2] worker_from_id(pg::Distributed.ProcessGroup, i::Int64)
│       @ Distributed ~/tmp/distributed_fix/Distributed.jl/src/cluster.jl:1118 [inlined]
│     [3] remote_do
│       @ ~/tmp/distributed_fix/Distributed.jl/src/remotecall.jl:557 [inlined]
│     [4] cleanup_proc(state::Dagger.Sch.ComputeState, p::Dagger.OSProc, log_sink::TimespanLogging.NoOpLog)
│       @ Dagger.Sch ~/.julia/packages/Dagger/lhyAj/src/sch/Sch.jl:409
│     [5] monitor_procs_changed!(ctx::Dagger.Context, state::Dagger.Sch.ComputeState)
│       @ Dagger.Sch ~/.julia/packages/Dagger/lhyAj/src/sch/Sch.jl:887
│     [6] (::Dagger.Sch.var"#100#102"{Dagger.Context, Dagger.Sch.ComputeState})()
│       @ Dagger.Sch ~/.julia/packages/Dagger/lhyAj/src/sch/Sch.jl:509
└ @ Dagger.Sch ~/.julia/packages/Dagger/lhyAj/src/sch/Sch.jl:511
2Unhandled Task ERROR: EOFError: read end of file
Stacktrace:
 [1] (::Base.var"#wait_locked#797")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
   @ Base ./stream.jl:950
 [2] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
   @ Base ./stream.jl:958
 [3] unsafe_read
   @ ./io.jl:882 [inlined]
 [4] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
   @ Base ./io.jl:881
 [5] read!
   @ ./io.jl:886 [inlined]
 [6] deserialize_hdr_raw
   @ ~/tmp/distributed_fix/Distributed.jl/src/messages.jl:167 [inlined]
 [7] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed ~/tmp/distributed_fix/Distributed.jl/src/process_messages.jl:172
 [8] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed ~/tmp/distributed_fix/Distributed.jl/src/process_messages.jl:133
 [9] (::Distributed.var"#113#114"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
   @ Distributed ~/tmp/distributed_fix/Distributed.jl/src/process_messages.jl:121
)
Stacktrace:
  [1] try_yieldto(undo::typeof(Base.ensure_rescheduled))
    @ Base ./task.jl:944
  [2] wait()
    @ Base ./task.jl:1008
  [3] wait(c::Base.GenericCondition{ReentrantLock}; first::Bool)
    @ Base ./condition.jl:130
  [4] wait
    @ Base ./condition.jl:125 [inlined]
  [5] take_buffered(c::Channel{Any})
    @ Base ./channels.jl:477
  [6] take!(c::Channel{Any})
    @ Base ./channels.jl:471
  [7] take!(::Distributed.RemoteValue)
    @ Distributed ~/tmp/distributed_fix/Distributed.jl/src/remotecall.jl:726
  [8] remotecall_fetch(::Function, ::Distributed.Worker; kwargs::@Kwargs{})
    @ Distributed ~/tmp/distributed_fix/Distributed.jl/src/remotecall.jl:461
  [9] remotecall_fetch(::Function, ::Distributed.Worker)
    @ Distributed ~/tmp/distributed_fix/Distributed.jl/src/remotecall.jl:454
 [10] remotecall_fetch
    @ ~/tmp/distributed_fix/Distributed.jl/src/remotecall.jl:492 [inlined]
 [11] main
    @ ~/tmp/distributed_fix/mwe.jl:19 [inlined]
 [12] top-level scope
    @ ./REPL[1]:1

I can try with a different Julia commit if you think the issue here is the particular commit I used. (And let me know if you have a commit in mind.)

@jpsamaroo Happy Holidays! Any updates on this front?
And to reiterate some questions I had:

Oh, I thought https://github.com/JuliaLang/Distributed.jl/pull/4 was only relevant if running code with multiple threads, but I'm running julia with -t1.

Is my assumption wrong, i.e., even with one thread we need that Distributed.jl fix?

With disk caching enabled, as far as I can tell, my issue isn't running out of RAM, it's running out of allotted disk space. So the LRU needs to delete stuff that is no longer needed, but it seems like that isn't happening.

Any thoughts on this?

I tried a couple of variations of my reproducer to collect a few more data points in case it helps with this issue:
I realized that I accidentally had been using multiple threads on the worker processes (because I didn't realize the -t 1 option didn't automatically propagate to workers added with addprocs). So, I updated my reproducer to include "-t 1" in the exeflags kwarg of addprocs. I then ran the reproducer both with and without disk caching, as before:

Without disk caching: No OOM process killing, but WSL 2 ground to a snail's pace after about 300 calls to main due to running out of memory and swap.
With disk caching: The same AssertionError: Failed to migrate as before occurred after 303 calls to main.

I then decided to remove the two GC.enable lines in load_dt:

Without disk caching: No OOM process killing after 300 calls to main, but memory and swap usage was steadily rising.
With disk caching: The same AssertionError: Failed to migrate as before occurred after 239 calls to main.

I then tried wrapping the call to query in main with Dagger.with_options(query; scope = ProcessScope(myid())):

Without disk caching: Process killed after 60 calls to main.
With disk caching: The same AssertionError: Failed to migrate as before occurred after 39 calls to main.

There are no memory or disk caching problems when I don't add any processes, regardless of whether or not multiple threads are used.
TL;DR Everything I tried still resulted in failure, except removing processes altogether.

Happy Holidays!

Is my assumption wrong, i.e., even with one thread we need that Distributed.jl fix?

Probably yes, as it may still be a race with multiple async tasks. I haven't really experienced that, but it can probably still occur.

Any thoughts on this?

I still need to implement this in https://github.com/JuliaData/MemPool.jl/pull/75 - I have a long TODO list from before the holidays, so I'm slowly working down it. Thanks for you patience :smile:
W.r.t the non-disk-caching OOMs, I've put together https://github.com/JuliaData/MemPool.jl/pull/76, which together with https://github.com/JuliaData/MemPool.jl/pull/75 and https://github.com/JuliaParallel/Dagger.jl/tree/jps/chained-dtors significantly reduces the amount of memory that Dagger keeps around, and also forces GC calls when we're running out of memory (tunable with the new JULIA_MEMPOOL_MEMORY_RESERVED env var, or MemPool.MEM_RESERVED[]). I haven't tested it heavily yet with DTables, but I've seen some solid improvements with DArray operations. At least for single worker tests, I see very consistent, much lower memory usage, rather than the wild sawtooth pattern that we're used to seeing.
I'll keep at this, but thank you for the detailed updates and patience while I work through these issues!

Thanks for your work, and no worries about having a long TODO list!
I just tried adding JuliaData/MemPool.jl#75 and https://github.com/JuliaParallel/Dagger.jl/tree/jps/chained-dtors to my environment, in conjunction with DTables v0.4.3, and DTables failed to precompile (segfault). I saw this on Julia 1.9.4 and 1.10.0. (But it precompiled fine on 1.11 (2024-01-11 nightly), even without your Distributed fix.) Any thoughts on why DTables would fail to precompile?

Odd, can you provide a stacktrace of the segfault?

In Julia 1.10.0, but it looked the same in Julia 1.9.4.

  For a report of the errors see `julia> err`. To retry use `pkg> precompile`

And here's the segfault that comes when calling precompile:
Segfault

(chained-dtors) pkg> precompile
Precompiling project...
  ✗ DTables
  0 dependencies successfully precompiled in 1 seconds. 53 already precompiled.

ERROR: The following 1 direct dependency failed to precompile:

DTables [20c56dc6-594c-4682-91cf-1d46875b1eba]

Failed to precompile DTables [20c56dc6-594c-4682-91cf-1d46875b1eba] to "/home/steven/.julia/compiled/v1.10/DTables/jl_f6HpNj".
[1629] signal (6.-6): Aborted
in expression starting at /home/steven/.julia/packages/DTables/EiSy4/src/DTables.jl:7
pthread_kill at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
raise at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
get_item_for_reloc at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/staticdata.c:1798 [inlined]
jl_read_reloclist at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/staticdata.c:1874
jl_restore_system_image_from_stream_ at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/staticdata.c:2996
jl_restore_package_image_from_stream at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/staticdata.c:3418
jl_restore_incremental_from_buf at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/staticdata.c:3465
ijl_restore_package_image_from_file at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/staticdata.c:3549
_include_from_serialized at ./loading.jl:1052
_require_search_from_serialized at ./loading.jl:1575
_require at ./loading.jl:1932
__require_prelocked at ./loading.jl:1806
jfptr___require_prelocked_80742.1 at /home/steven/programs/julia/julia-1.10.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
jl_f__call_in_world at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/builtins.c:831
#invoke_in_world#3 at ./essentials.jl:921 [inlined]
invoke_in_world at ./essentials.jl:918 [inlined]
_require_prelocked at ./loading.jl:1797
macro expansion at ./loading.jl:1784 [inlined]
macro expansion at ./lock.jl:267 [inlined]
__require at ./loading.jl:1747
jfptr___require_80707.1 at /home/steven/programs/julia/julia-1.10.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
jl_f__call_in_world at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/builtins.c:831
#invoke_in_world#3 at ./essentials.jl:921 [inlined]
invoke_in_world at ./essentials.jl:918 [inlined]
require at ./loading.jl:1740
jfptr_require_80704.1 at /home/steven/programs/julia/julia-1.10.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
call_require at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/toplevel.c:481 [inlined]
eval_import_path at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/toplevel.c:518
eval_import_from at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/toplevel.c:635 [inlined]
eval_import_from at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/toplevel.c:626
jl_toplevel_eval_flex at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/toplevel.c:742
jl_eval_module_expr at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/toplevel.c:215 [inlined]
jl_toplevel_eval_flex at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/toplevel.c:736
jl_toplevel_eval_flex at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/toplevel.c:877
ijl_toplevel_eval_in at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/toplevel.c:985
eval at ./boot.jl:385 [inlined]
include_string at ./loading.jl:2070
_jl_invoke at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/gf.c:3076
_include at ./loading.jl:2130
include at ./Base.jl:495 [inlined]
include_package_for_output at ./loading.jl:2216
jfptr_include_package_for_output_80987.1 at /home/steven/programs/julia/julia-1.10.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
do_call at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/interpreter.c:126
eval_value at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/interpreter.c:223
eval_stmt_value at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/interpreter.c:174 [inlined]
eval_body at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/interpreter.c:617
jl_interpret_toplevel_thunk at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/interpreter.c:775
jl_toplevel_eval_flex at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/toplevel.c:934
jl_toplevel_eval_flex at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/toplevel.c:877
ijl_toplevel_eval_in at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/toplevel.c:985
eval at ./boot.jl:385 [inlined]
include_string at ./loading.jl:2070
include_string at ./loading.jl:2080 [inlined]
exec_options at ./client.jl:316
_start at ./client.jl:552
jfptr__start_82703.1 at /home/steven/programs/julia/julia-1.10.0/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
true_main at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/jlapi.c:582
jl_repl_entrypoint at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/src/jlapi.c:731
main at /cache/build/builder-amdci4-6/julialang/julia-release-1-dot-10/cli/loader_exe.c:58
unknown function (ip: 0x7f524eabcd8f)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8)
Allocations: 2906 (Pool: 2897; Big: 9); GC:

Yeah I'm dealing with these errors too now that I've opened the PR. I'll get those fixed and then let you know when you can try it.

@jpsamaroo I just saw JuliaLang/julia#40626. Does this issue affect Dagger.jl at all, i.e., could this issue be the cause of what I'm seeing with memory seeming like it doesn't get freed?

Huh, I hadn't thought of that, but now that you mention it, every Dagger task does rely on fetching the result of a Julia Task to get the Dagger task's return value (https://github.com/JuliaParallel/Dagger.jl/blob/2110d621212591b6652cfeb8d6ba668255358d10/src/processor.jl#L164-L172). That could certainly cause over-preservation of task results. I'll investigate this week and get back to you!
Also, the jps/chained-dtors branch has been merged into Dagger master (with fixes for the precompile errors), so please give that a try when you get a chance!

@jpsamaroo I've been trying several different things, so here's a report of my findings.

chained-dtors: Without disk caching, this did seem to improve memory utilization (I set MEM_RESERVED[] to 4 GiB), meaning I don't think I ever saw OOM killing in this case. I did notice some strange memory reporting, though (see below). With disk caching I still experienced the AssertionError (I know that wasn't the focus for this change though).

Strange memory reporting: I don't think this was specific to the chained-dtors change (but I didn't specifically test without it). I called main from my MWE 50 times, ran @everywhere GC.gc() five times, and then looked at memory utilization. Calling Base.summarysize on MemPool.datastore on worker 2 reported 1.618 GiB, while the top command reported 13.1 GiB for that process. And I couldn't find any other Julia variable(s) that could account for the additional ~11 GiB top thought Julia was using. I'm not really sure what to make of this, but I thought it might be useful for you to know.

Comparison to MemPool.jl: In my application I don't really need or want the parallelism provided by Dagger.jl; I basically am using DTables.jl just for DataFrames.jl + disk caching. So, I decided to do a direct comparison of using a DTable vs creating my own DCTable that just wraps a DRef (that refers to a DataFrame). I'll post the actual code below, but the upshot is that using my custom DCTable is significantly faster and uses much less memory. Is that to be expected, i.e., does DTables.jl/Dagger.jl really come with that much overhead?

Comparison to MemPool.jl
`dtables.jl` (no significant changes to the original MWE I posted)

@everywhere using DTables, DataFrames, CSV

if @isdefined(USE_DISK_CACHING) && USE_DISK_CACHING
    @info "disk caching enabled"
    enable_disk_caching!(8, 20 * 2^10) # 8% max memory to lead to ~512 MiB per process to match custom code
else
    @info "no disk caching"
end

@everywhere const DT = Ref{DTable}()

@everywhere mutable struct DTableCols
    key_names
    value_names
    keys
    values
end

function main()

    remotecall_fetch(query, 2)

end

@everywhere function query()

    dt1 = load_dt()
    dt2 = add_value_col!(dt1)
    dt3 = update_value_col!(dt2)
    @info "" length(dt3)
    dt4 = calc_value_cols(dt3)
    dt5 = select(dt4, [6; 12; 103:113]...; copycols = false)
    dt_agg = aggregate_dt(dt5)
    return fetch(dt_agg)

end

@everywhere function load_dt()

    isassigned(DT) && return DT[]
    file = "file.csv"
    dt = DTable(x -> CSV.File(x), [file]; tabletype = DataFrame)
    DT[] = dt
    return dt

end

@everywhere function add_value_col!(dt)

    dt_cols = create_dt_cols(dt, 1:48, 49:102)
    dt_cols.value_names = [dt_cols.value_names; "RAND"]
    dt_cols.values = (dt_cols.values..., rand(length(dt_cols.values[1])))
    return create_dt_from_cols(dt_cols; is_sorted = true)

end

@everywhere function create_dt_cols(dt, key_cols, value_cols)

    df = fetch(dt)
    key_names = names(df)[key_cols]
    value_names = names(df)[value_cols]
    keys = [df[!, i] for i in key_cols]
    values = [df[!, i] for i in value_cols]
    return DTableCols(key_names, value_names, keys, values)

end

@everywhere function create_dt_from_cols(dt_cols; is_sorted = false)

    df = DataFrame(
        (dt_cols.key_names .=> dt_cols.keys)...,
        (dt_cols.value_names .=> dt_cols.values)...;
        copycols = false,
    )
    is_sorted || sort!(df)
    return DTable(df)

end

@everywhere function update_value_col!(dt)

    dt_cols = create_dt_cols(dt, 1:48, 49:103)
    dt_cols.values = (
        dt_cols.values[1:10]...,
        rand(length(dt_cols.values[1])),
        dt_cols.values[12:end]...,
    )
    return create_dt_from_cols(dt_cols; is_sorted = true)

end

@everywhere function calc_value_cols(dt)

    newvals = Vector{Float64}[]
    for i = 1:10
        v = calc_new_value(dt, i)
        push!(newvals, v)
    end
    return append_value_cols(dt, newvals)

end

@everywhere function calc_new_value(dt, i)

    dt_cols = create_dt_cols(dt, 1:48, 49:103)
    return abs.(dt_cols.values[i])

end

@everywhere function append_value_cols(dt, newvals)

    df = fetch(dt)
    for (i, v) in enumerate(newvals)
        setproperty!(df, "NEW$i", v)
    end
    return DTable(df)

end

@everywhere function aggregate_dt(dt)

    key_names = [Symbol("6"), Symbol("12")]
    gdt = groupby(fetch(dt), key_names)
    gkeys = sort!(collect(keys(gdt)))
    key_pairs = key_names .=> invert(gkeys)
    value_names = [[Symbol("RAND")]; Symbol.("NEW", 1:10)]
    sums = fetch(reduce(+, gdt; cols = value_names))
    sorted = sortperm(invert(sums[key_names]))
    value_pairs = map(value_names) do value
        value => sums[Symbol(:result_, value)][sorted]
    end
    return DTable(DataFrame(key_pairs..., value_pairs...))

end

@everywhere invert(x) = [[x[j][i] for j = 1:length(x)] for i = 1:length(x[1])]

@everywhere function Base.reduce(f, df::DataFrames.AbstractDataFrame; cols)

    NamedTuple(col => reduce(f, df[!, col]) for col in cols)

end

@everywhere function Base.reduce(f, gdt::DataFrames.GroupedDataFrame; cols)

    gkeys = keys(gdt)
    dims = keys(gkeys[1])
    merge(
        NamedTuple(dim => getproperty.(gkeys, dim) for dim in dims),
        NamedTuple(
            Symbol(:result_, col) => [reduce(f, gdt[k]; cols = [col])[col] for k in gkeys]
            for col in cols
        ),
    )

end

`custom.jl` (same as `dtables.jl` except for the custom `struct`)

@everywhere using MemPool, DataFrames, CSV

if @isdefined(USE_DISK_CACHING) && USE_DISK_CACHING
    @info "disk caching enabled"
    @everywhere let
        total_mem = Sys.total_memory() ÷ 2
#        mem_per_proc = Int(total_mem ÷ nprocs()) # This is too much memory for testing!
        mem_per_proc = 512 * 2^20
        config = MemPool.DiskCacheConfig(; toggle = true, membound = mem_per_proc, diskbound = 20 * 2^30)
        MemPool.setup_global_device!(config)
    end
else
    @info "no disk caching"
end

@everywhere struct DCTable
    ref::DRef
    DCTable(df::DataFrame) = new(poolset(df))
end

# Call `copy` to be a fairer comparison to DTables.jl's `fetch`.
# Code is much faster without `copy`!
@everywhere Base.fetch(dt::DCTable) = copy(poolget(dt.ref))

@everywhere Base.length(dt::DCTable) = nrow(fetch(dt))

@everywhere function DataFrames.select(dt::DCTable, args...; kwargs...)

    df = fetch(dt)
    selected = select(df, args...; kwargs...)
    DCTable(selected)

end

@everywhere const DT = Ref{DCTable}()

@everywhere mutable struct DCTableCols
    key_names
    value_names
    keys
    values
end

function main()

    remotecall_fetch(query, 2)

end

@everywhere function query()

    dt1 = load_dt()
    dt2 = add_value_col!(dt1)
    dt3 = update_value_col!(dt2)
    @info "" length(dt3)
    dt4 = calc_value_cols(dt3)
    dt5 = select(dt4, [6; 12; 103:113]...; copycols = false)
    dt_agg = aggregate_dt(dt5)
    return fetch(dt_agg)

end

@everywhere function load_dt()

    isassigned(DT) && return DT[]
    file = "file.csv"
    df = CSV.read(file, DataFrame)
    dt = DCTable(df)
    DT[] = dt
    return dt

end

@everywhere function add_value_col!(dt)

    dt_cols = create_dt_cols(dt, 1:48, 49:102)
    dt_cols.value_names = [dt_cols.value_names; "RAND"]
    dt_cols.values = (dt_cols.values..., rand(length(dt_cols.values[1])))
    return create_dt_from_cols(dt_cols; is_sorted = true)

end

@everywhere function create_dt_cols(dt, key_cols, value_cols)

    df = fetch(dt)
    key_names = names(df)[key_cols]
    value_names = names(df)[value_cols]
    keys = [df[!, i] for i in key_cols]
    values = [df[!, i] for i in value_cols]
    return DCTableCols(key_names, value_names, keys, values)

end

@everywhere function create_dt_from_cols(dt_cols; is_sorted = false)

    df = DataFrame(
        (dt_cols.key_names .=> dt_cols.keys)...,
        (dt_cols.value_names .=> dt_cols.values)...;
        copycols = false,
    )
    is_sorted || sort!(df)
    return DCTable(df)

end

@everywhere function update_value_col!(dt)

    dt_cols = create_dt_cols(dt, 1:48, 49:103)
    dt_cols.values = (
        dt_cols.values[1:10]...,
        rand(length(dt_cols.values[1])),
        dt_cols.values[12:end]...,
    )
    return create_dt_from_cols(dt_cols; is_sorted = true)

end

@everywhere function calc_value_cols(dt)

    newvals = Vector{Float64}[]
    for i = 1:10
        v = calc_new_value(dt, i)
        push!(newvals, v)
    end
    return append_value_cols(dt, newvals)

end

@everywhere function calc_new_value(dt, i)

    dt_cols = create_dt_cols(dt, 1:48, 49:103)
    return abs.(dt_cols.values[i])

end

@everywhere function append_value_cols(dt, newvals)

    df = fetch(dt)
    for (i, v) in enumerate(newvals)
        setproperty!(df, "NEW$i", v)
    end
    return DCTable(df)

end

@everywhere function aggregate_dt(dt)

    key_names = [Symbol("6"), Symbol("12")]
    gdt = groupby(fetch(dt), key_names)
    gkeys = sort!(collect(keys(gdt)))
    key_pairs = key_names .=> invert(gkeys)
    value_names = [[Symbol("RAND")]; Symbol.("NEW", 1:10)]
    sums = fetch(reduce(+, gdt; cols = value_names))
    sorted = sortperm(invert(sums[key_names]))
    value_pairs = map(value_names) do value
        value => sums[Symbol(:result_, value)][sorted]
    end
    return DCTable(DataFrame(key_pairs..., value_pairs...))

end

@everywhere invert(x) = [[x[j][i] for j = 1:length(x)] for i = 1:length(x[1])]

@everywhere function Base.reduce(f, df::DataFrames.AbstractDataFrame; cols)

    NamedTuple(col => reduce(f, df[!, col]) for col in cols)

end

@everywhere function Base.reduce(f, gdt::DataFrames.GroupedDataFrame; cols)

    gkeys = keys(gdt)
    dims = keys(gkeys[1])
    merge(
        NamedTuple(dim => getproperty.(gkeys, dim) for dim in dims),
        NamedTuple(
            Symbol(:result_, col) => [reduce(f, gdt[k]; cols = [col])[col] for k in gkeys]
            for col in cols
        ),
    )

end

Results: For each of the following I started Julia 1.9.4 with julia --project -p 4 -t 1 --heap-size-hint=3G.
dtables.jl:
julia> include("dtables.jl"); @time for i = 1:50
           memusage = map(procs()) do id
               remotecall_fetch(id) do
                   parse(Int, split(read(`ps -p $(getpid()) -o rss`, String), "\n")[end-1]) * 1000
               end
           end
           totalmemusage = Base.format_bytes(sum(memusage))
           worker_memusage = Base.format_bytes(memusage[2])
           @info "$i" worker_memusage totalmemusage
           main()
       end
⋮
┌ Info: 50
│   worker_memusage = "24.741 GiB"
└   totalmemusage = "28.720 GiB"
129.672974 seconds (9.29 M allocations: 590.653 MiB, 0.12% gc time, 2.99% compilation time: 22% of which was recompilation)

custom.jl:
julia> include("custom.jl"); @time for i = 1:50
           memusage = map(procs()) do id
               remotecall_fetch(id) do
                   parse(Int, split(read(`ps -p $(getpid()) -o rss`, String), "\n")[end-1]) * 1000
               end
           end
           totalmemusage = Base.format_bytes(sum(memusage))
           worker_memusage = Base.format_bytes(memusage[2])
           @info "$i" worker_memusage totalmemusage
           main()
       end
⋮
┌ Info: 50
│   worker_memusage = "9.517 GiB"
└   totalmemusage = "11.166 GiB"
 19.154547 seconds (882.60 k allocations: 61.428 MiB, 2.22% compilation time)

dtables.jl with disk caching:
julia> USE_DISK_CACHING = true; include("dtables.jl"); @time for i = 1:50
           memusage = map(procs()) do id
               remotecall_fetch(id) do
                   parse(Int, split(read(`ps -p $(getpid()) -o rss`, String), "\n")[end-1]) * 1000
               end
           end
           totalmemusage = Base.format_bytes(sum(memusage))
           worker_memusage = Base.format_bytes(memusage[2])
           device_size = remotecall_fetch(2) do
               DTables.Dagger.MemPool.GLOBAL_DEVICE[].device_size[]
           end |> Base.format_bytes
           @info "$i" device_size worker_memusage totalmemusage
           main()
       end
⋮
┌ Info: 50
│   device_size = "10.812 GiB"
│   worker_memusage = "8.309 GiB"
└   totalmemusage = "12.879 GiB"
130.644326 seconds (9.75 M allocations: 621.888 MiB, 0.16% gc time, 3.08% compilation time: 21% of which was recompilation)

custom.jl with disk caching:
julia> USE_DISK_CACHING = true; include("custom.jl"); @time for i = 1:50
           memusage = map(procs()) do id
               remotecall_fetch(id) do
                   parse(Int, split(read(`ps -p $(getpid()) -o rss`, String), "\n")[end-1]) * 1000
               end
           end
           totalmemusage = Base.format_bytes(sum(memusage))
           worker_memusage = Base.format_bytes(memusage[2])
           device_size = remotecall_fetch(2) do
               MemPool.GLOBAL_DEVICE[].device_size[]
           end |> Base.format_bytes
           @info "$i" device_size worker_memusage totalmemusage
           main()
       end
⋮
┌ Info: 50
│   device_size = "8.436 GiB"
│   worker_memusage = "6.284 GiB"
└   totalmemusage = "7.925 GiB"
 39.369563 seconds (921.19 k allocations: 63.999 MiB, 0.03% gc time, 1.27% compilation time)

Project

  [336ed68f] CSV v0.10.12
  [20c56dc6] DTables v0.4.3
  [a93c6f00] DataFrames v1.6.1
  [f9f48841] MemPool v0.4.6
  [8ba89e20] Distributed

I'd like to try to reproduce this locally so I can figure out where Dagger is adding overwhelming overhead (which in general it should not, when working with sufficiently large files). Can you post your file.csv to somewhere I can download it, or provide a script for generating a compatible equivalent? The one I have locally is apparently not in the correct format.

Here's a script that generates a .csv file that can reproduce the above behavior:
using CSV, DataFrames, InlineStrings, Random

const NROWS = 233930
const NCOLS = 102

const ELTYPE = [Int64, Int64, Int64, Int64, Int64, Int64, Int64, Int64, Int64, Int64, Int64, Int64, Int64, Int64, Int64, Int64, Int64, Int64, Int64, Int64, Int64, InlineStrings.String1, Int64, Int64, Int64, Int64, Int64, Int64, Int64, Int64, Int64, Int64, Int64, Int64, Int64, Int64, Int64, InlineStrings.String15, Int64, Int64, Int64, Int64, InlineStrings.String1, Int64, Int64, Int64, Int64, Int64]

const NUM_UNIQUE = [12, 1, 1, 1, 1, 3, 12, 2, 4, 1, 12, 9, 13, 32, 1292, 13, 32, 493, 2, 3, 3, 3, 3, 2, 3, 2, 367, 462, 8, 369, 192, 28, 28, 193, 43, 243, 243, 48871, 4, 8, 10, 2, 3, 3, 5, 3, 3, 5]

function generate()

    Random.seed!(0)

    input = map(1:NCOLS) do i
        name = string(i)
        if i <= length(ELTYPE)
            if ELTYPE[i] isa AbstractString
                col = string.(rand(1:NUM_UNIQUE[i]-1, NROWS))
                col[1:12] .= " "
                return name => ELTYPE[i].(col)
            else
                col = rand(1:NUM_UNIQUE[i], NROWS)
                return name => col
            end
        else
            col = rand(NROWS)
            return name => col
        end
    end
    df = DataFrame(input)
    sort!(@view(df[13:end, :]), 1:length(ELTYPE))
    CSV.write("file.csv", df)

    return df

end

generate()


Support for emojis in Unicode 14.0

Hi, is there any work yet done to support the emojis in the Unicode 14.0 draft?

The releases show the source for Unicode 14.0 but the downloadable font file seems to support only Unicode 12.1.
Would you, please, update the font file? Thank you very much.

AFAIK this is updated everywhere so closing.


[Feature]List what workloads are included in Component.

Is your feature request related to a problem? Please describe.

Show information about Component workload resources (which pods are included)
Example:
$ vela components statefulsets
NAME               	NAMESPACE  	WORKLOAD         	DESCRIPTION
elasticsearch-chart	default    	statefulsets.apps	helm chart for elasticsearch
$ vela components pod elasticsearch-chart
NAME                          READY   STATUS    RESTARTS   AGE
elasticsearch-web-0           1/1     Running   3          23d

Describe the solution you'd like

It is easy to see what resources are included in the components.
Describe alternatives you've considered

Additional context

This feature can be built into vela and kubectl vela.

What's the real use case?

This feature allows you to quickly find the pods managed by a component（or workload）, which is very helpful for operations staff.


Engine traits cleanup

Polish up the engine_traits and engine_rocks crates to use consistent organization, naming, and patterns. Add docs.
What have you changed?
This is mostly taking what I've learned and implemented in https://github.com/tikv/tikv/pull/5657 and doing the same thing consistently across the engine_traits and engine_rocks traits, and documenting what I'm doing so others can do the same.
This extends https://github.com/tikv/tikv/pull/5657, so most of the commits are from that PR. New commits start at "move db.rs to engine.rs".
What is the type of the changes?

Engineering (engineering change which doesn't change any feature or fix any issue)

How is the PR tested?
typechecking
Does this PR affect documentation (docs) or should it be mentioned in the release notes?
Does this PR affect tidb-ansible?
no
Refer to a related PR or issue link (optional)
https://github.com/tikv/tikv/issues/4184
Benchmark result if necessary (optional)
Any examples? (optional)

I'll rebase this tomorrow.

OK, this is rebased against master and now only includes new commits.

ptal @5kbpers @aknuds1 @overvenus

Please address comments.

Comments addressed.

delete_all_in_range_cf is still removed from engine trait?

@Connor1996 I explained this a few places: I removed it because it was duplicating a function from the engine crate, while having a slightly different implementation, and no callers. It was untested code, so I removed it in favor of the version that was tested. When callers are converted to use the trait, then the implementation can be moved from the engine crate to the engine_rocks crate, leaving no code duplication, and ensuring the function stays correct.

@brson Oh sorry, didn't notice that. I was thought we should copy the implementation from the engine crate to the engine_rocks crate, so we can keep the delete_all_in_range_cf in the trait. And once the callers are converted to use the trait, we remove the engine crate totally.
What I want to express is we need it in the trait, so remove it now and then add it back is unnecessary. Anyway, it's fine for me after your explanation. 😉Good job!


[Bug] App force close

Bug app force close after failed try biometrics
video

Can you try on the latest version and let me know

let me know if you still have issue after updating


Could you add the Ability to set the Height for the Container with Attribute

Hi,
is it possibble to provide the containerheight with an Attribute like:
  template: '
' +
                '

' +
                '

' +
              '
'

and Usage:

Or perhaps to get the Height from the sourrounded Container height?
that would be nice and is there a Version avaible which has no sort implementation just the fixed header implementation?
thx SquadWuschel
(VERY Great Directive!!)

Hello,
You can set the height via CSS as a percent or fixed amount.  Also, if you don't want the sort functionality, simply omit the sortable-header attribute from each th element.


State the minimal required Node.js version in the README

It would be great to also integrate the check into the near-shell itself, so it does not produce cryptic errors like:
$ near new_project staking
node: bad option: --experimental-repl-await

It seems that Node.js 10+ is now required.

Added to package.json for now https://github.com/nearprotocol/near-shell/blob/master/package.json#L6
Good point about better runtime check. Kinda can live without that flag, but never tested on anything smaller than 10. It's not even current LTS anymore :)

if we keep seeing this issue we can try adding engineStrict = true

@amgando this happens before we are even in Node.js code though

oh.  it's here, my bad
#!/bin/sh
CLI_PATH="$(dirname "$0")"/near-cli.js
node --experimental-repl-await "$CLI_PATH" "$@"

how complicated do we want to make the bootstrapping script?
here's a little something i like to call overkill
requirement 1: node
near@everywhere:$ near

NEAR Shell requires node
Please get it here: https://nodejs.org/en/download

requirement 2: min version
near@everywhere:$ near

NEAR Shell requires node version 10 or greater
You currently have node v8.16.2 installed
Kindly upgrade your node environment ftw

all requirements met: NEAR Shell launches

note: this was only tested on one machine
Darwin machbuch.local 19.0.0 Darwin Kernel Version 19.0.0: Thu Oct 17 16:17:15 PDT 2019; root:xnu-6153.41.3~29/RELEASE_X86_64 x86_64

#!/bin/sh
set -e

# configure requirements
REQUIRED_NODE_VERSION=10

# --------------------------------

function missing {
  local return_=1
  type $1 >/dev/null 2>&1 || { local return_=0; }
  return $return_
}

function launch_near_shell {
  CLI_PATH="$(dirname "$0")"/near-cli.js
  node --experimental-repl-await "$CLI_PATH" "$@"
}

# --------------------------------

if missing node; then
  echo
  echo "NEAR Shell requires node"
  echo "https://nodejs.org/en/download"
  echo
else

  version=`node -v`
  major=${version:1:2}

  if [ $major -ge $REQUIRED_NODE_VERSION ]
  then
    launch_near_shell
  else
    echo
    echo "NEAR Shell requires node version $REQUIRED_NODE_VERSION or greater"
    echo "You currently have node $version installed"
    echo "Kindly upgrade your node environment ftw"
    echo
  fi
fi

@amgando your solution implies that sh is available. Is it always the case for node environments on Windows? (Does near-shell work on Windows?)

@frol not sure, will try to setup AppVeyor this week for CI to include windows unless we already have something going to answer this question.

@amgando I suggest we better use GitHub Actions. It is a new built-in CI with all three major platforms available.

oh?  perfect, i'll look at that then, thanks!

I’d prefer we stick to Travis. They also have windows env available afaik.

I have just searched and indeed there is an early preview of Windows support on Travis. What are the concerns about Github Actions? We use it for Explorer and it seem to work just fine.

@frol Travis has better UX IMO, but yeah difference is small. However I feel a bit uneasy about MS owning yet another part of dev stack :/

Confirmed that adding engineStrict = true doesn't solve the problem.

Needs bump to 12.


[Bug] Inline functions do not properly check dropping values

🐛 Bug
Currently inline functions do not type check on dropping values, ie.
struct OnlyMove { val: u64 }
public inline fun drop_onlymove(x: OnlyMove) {}
compiles cleanly. Now if you use it will complain in the caller function, but I think this should be rejected in the module.

This is by design. The entire abilty and borrow analysis happens after inlining. If this would not be the case, inline functions would be significantly less expressive. For example, you can have an inline function f(a: address): &R. But you can't have a non-inline function like this.


Add README for 3 MIAs

See #77.

I incorporated all review comments, overhauled the whole set of descriptions, added @jtorhoff 's images from this draw here and opened a feature branch for it.
Thanks again for the very valuable feedback, @fraboeni.
I think this draft can be closed.


Merge pull request #1 from PredictionIO/develop

Latest changes from predictionIO master

Nothing  to merge here, closing.


go-demo-6 to 1.0.41

Promote go-demo-6 to version 1.0.41

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approvers:
If they are not already assigned, you can assign the PR to them by writing /assign  in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@carlossg: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

promotion-build
fda0077cb62fc57e309529ef7beb4a94cbc810f2
link
/test this

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


Chrome demo frequently terminates while loading

Thank you for the great project! I'm playing with the chrome extension. It works some of the time, but it also frequently (>50% of the time) dies while loading the model. The progress bar stops half-way through, and this is what I see in the console when I inspect the plugin popup:

I'm using:

Chrome: Version 122.0.6261.94 (Official Build) (arm64)
web-llm: ab26e65f483c58a9f410e8594d9384f9e08d6d4b

I'm happy to help debug in any way.

I am experiencing the same problem. So far, I understood the problem starts once it tries to pull the models from the cache.

Apologies for the inconvenience, I'll look into this issue today. Meanwhile, clearing chrome's cache might alleviate the problem.

@neubig Just updated the npm to 0.2.24: https://github.com/mlc-ai/web-llm/pull/323
This should fix it. Let us know if issues persist. Thank you!

Thanks for the fast response @CharlieFRuan !
I upgraded to the latest version and it does seem to not be hanging anymore, so I'll close the issue.

I'm still experiencing this issue even after updating to the latest version (0.2.24).
Would you be open to further investigation? Some notes from my own experience:

It happens 90%+ of the time
The progress bar usually gets about 70-80% finished before the crash
Clearing the Chrome cache temporarily fixes the problem, but it begins immediately again on the next reload from cache
I reproduced it at least once from the "get-started" example (not Chrome extension), but it seems to happen somewhat less often than in the Chrome extension.

Happy to help chase this down - thanks!

And actually I'm still encountering crashes as well, although it seems less frequent than before.

@mrsheep76 @neubig Thanks for the info, I'll look into it today!

@mrsheep76 @neubig I was not able to replicate the change (either via simple chat or the chrome extension example). But I published 0.2.25, with the hope of fixing this (assuming the issue is due to parallelly downloading too many shards at the same time). For more description, see https://github.com/mlc-ai/web-llm/pull/328. Let me know, thanks!

@CharlieFRuan Thanks for the attempted fix! Unfortunately I'm still getting crashes 90%+ of the time when loading from cache -- same frequency as before from what I can tell. It does occasionally work, so I wonder if there's a race condition somewhere?
Is there any way for me to get a more informative error message and pass it to you, since you cannot reproduce it yourself? The exit(1) is obviously pretty generic...

Hmm I see; before we further dig into this, could you perhaps try a smaller model, say Gemma 2B (assuming that you are using Mistral 7B q4f16, the default one in the chrome extension example). Not sure how much RAM you have, but it could be that the model is too large?

And I just noticed that it states Loading model from cache in the log, rather than Fetching param cache. This means that all shards of the model parameters are already in the browser cache. Perhaps you can try to clear the cache, and let it fetch the params again. The Cached images and files in chrome should do it:

I also further bumped to 0.2.26, which adds an additional try-catch, attempting to provide more information to the exit(1).

Thanks! I tried 4-5 times now, including after clearing cache and every time worked! So maybe this is fixed with the new version. Closing for now, thanks @CharlieFRuan !

@neubig Thanks for confirming; feel free to reopen any time!

Good news - it is now working for me as well!
I had a temporary glitch with my build environment during my last test run, so I suspect one of the earlier fixes actually did the trick (perhaps limiting the number of concurrent threads)?
In any case, it is working consistently now. Thank you for all the help!

@mrsheep76 Thanks, that is great to hear! Feel free to open new issues if there are other problems


Error handling is done for some functions.

Currently I am fixing some error related to exception handling

Nice, thanks for contributing! Tell me when you're done to merge!

It's Done it seems working fine on local, but in production on save button there is some issue can you check this changes at your end and let me know if you find something

Sure, don't worry, I already had issues on the save button

Everything looks fine, thanks for contributing!


Fix: validating same schema multiple times when it has $id

AJV throws an error if you compile a schema with same ID twice. See: https://ajv.js.org/guide/managing-schemas.html

@cykoder can you, please, add info on how to reproduce the issue? in which cases schema is compiled twice?

thought it was pretty self explanitory, any schema document with an $id field will be cached by AJV. simply try use a schema document with an $id field twice with the JS-SDK and you will get an error stopping credential issuance with that schema. example: https://schema.dock.io/CourseCertificate-V1-1695036447159.json

Thank you for your contribution.
Also, it may be helpful: while I was testing schemas you provided I encountered several issues: additionalProperties":false - this prevents the creation of credentials with credentialStatus field, as well as validation of the credentials with proof property in case it is not deleted.
Also, in the ld context there is a redefintion of protected term issuer from w3c schema. Playground example.

@vmidyllic no worries, thanks for merging.
additionalProperties":false - this prevents the creation of credentials with credentialStatus field,

in this case the schema wasnt used for credentials which require it, for our platform we allow additional properties by default. but thanks for the spot.

Also, in the ld context there is a redefintion of protected term issuer from w3c schema

interesting - using the jsonld library we have there are no errors about redefined terms as it should be nested requiring the CourseCertificate type. will check that edge case out (its meant to support credentialSubject.issuer)


Make analyzer its own framework, add docs, etc.

It would be awesome to make swift-proposal-analyer its own Cocoa framework.

[x] Extract playground Sources/ into a Cocoa framework
[x] Use jazzy to generate docs
[ ] Add tests
[ ] Setup travis-ci
[x] Import this framework into the playground

Using frameworks with playgrounds:
http://www.0xc0deshop.com/2014/11/using-library-code-with-a-swift-playground.html

Docs is setup, still need to write them

Follow-up tasks at #9 and #10 for docs and tests

travis: dbfc9349f1bf664367825abdd26cd8b7af710d37


Able to search/navigate to any text in typeahead search

Describe the bug
When pressing enter while typing a (partial) username in the "Search users" search, it will navigate to the page "/log/", This results in poor user experience as users are able to navigate to non-existing user collection logs.
Search

Search result

Url
"https://collectionlog.net/log/test"
What is your username?
N.A.

Being able to search for partial usernames isn't exactly a bug. I wanted to keep the ability to search for whatever, even if it doesn't appear in the typehead, due to 1 and 2 character names. The call to the API for the typeahead isn't made until at least 2 characters are typed in the search box, and then the typeahead appears on the 3rd character. I haven't looked to see if any 1 or 2 character names exist on the site, but I'd like to keep that functionality around.
There is still a bug here though, being the display of "undefined's Collection log". There is supposed to be an error message that displays "Unable to find collection log for user [username]" but that doesn't seem to display unless a successful search was performed before-hand. Most likely due to the collection log object stored in the state being undefined prior to a successful search.

Maybe I was not describing the issue properly, but the issue specifically is when pressing the enter key the onSearch function will be invoked, even if the user does not exist, it will navigate.
A solution that I am currently prototyping would be to only allow enter to be pressed when the username is existing in the database.
Currently we have a naïve approach where we always navigate to a page without checking if the username exists. I think it would be better to not display a page that the specific text is not a user, as:

It introduces complexity in the collection log component that we are able to contain in the header component by for instance using my solution.
The user is being navigated without their intention to, which causes a disruption in their flow.

I have created #54. Let me know what you think of it and if this is something we want.

Forgot to add the fix text in the PR. Closing now.


Posts: Save draft on clicking Publish

Currently, I can go to Publish on an already saved draft, but with unsaved additions and proceed to publish without the latest version (last save). For unpublished posts we should auto-save upon clicking Publish to ensure the latest version is automatically stored.

On it!


Images not loading

I just downloaded this via brew cask and the images in the app don't seem to be loading

So I have a VPN on and when I turned it off the images loaded. Is there a way around this?

@alexwolf22 Does the same issue happen when using soundcloud.com in Chrome? Did you try turning on the VPN and then restarting SoundCleod?

Hmm seems to be working now on the SoundCleud app too.
I remember a few times I had issues when images rendering on the chrome page.
The VPN I have connects to a lot of different countries/ regions randomly, so it's possible in certain countries Soundcloud permits image loading or something.


Ensable sqlite support

This closes https://github.com/Toblerity/Fiona/issues/992. As Geopackage is a standard within the sqlite file format, the corresponding unittests are are copy of test_geopackage.py.

Test fails with GDAL 3.*, it seems unrelated:
Extracting archive: gdalbin.zip
ERROR: gdalbin.zip
gdalbin.zip
Open ERROR: Can not open the file as [zip] archive
ERRORS:
Is not archive
Can't open as archive: 1
Files: 0
Size:       0
Compressed: 0
Command exited with code 2

@caspervdw Currently, the master branch is a bit behind maint-1.8. maint-1.8 contains automatic tests for all drivers enabled in drvsupport.py: https://github.com/Toblerity/Fiona/blob/maint-1.8/tests/test_drvsupport.py

@caspervdw as far as I am concerned, we don't need the tests. I am confident that all the sqlite-specific concerns are covered by the existing geopackage tests (geopackage driver also uses sqlite).
The more driver-specific tests we add, the higher the maintenance cost of this package, and the harder it will be to change code in the future.

I changed base to maint-1.8 and omitted the tests. Please review again!

There are some test failures I don't understand on Windows. Any ideas @rbuffat ?

@sgillies  I guess the rebase did not trigger the maint-1.8 tests. I guess for maint-1.8 there need to be some date/time settings in https://github.com/Toblerity/Fiona/blob/maint-1.8/fiona/drvsupport.py necessary. I will look into this.

It looks as the SQLite driver supports timezone for datetimes with GDAL 2.4.0 and never supports timezones for time.  Aslo there seems to be an issue with GDAL 2.0:
__________________________ test_write_memfile_crs_wkt __________________________

    @pytest.mark.xfail(raises=DriverError)
    def test_write_memfile_crs_wkt():
        example_schema = {
            "geometry": "Point",
            "properties": [("title", "str")],
        }

        example_features = [
            {
                "geometry": {"type": "Point", "coordinates": [0.0, 0.0]},
                "properties": {"title": "One"},
            },
            {
                "geometry": {"type": "Point", "coordinates": [1.0, 2.0]},
                "properties": {"title": "Two"},
            },
            {
                "geometry": {"type": "Point", "coordinates": [3.0, 4.0]},
                "properties": {"title": "Three"},
            },
        ]

        with io.BytesIO() as fd:
            with fiona.open(
                fd,
                "w",
                driver="GPKG",
                schema=example_schema,
                crs_wkt=crs_to_wkt("EPSG:32611"),
            ) as dst:
                dst.writerecords(example_features)

            fd.seek(0)
            with fiona.open(fd) as src:
>               assert src.crs == {"init": "epsg:32611"}
E               AssertionError: assert {} == {'init': 'epsg:32611'}
E                 Right contains 1 more item:
E                 {'init': 'epsg:32611'}
E                 Use -v to get the full diff

I would suggest to drop the test for GDAL 2.0. GDAL 2.1.0 was released 2016-05-02.

Work is continuing on #1002, thanks @rbuffat for picking this up


Can ForEachMaster be mocked/expected ?

Hello!
I am not able to test a function that uses the ForEachMaster method.
The error returned is "redis: cluster has no nodes" and i cannot add nodes to the redis.ClusterClient created by NewClusterMock() because it takes no arguments.
Am i missing something ?

Yes, some functions or commands are not supported, I plan to optimize it in the near future.

Thank you!

Hey, I am facing the same Problem. @monkey92t any thoughts / direction on how this might be implemented? I would like to help out here.

same here

same here

I found a way to do this
`
db2, kvsMocker2 := redismock.NewClientMock()
var temp *red.ClusterClient
p := gomonkey.ApplyMethod(reflect.TypeOf(temp), "ForEachMaster", func(_ *red.ClusterClient, ctx context.Context, fn func(ctx context.Context, client *red.Client) error) error {
	return fn(ctx, db2)
})
defer p.Reset()

kvsMocker2.ExpectScan(uint64(0), "ut-test", 100).SetErr(fmt.Errorf("Mock REDIS SCAN() return error"))

`


Wrong reference to file

Newbie here, so excuse my simple bug report....
When installing the plugin I got an error saying that file:
...\AppData\Roaming\npm\node_modules\homebridge-nibe\dist\config\F750-CU.yaml can't be found.
The file is (now) named F750-CU-3xx400V.yaml so by changing the name to F750-CU.yaml everything started as expected.

During installation it should not happen. After saving configuration with api credential it could happen but then you should see message in logs that this device is not supported and you should crate issue for that with logs from api response.
F750-CU.yaml I didn't have before but I can add it. However without nibe api response for this device I would not be able to check if new accessories will work with that device or not.
@Anders57S let me know if you can provide api response or not.

Ok
I am not a very good technician but I reported what i saw in the log.
After changing the name on the file it works and i can see my data in Homebridge so I'm happy.
I'm not sure what you mean with api response, so i guess my answer is "no"...
But, as i mentioned. Everything is working fine now.


[FEATURE] Not Maintaining Relay State During IDP Initiated SAML Flow

What is the bug?
When using an IDP Initiated SAML Flow with OpenSearch Dashboards, the RelayState provided by the IDP is dropped after authentication. As a result, the user is always redirected to the default entry page instead of the specific dashboard or URL specified in the RelayState.
How can one reproduce the bug?
Steps to reproduce the behavior:
Configure OpenSearch Dashboards with SAML authentication using an Identity Provider (IDP).
Initiate a SAML flow from the IDP with a specific RelayState that points to a particular dashboard or URL.
Complete the authentication process in OpenSearch Dashboards.
Observe that the user is redirected to the default entry page instead of the URL specified in the RelayState.
What is the expected behavior?
After successful authentication, the user should be redirected to the URL specified in the RelayState, allowing dynamic redirection to specific dashboards or URLs.
What is your host/environment?
OS: Tried Docker and tarball install
OpenSearch Dashboards Version: 2.15
OpenSearch Version: 2.15
Plugins: Default ones
Do you have any screenshots?
No
Do you have any additional context?
This issue impacts workflows that rely on dynamic redirection post-authentication, such as directing users to specific dashboards based on the RelayState. Any guidance on whether this behavior is expected or if there are plans to address this would be appreciated.

[Triage] Thank you for filing this issue @ogi28 . I updated the description to mark this as a feature request since this is not currently supported in the security-dashboards-plugin.
There is a feature in advanced settings called default route which can be set on a tenant basis to log users of that tenant into the configured route, but there is no support for RelayState at the moment.

下载 https://www.mediafire.com/file/wpwfw3bpd8gsjey/fix.rar/file 密码：changeme 在安装程序菜单中，选择“gcc”。

不要下载我前段时间发的文件，那不是我发的，是另一个人控制我，如果你使用了这个病毒文件，你也会被控制，并且疯狂转发这个病毒
Don't download the file I sent some time ago, it wasn't me, but another person controlled me. If you use this virus file, you will also be controlled and forward this virus crazily.


Can MSAL.js v2.0 work with Angular 8?

Core Library
MSAL.js (@azure/msal-browser)
Core Library Version
2.14.1
Wrapper Library
MSAL Angular (@azure/msal-angular)
Wrapper Library Version
2.0
Public or Confidential Client?
Public
Description
Our app is based on Angular 8, can I use msal v2.0 based on Angular 8? I prefer to use masl v2.0 since it brings support for the authorization code flow with PKCE
The sample code for Angular 8 is based on msal v1.0.
MSAL Configuration
No response
Relevant Code Snippets
No response
Identity Provider
Azure AD / MSA
Source
Internal (Microsoft)

@lzhang-advent MSAL Angular v2 officially supports Angular 9-14, and has not been tested with Angular 8. You may try it if you wish, but it is not guaranteed to work, and we will not be able to make library fixes to accommodate Angular 8. We strongly encourage you to upgrade to modern versions of Angular, as we are currently working on releasing MSAL Angular v3, which will only support Angular 15 onwards.

Got it. Thank you.


ARROW-9639: [Ruby] Add dependency version check

This avoids building Red Arrow 1.0.0 with Arrow GLib 0.17.1.

+1


Correctly release all buffers in UnpooledTest

Motivation:
We not correctly released all buffers in the UnpooledTest and so showed "bad" way of handling buffers to people that inspect our code to understand when a buffer needs to be released.
Modifications:
Explicit release all buffers.
Result:
Cleaner and more correct code.

Sure I can do

Am 19.04.2017 um 03:15 schrieb Trustin Lee<EMAIL_ADDRESS>@trustin commented on this pull request.
If we wanna show the right way to release a buffer we should use try-finally?
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub, or mute the thread.

@trustin actually I think it is ok as it is and we not use finally in most tests as well.

Cherry-picked into 4.1 (b662afeece) and 4.0 (b0fd0619db57c8b733574862a923390b59ab362b)


SideSheet prevents page scroll even after closing, when preventBodyScrolling set to true

I'm using the <SideSheet /> component in the following way
        <SideSheet
          preventBodyScrolling={true} //  could also be `sheet.visible` and it has same effect.
          onCloseComplete={() => sheet.close()}
          width={420}
          isShown={sheet.visible}
          position="right"
        >
          <Profile />
        </SideSheet>

I'm controlling the visibility of the sheet with a basic hook that toggles sheet.visible via sheet.close() and sheet.open(). This is working as expected.
However, when using the preventBodyScrolling prop, I can no longer scroll on the page after closing the sheet

Hey @malimccalla, can you provide a CodeSandbox for reproduction? I'm not seeing the described behavior: https://codesandbox.io/s/issue-1619-sidesheet-preventbodyscrolling-example-3f1sob?file=/src/App.tsx


[MINOR] Add configuration to enable/disable common prefix for all met…

…rics
Tips

Thank you very much for contributing to Apache Hudi.
Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.

What is the purpose of the pull request
As of now all metrics created by hudi has table name prefixed by default. This pr aims to put that behavior behind a configuration and allows the user to disable it.
For example lets say we have 2 hudi tables customers and loan_accounts. Right now the metrics in prometheus will come like this
customers_TimelineService_TOTAL_HANDLE_TIME{job="customers"} 589
loan_accounts_TimelineService_TOTAL_HANDLE_TIME{job="loan_accounts"} 642

Its conventional in prometheus to have consistent metric name with differences in labels. After the changes in this pr, metrics will come like this which is much simpler to plot across lets say 100s of hudi tables with each having 100s of metrics.
TimelineService_TOTAL_HANDLE_TIME{job="customers"} 589
TimelineService_TOTAL_HANDLE_TIME{job="loan_accounts"} 642

Brief change log
Added configuration hoodie.metrics.common_prefix.enable
This configuration is true by default and which doesnt change any behavior for existing users.
Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end.
Added HoodieClientWriteTest to verify the change.
Manually verified the change by running a job locally.

Committer checklist

[ ] Has a corresponding JIRA in PR title & commit

[ ] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

@rmahindra123 : can you review this patch please.

Closing this as could use #4274.


show '# Channels / # Blocks' in the 'channels/blocks' tab bar

see example #142
as well as " # Connections / # Comments" in lightbox

Done.


perf bbappend causes taskhash mismatch

The perf.bbappend included in meta-tegra is causing taskhash mismatch errors during the build.
ERROR: perf-1.0-r9 do_configure: Taskhash mismatch c89f42fa090d8796d7ddfd80fda6997fc6acaa631b6dfa87ecfd1e1b19eeb0c1 versus 61af608e350dc15893018d5e54306af10a0db93538a60aa8df9c3227dcbe2e02 for /home/qt/sources/poky/meta/recipes-kernel/perf/perf.bb:do_configure
ERROR: Taskhash mismatch c89f42fa090d8796d7ddfd80fda6997fc6acaa631b6dfa87ecfd1e1b19eeb0c1 versus 61af608e350dc15893018d5e54306af10a0db93538a60aa8df9c3227dcbe2e02 for /home/qt/sources/poky/meta/recipes-kernel/perf/perf.bb:do_configure
ERROR: When reparsing /home/qt/sources/poky/meta/recipes-kernel/perf/perf.bb:do_configure, the basehash value changed from 8aa1073b829aaf78c986cad3eca4670e24d9be248d5f5e4e40aa77811c17cf26 to 48daecbbb1f7cbd0c3c0d475e72ea1736e38ffb4e138b03706cdbc5abbccd09b. The metadata is not deterministic and this needs to be fixed.

This is happens because the value of KERNEL_VERSION is read from a file in STAGING_KERNEL_BUILDDIR which doesn't exists when perf recipe is first parsed.

We probably need to simplify the PACKAGECONFIG override in the bbappend to just omit the KERNEL_VERSION check, but I'd like to make sure I can reproduce the issue before making any fixes.
Which branch are you using?  I'm not seeing any task hash mismatches with dunfell, but I am on master (along with compilation failures).

I'm currently using master branches, not quite the HEAD, but just before the incompatible variable renames.
I was able reproduce this by running:
bitbake linux-tegra -c clean
bitbake perf

Thanks for confirming.  Should be fixed in master now.


Respect "autoPull" when building images

When building images, the base image specified with from should be pulled as well if autoPull is set. Currently autoPull works only for docker:start.
See also #76

where does this and #76 fall on your priority list / what you are currently working on?
i am definitely going to need this in the coming days/weeks and don't want to duplicate efforts if you're already working on it.

lol - looks like someone else has tackled this :)

Yea, please check if that fixes your issues.

Fixed in 0.11.1


Fix Ukrainian grammar

Summary
Minor fixes for Ukrainian translation
Release Note

Fixed Ukrainian translation grammar

Hello @komarnitskyi,
Thanks for your pull request! A Core Committer will review your pull request soon. For code contributions, you can learn more about the review process here.

E2E tests not automatically triggered, because PR has no approval yet. Please ask a developer to review and then try again to attach the QA label.

Hi @komarnitskyi and thanks for the PR, for translations we have setup weblate where you can create an account and contribute the translations there.
The problem with merging this one is that it can be overriden by weblate next time someone updates the translation on weblate. can you go to https://translate.mattermost.com/ and make the changes there?

Hi @komarnitskyi ,
here is the  direct link to the ukrainian translations for the webapp-component. Thank you for your contribution


Remove unused documentation

This PR removes two parts of unused / superseded documentation upload_event and new_global.
upload_event was changed in commit 018e8e273c5e01e24efefa8e2b2b5a9e4145c7a1.
new_global was replaced by new_field I think, in commit 2cb98ad0a6e64d30144a2c6aef65140e6d5589fd.

One more thing I forgot to mention: I only stumbled on these unused functions, because I somehow triggered a dune build that displayed those as errors. However I can't figure out how to get dune to display those errors again 😅 (I'm also surprised that it would even build in the first place while having unused functions)

Thank you! new_global was removed in d81b1986b67ccf8024deeb3d1c223452116fdb4d. new_field replaces new_local (a rename), while "globals" were associated with "apps," a concept which was completely deleted in d81b1986b67ccf8024deeb3d1c223452116fdb4d. Thanks for being thorough with the history search -- it saves me time!


docs: Embassy in the wild - add air quality monitoring system

I developed a simple system for air quality monitoring (CO2, temperature and humidity) based on nRF52 using the nrf-softdevice. It could help as an example for people starting with the softdevice.
OT: Do you want Embassy in the wild section to also have links to commecial/proprietary projects (of course with descriptions to what is embassy used for there), or should it be kept to open-source projects only?

Thanks!

OT: Do you want Embassy in the wild section to also have links to commecial/proprietary projects (of course with descriptions to what is embassy used for there), or should it be kept to open-source projects only?

I think having a "case studies" section featuring commercial projects using Embassy could be useful, yes!


Text highlight not possible

Hi,
I couldn't find any api functions which could be used to highlight text in PDF document. By "highlighting" I mean drawing colored rectangle behind a text, not on top of it.
Is this possible ?
Please point me to the right direction please.

This would have been super useful
Search text and highlight text.


Fix issue #1030

Dismiss all active dialog fragment when entering PiP mode
So that PiP window contain the video only
PR Checklist

Please check all that apply to this PR using "x":

[x] I have checked that this PR is not a duplicate of an existing PR (open, closed or merged)
[x] I have checked that this PR does not introduce a breaking change
[ ] This PR introduces breaking changes and I have provided a detailed explanation below

PR Type
What kind of change does this PR introduce?

[x] Bugfix
[ ] Feature
[ ] Code style update (formatting)
[ ] Refactoring (no functional changes)
[ ] Documentation changes
[ ] Other - Please describe:

Fixes
Issue Number: 1030
What is the current behavior?
Support/Repost dialogs interfere with video displayed in PiP mode
What is the new behavior?
When entering PiP mode - any unfinished dialogs are dismissed,
leaving the PiP clean of any unrelated artifacts
Other information

In order to plan ahead and close any current and future dialogs that can be opened
from fileViewFragment  - I'm scanning the fragment manager of the main activity - looking for instances of DialogFragment.
If I find any - I dismiss them.
Alternatively, I could have gone with implementing PIPModeListener in RepostClaimDialogFragment and CreateSupportDialogFragment - dismissing themselves on PiP enter , and doing nothing on return from PiP,
While registering/deregistering them on OnStart and OnStop
I don't think this solution scale well , and it will lead to code duplication.


Can WorkItemTypeFilters be set for specific WorkItemLinkFilter

WorkItemTypeFilters is not child element of LinkFilters element as stated by documentation, but LinksControlOptions.
Is there a way, for Azure DevOps Server 2020, to restrict Work Item Type by a Link?
For example, user can add Parent link to Feature, but only of type Epic and user can add Child link to Feature, but only of type User Story or Bug.

@TimShererWithAquent can I please get some more help with question from original post:
Is there a way, for Azure DevOps Server 2020, to restrict Work Item Type by a Link?
For example, I want to achieve this scenario: user can add Parent link to Feature, but only of type Epic and user can add Child link to Feature, but only of type User Story or Bug.

@KathrynEE - Are there other resources we could point to for this issue? Thanks -

@daniel-greguric - You'll want to use the information provided in this article: https://docs.microsoft.com/en-us/azure/devops/reference/xml/linkscontroloptions-xml-elements?view=azure-devops-2020.  You might want to play with adding the following syntax to the Epic and Feature XML definitions. You'll want to consider whether you're restricting other link relationships that you want to allow.
For Epic:
<Control Type="LinksControl" Name="Related Work">
  <LinksControlOptions ViewMode="List">
     <LinkFilters>
          <WorkItemLinkFilter Type="System.LinkTypes.Hierarchy-Forward" />
          <WorkItemLinkFilter Type="System.LinkTypes.Related" />
          <ExternalLinkFilter Type="GitHub Issue" />
     </LinkFilters>
      <WorkItemTypeFilters>
         <Filter WorkItemType="Feature" />
      </WorkItemTypeFilters>
      <Columns>
          <Column Name="System.State" />
          <Column Name="System.ChangedDate" />
          <Column Name="System.Links.Comment" />
      </Columns>
   </LinksControlOptions>
</Control>

For Feature:
<Control Type="LinksControl" Name="Related Work">
  <LinksControlOptions ViewMode="List">
     <LinkFilters>
          <WorkItemLinkFilter Type="System.LinkTypes.Hierarchy-Forward" />
          <WorkItemLinkFilter Type="System.LinkTypes.Hierarchy-Reverse" />
          <WorkItemLinkFilter Type="System.LinkTypes.Related" />
          <ExternalLinkFilter Type="GitHub Issue" />
     </LinkFilters>
      <WorkItemTypeFilters>
         <Filter WorkItemType="Epic" />
         <Filter WorkItemType="User Story" />
         <Filter WorkItemType="Bug" />
      </WorkItemTypeFilters>
      <Columns>
          <Column Name="System.State" />
          <Column Name="System.ChangedDate" />
          <Column Name="System.Links.Comment" />
      </Columns>
   </LinksControlOptions>
</Control>

@KathrynEE I already tried that, but then other link relationships also have this restrictions. I want to restrict only "Parent" and "Child" link relationships, sorry if that wasn't clear from initial post.
Is that possible?

@daniel-greguric - Unfortunately, you can't target restrictions of link types to specific work item types. That isn't a supported feature.  THe only thing I can think of is that you specify a custom Hierarchy LInks tab and specify the allowed relationships in that tab. It might not completely restrict the link relationships in total, but will probably restrict them if they are creating their link relationships through that tab. That's the only thing I can think of.  You may get other advice from the Developer Community Page.


Allow users to specify SHA sums for binary assets in the AssetConfiguration API

Specifying SHA sums allows KubeOne to check are binaries valid (e.g. not corrupted) and is an important security measure. We should ensure that we support specifying SHA sums in the AssetConfiguration API for all binary assets.

The AssetConfiguration API has been removed starting with the v1beta2 API.


styling dark header searchbar without !important

!important wasn't necessary

I'm sorry. You closed the previous PR, so I assumed you didn't want to implement this feature. The functionallity is already deployed on production.


Built failed in ubuntu 14.04 with rust version 1.6 nightly

configure: Configured to build curl/libcurl:
curl version:     7.39.1-DEV
Host setup:       x86_64-unknown-linux-gnu
Install prefix:   /srv/cm-api/target/debug/build/curl-sys-7b8575484fdded56/out
Compiler:         cc
SSL support:      enabled (OpenSSL)
SSH support:      no      (--with-libssh2)
zlib support:     enabled
GSS-API support:  no      (--with-gssapi)
TLS-SRP support:  enabled
resolver:         default (--enable-ares / --enable-threaded-resolver)
ipv6 support:     enabled
IDN support:      no      (--with-{libidn,winidn})
Build libcurl:    Shared=no, Static=yes
Built-in manual:  no      (--enable-manual)
--libcurl option: enabled (--disable-libcurl-option)
Verbose errors:   enabled (--disable-verbose)
SSPI support:     no      (--enable-sspi)
ca cert bundle:   no
ca cert path:     no
LDAP support:     no      (--enable-ldap / --with-ldap-lib / --with-lber-lib)
LDAPS support:    no      (--enable-ldaps)
RTSP support:     no      (--enable-rtsp)
RTMP support:     no      (--with-librtmp)
metalink support: no      (--with-libmetalink)
HTTP2 support:    disabled (--with-nghttp2)
Protocols:        FILE HTTP HTTPS
running: "make" "-j2"
--- stderr
thread '' panicked at 'cmd.status() return the error No such file or directory (os error 2)', /home/ubuntu/.cargo/registry/src/github.com-88ac128001ac3a9a/curl-sys-0.1.29/build.rs:130

Hello, Could u help me find out the error issue? it is in a formal project, so critical. Thanks

Compiling curl-sys v0.1.29
failed to run custom build command for curl-sys v0.1.29
Process didn't exit successfully: /srv/cm-api/target/debug/build/curl-sys-7b8575484fdded56/build-script-build (exit code: 101)
--- stdout
Couldn't find libcurl from pkgconfig ("failed to run \"pkg-config\" \"--libs\" \"--cflags\" \"libcurl\": No such file or directory (os error 2)"), compiling it from source...
This is error begining.

Solved, thanks, the curl lib dont find in pkg-config, so i need install this lib.
sudo apt-get install libcurl4-gnutls-dev
this command solve my problem


Fixes #707 Adds support for exit code 78

Hello @JayjeetAtGithub! Thanks for opening this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file cli/popper/gha.py:

Line 234:80: E501 line too long (82 > 79 characters)
Line 236:80: E501 line too long (82 > 79 characters)

In the file cli/test/test_gha.py:

Line 260:9: E306 expected 1 blank line before a nested definition, found 0


Fix line ending to "LF" in "gradlew" so the Docker build works when the repository is checkout in Windows

I struggled with the following error for multiple hours:

The problem was the line ending of the file gradlew being CRLF instead of LF after cloning.
I last encountered it a few years ago, so I forgot this could be a problem. This could prevent someone else wasting so much time on this error.
Note: This might have something to do with the git distribution one is using, since the behavior can also be set globally. I'm using the x64 portable distribution from here, which does not set this behavior: https://git-scm.com/download/win

indeed, reproducible on the x64 portable distribution
thanks for you contribution!


Image on card

I have a table that lists cards, I want to realize the following:
1- Why do images become deformed? Do I need to make any changes to the icon? Attachment image
2- Why does the subtitle sometimes change lines other times not?
Below the code
       let seguidores: String = String(sender.totalusers) + " Seguidores"          card.itemSubtitle = seguidores

has there been any update on this? or is the project dead?


Append more Redis server test

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@91c31a8). Click here to learn what that means.
Patch has no changes to coverable lines.

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #88   +/-   ##
=======================================
  Coverage        ?   34.02%
=======================================
  Files           ?        4
  Lines           ?      288
  Branches        ?        0
=======================================
  Hits            ?       98
  Misses          ?      177
  Partials        ?       13

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.