Further Improve Execution Speed with Explicit Bounds Checks

I've been digging into Cranelift's and Wasmtime's code quality and Wasm execution throughput when "dynamic memories" with explicit bounds checks are used to implement Wasm linear memories. Here I'm summarizing the progress made thus far here, and laying out what I believe the next steps to continue this effort should be.
Static memories require large reservations of virtual memory, to the extent that it can become a bottleneck for how many Wasm instances can exist concurrently in a system. Dynamic memories, on the other hand, require very little or even zero virtual memory reservations (depending on configuration) beyond the actually resident pages backing the Wasm linear memory itself. Additionally, when dynamic memories are configured without any guard regions, they incur zero virtual memory-related syscalls on instantiation and tear down (modulo any copy-on-write heap image initialization). These syscalls have been observed to be bottlenecks in concurrent systems due to VMA locks taken in the kernel.
Furthermore these issues will only compound in the future. Right now, each tenant in a multi-tenant system is a single core Wasm module with a single Wasm linear memory.  If the average Wasm component contains five linear memories, then the multi-tenant system can only support one fifth as many concurrent tenants under the component model, and instantiating and tearing down each tenant's service will take five times as many syscalls as it does today.
So there is a lot of pressure towards implementing Wasm linear memories with dynamic memories. The downside is that dynamic memories currently incur roughly 55% overhead to Wasm execution throughput compared to static memories. That is, a Wasm computation that takes 1 second in a static memory will take about 1.55 seconds in a dynamic memory. This is because dynamic memory accesses must be explicitly bounds checked rather than caught after the fact via page faults like out-of-bounds static memory accesses. The goal of the effort described here is to minimize this overhead.
I don't intend to continue this work in the immediate future, so I wanted to make sure everything was summarized and documented for posterity, as well as written down while it is still fresh in my mind. Additionally I have proposals for new optimization passes we should implement in the future, and this issue can serve as a forum for discussing them.
Progress Made Thus Far
Before I began this effort, running the spidermonkey.wasm benchmark in Sightglass with static memories was 3.06x faster than doing the same with dynamic memories. Now we are down to static memories being only 1.55x faster.
This improvement comes from a number of sources:

Deduplicating explicit bounds checks and Spectre mitigations, since our Spectre mitigations effectively perform bounds checks already, the additional explicit bounds checks were redundant.
Ensuring that our global value numbering (GVN) optimization pass could optimize and deduplicate uadd_overflow_trap and select_spectre_guard instructions, both of which are emitted in our bounds-checking sequence.
The introduction of a variety of new peephole optimizations and constant propagation rules that were automatically synthesized by Souper from snippets of code harvested from spidermonkey.wasm.
New special cases for certain dynamic memory configurations in our bounds-checking implementation.

Additionally, I created the wasmtime explore subcommand to dig into native code disassemblies and associate each native instruction with the Wasm bytecode it implements (similar to the Godbolt Compiler Explorer). This new tool has already been incredibly helpful for analyzing what room for improvement still exists, and should help further future optimization efforts in Cranelift and Wasmtime.
Recent Sightglass Benchmark Results
Here are some recent Sightglass benchmark results comparing static memories (static.so) versus dynamic memories with a small (64KiB) guard region (dyn-with-guard.so) versus dynamic memories without a guard region (dyn-without-guard.so).
static.so vs dyn-with-guard.so

execution :: cycles :: benchmarks/spidermonkey/benchmark.wasm

  Δ = 517183665.48 ± 6392410.71 (confidence = 99%)

  static.so is 1.55x to 1.56x faster than dyn-with-guard.so!

  [1420787062<PHONE_NUMBER>.68<PHONE_NUMBER>] dyn-with-guard.so
  [914013248 933017938.20<PHONE_NUMBER>] static.so

execution :: cycles :: benchmarks/bz2/benchmark.wasm

  Δ = 45802386.02 ± 1486278.64 (confidence = 99%)

  static.so is 1.42x to 1.45x faster than dyn-with-guard.so!

  [146742085 150423911.30 159078923] dyn-with-guard.so
  [100319894 104621525.28 149448856] static.so

execution :: cycles :: benchmarks/pulldown-cmark/benchmark.wasm

  Δ = 3014103.57 ± 316843.43 (confidence = 99%)

  static.so is 1.32x to 1.40x faster than dyn-with-guard.so!

  [10662276 11365102.25 16630094] dyn-with-guard.so
  [7683153 8350998.68 13449448] static.so

compilation :: cycles :: benchmarks/bz2/benchmark.wasm

  Δ = 45130213.24 ± 4838200.92 (confidence = 99%)

  static.so is 1.18x to 1.22x faster than dyn-with-guard.so!

  [249993052 271654226.14 316970220] dyn-with-guard.so
  [208845361 226524012.90 290090035] static.so

compilation :: cycles :: benchmarks/pulldown-cmark/benchmark.wasm

  Δ = 53326476.11 ± 5528961.43 (confidence = 99%)

  static.so is 1.16x to 1.19x faster than dyn-with-guard.so!

  [330373001 356824375.72 397185785] dyn-with-guard.so
  [279072702 303497899.61 348984338] static.so

compilation :: cycles :: benchmarks/spidermonkey/benchmark.wasm

  Δ =<PHONE_NUMBER>.64 ± 81595847.03 (confidence = 99%)

  static.so is 1.15x to 1.18x faster than dyn-with-guard.so!

  [8097294016<PHONE_NUMBER>.47<PHONE_NUMBER>6] dyn-with-guard.so
  [6969056325<PHONE_NUMBER>.83<PHONE_NUMBER>] static.so

static.so vs dyn-without-guard.so

execution :: cycles :: benchmarks/spidermonkey/benchmark.wasm

  Δ = 594932587.44 ± 6811906.18 (confidence = 99%)

  static.so is 1.63x to 1.65x faster than dyn-without-guard.so!

  [1494952789<PHONE_NUMBER>.11<PHONE_NUMBER>] dyn-without-guard.so
  [908877847 930012836.67 980176520] static.so

execution :: cycles :: benchmarks/bz2/benchmark.wasm

  Δ = 49482140.46 ± 2175287.73 (confidence = 99%)

  static.so is 1.45x to 1.49x faster than dyn-without-guard.so!

  [149370743 154540290.45 163713075] dyn-without-guard.so
  [99902180 105058149.99 163151023] static.so

compilation :: cycles :: benchmarks/bz2/benchmark.wasm

  Δ = 54959651.89 ± 5371934.09 (confidence = 99%)

  static.so is 1.22x to 1.26x faster than dyn-without-guard.so!

  [258958338 282812487.71 327560254] dyn-without-guard.so
  [210542623 227852835.82 315990395] static.so

compilation :: cycles :: benchmarks/spidermonkey/benchmark.wasm

  Δ =<PHONE_NUMBER>.89 ± 100212378.19 (confidence = 99%)

  static.so is 1.21x to 1.24x faster than dyn-without-guard.so!

  [8494305552<PHONE_NUMBER>.14<PHONE_NUMBER>2] dyn-without-guard.so
  [6985640940<PHONE_NUMBER>.25<PHONE_NUMBER>] static.so

compilation :: cycles :: benchmarks/pulldown-cmark/benchmark.wasm

  Δ = 66518323.42 ± 5778757.98 (confidence = 99%)

  static.so is 1.20x to 1.24x faster than dyn-without-guard.so!

  [342947509 371023668.35 415514806] dyn-without-guard.so
  [277471563 304505344.93 339940435] static.so

execution :: cycles :: benchmarks/pulldown-cmark/benchmark.wasm

  Δ = 1217702.33 ± 627813.45 (confidence = 99%)

  static.so is 1.06x to 1.18x faster than dyn-without-guard.so!

  [11111517 11632130.88 14705814] dyn-without-guard.so
  [7682848 10414428.55 14122261] static.so

Comparing Wasmtime and SpiderMonkey
To get a sense of whether our 1.55x slowdown when enabling explicit bounds checks was typical of other engines, I also benchmarked SpiderMonkey (native) running Sightglass's spidermonkey.wasm benchmark with virtual memory guard pages vs with explicit bounds checks.
execution :: milliseconds :: spidermonkey.wasm

  Δ = 163.62 ± 5.57 (confidence = 99%)

  virtual-memory-spidermonkey is 1.52x to 1.56x faster than bounds-checks-spidermonkey!

  [446 465.49 520] bounds-checks-spidermonkey
  [290 301.87 353] virtual-memory-spidermonkey

Because SpiderMonkey's slowdown is almost identical to Wasmtime's, we can infer that there aren't any critical bounds-checking optimizations that SpiderMonkey performs but Wasmtime/Cranelift does not. This is reassuring: it means we can basically ignore other engines (or at least SpiderMonkey) for now, when trying to further improve codegen when explicit bounds checks are enabled, and focus only on ourselves.
Proposed Future Work
What fruit remains to be harvested? I've been digging into spidermonkey.wasm's hot functions (as reported by perf on Linux) and staring at disassemblies to get an idea of how much room for improvement we still have. In short: a lot!
I found many sequences of back-to-back memory accesses in hot functions, often with

the same dynamic index,
different static offsets, and
ordered such that accesses with larger static offsets came first (or at least early) within the sequence of accesses.

This is great because it is the next-easiest pattern to optimize for (after sequences of identical accesses, which we can already optimize well).
Even when the accesses weren't roughly sorted from largest to smallest static offset, the offsets were always small enough to be covered by a small guard region. This allows us to bounds check index > bound instead of index + offset > bound which allows us to GVN these comparisons even when their static offsets aren't in order. After that, all we are left with are redundant select_spectre_guards, which I will describe how we can eliminate in a moment.
For example, I found a back-to-back sequence of nine stores that all have the same dynamic index operand but different static offset immediates, all in the same basic block:
local.get 2
...
i64.store offset=368
local.get 2
...
i64.store offset=352
local.get 2
...
i32.store offset=336
local.get 2
...
i32.store offset=328
local.get 2
...
i32.store offset=320
local.get 2
...
i32.store offset=312
local.get 2
...
i32.store offset=304
local.get 2
...
i32.store offset=296
local.get 2
...
i32.store offset=360

Only the first store in this sequence needs to be bounds checked! All the following stores' bounds checks are implied by the first check, but we emit the bounds check for every single one of them.
Based on this investigation, I suggest we implement two new optimization passes:

a select_spectre_guard-elimination pass,
and a basic constraint-elimination pass.

select_spectre_guard-Elimination Pass
We should develop an optimization pass to remove redundant select_spectre_guard instructions.
This optimization pass would walk the dominator tree and maintain the set of spectre_select_guard conditions in scope that we've already tested and whose results have flowed into an instruction that will raise a trap if the select_spectre_guard failed. That is, maintain the set of xs for all sequences like
load(select_spectre_guard(x, 0, _))

that dominate the current location in our traversal.
Then, whenever we see a new select_spectre_guard(x, 0, y), we can rewrite it to simply y, since the dominating load(select_spectre_guard(x, 0, _)) would have raised a trapped already if x was truthy, so control flow reaching this location means that x must not be truthy and therefore select_spectre_guard(x, _, y) will always evaluate to y.
This does not break the speculative sandboxing function of the optimized-away select_spectre_guards because there will always still be at least one dominating select_spectre_guard for the same condition.
This optimization pass should integrate with the e-graphs framework in the same way that our alias analysis optimization pass does. It should expose a "push"-based interface, where blocks and instructions are fed into the analysis by an external traversal, rather than having the pass walk the IR itself. This will allow us to fuse the pass with existing IR traversals.
Creating this new optimization pass is not super easy, so if we wanted to get an idea of what kind of speed ups it would give, we can implement the much easier https://github.com/bytecodealliance/wasmtime/issues/6055, which would give us the non-Spectre equivalent of this optimization pass. The amount that implementing #6055 speeds up benchmarks in an environment configured with Spectre mitigations disabled is roughly what we could expect from this new optimization pass with Spectre mitigations enabled.
A Potential Variation
This analysis is slightly subtle in that it requires chaining a few things together (loads/stores that consume a select_spectre_guard that then has a condition x). In general, the fewer links in the chain, the simpler the analysis's implementation will be and the more confidence we can have in its correctness. To that end, we could consider having spectre_load(oob_condition, addr) and spectre_store(oob_condition, addr, value) instructions that are morally equivalent to load(select_spectre_guard(oob_condition, 0, addr)) and similar for stores, but which do not get legalized in the mid-end and go all the way to our backends. By fusing the instructions, we remove one link in that proof chain that the analysis has to create, making it simpler and more likely to be correct. The downside is that we don't only use select_spectre_guard with plain load and store CLIF instructions; there are {u,s}{load,store}{8,16,32} instructions which would all either need Spectre-fused variants or to continue using the old select_spectre_guard (and presumably not reap the benefits of this simpler analysis), SIMD load instructions, atomic access instructions, etc. I've filed https://github.com/bytecodealliance/wasmtime/issues/6056 to help consolidate some of these instructions, which would slightly alleviate this downside, but it would not be fully addressed. Because of that, I'm thinking that this approach with Spectre-fused instructions is probably not worth the effort.
Basic Constraint-Elimination Pass
We should develop a very basic constraint-elimination optimization pass targeted at bounds checks for 32-bit Wasm memories on 64-bit hosts.
This would not be a fully general constraint-elimination pass, like what LLVM has recently added.  Instead it would focus only on comparisons of roughly the form index + constant > heap_bound.
Consider this example, with constants X and Y:
v3 = index + X > heap_bound
...
v6 = index + Y > heap_bound

When X > Y, then v3 implies v6, and we can rewrite the example into the following:
v3 = index + X > heap_bound
...
v6 -> v3

The catch is that this is only correct if index + X does not overflow. Otherwise, it is possible that index + X is "logically out of bounds" but because it overflowed and wrapped around, it ended up back in bounds again. In this scenario, Y might not be large enough for index + Y to overflow, and index + Y could remain out of bounds.
We can avoid this pitfall by limiting the pass's scope to optimizing bounds checks for only 32-bit memories on 64-bit hosts, which have the following pattern:
uextend(index:i32) + X:i64 > heap_bound:i64

For these 32-bit bounds checks, the constant X fits in an i32 -- despite being represented as an i64 -- and therefore we are guaranteed that index + X will not overflow. As long as the proposed optimization pass checks this constraint on X when matching that code pattern, it will be sound.
This optimization pass would precede the select_spectre_guard-elimination pass described previously, since it would deduplicate bounds checks, which then feed into spectre_select_guards, which that pass would then be able to completely remove.
For example, consider this input CLIF for two back-to-back loads:
    ;; First we load the Wasm address `index + X`.

    ;; The `index + X > heap_bound` check.
    v0 = uextend.i64 index
    v1 = iadd_imm v0, X
    v2 = icmp_imm ugt v1, heap_bound
    ;; The native address for the first Wasm load.
    v3 = iadd heap_base, v1
    ;; The Spectre guard for the first Wasm load.
    v4 = select_spectre_guard v2, null, v3
    ;; The first Wasm load itself.
    v5 = load.i32 v4

    ;; Second we load the Wasm address `index + Y`, where `X > Y`.
    v6 = iadd_imm v0, Y
    v7 = icmp_imm ugt v6, heap_bound
    v8 = iadd heap_base, v6
    v9 = select_spectre_guard v7, null, v8
    v10 = load.i32 v9

After running this proposed constraint-elimination pass, we learn that v2 implies v7 because X > Y and so we can rewrite uses of v7 with v2:
    ;; First load (unchanged).
    v0 = uextend.i64 index
    v1 = iadd_imm v0, X
    v2 = icmp_imm ugt v1, heap_bound
    v3 = iadd heap_base, v1
    v4 = select_spectre_guard v2, null, v3
    v5 = load.i32 v4

    ;; Second load (comparison replaced with alias of previous comparison).
    v6 = iadd_imm v0, Y
    v7 -> v2
    v8 = iadd heap_base, v6
    v9 = select_spectre_guard v2, null, v8
    v10 = load.i32 v9

Finally, after running the previously described select_spectre_guard-elimination pass, we can completely remove the second select_spectre_guard bounds check since it is identical to and dominated by the first:
    ;; First load (unchanged).
    v0 = uextend.i64 index
    v1 = iadd_imm v0, X
    v2 = icmp_imm ugt v1, heap_bound
    v3 = iadd heap_base, v1
    v4 = select_spectre_guard v2, null, v3
    v5 = load.i32 v4

    ;; Second load (all bounds checks removed).
    v6 = iadd_imm v0, Y
    v7 -> v2
    v8 = iadd heap_base, v6
    v9 -> v8
    v10 = load.i32 v8

This final code is the optimal implementation of the original input.
This optimization pass should also integrate with the e-graphs framework in the same way that the previously described optimization pass would and the way that our alias analysis optimization pass does.
To get an idea of the speed up this optimization can give dynamic memories without any guard pages, we can look at the speed up that #6055 gives dynamic memories with guard pages when Spectre mitigations are disabled. This is at least 1.08x speed up, as that is what we've seen from deduplicating bounds checking comparisons (but not the select_spectre_guards) in #6031, but probably closer to at least 1.15x when paired with removing no-longer-necessary select_spectre_guards if I were to take a wild guess.
Optimizing Memory Accesses with Distinct Dynamic Indices
The examples we've considered thus far have a series of memory accesses that use the same dynamic index, but different static offsets. For example, each of the following loads all have the same local.get 4 dynamic index:
local.get 4
i32.load offset=24
local.get 4
i32.load offset=18
local.get 4
i32.load offset=12

We haven't considered when the memory accesses have different dynamic indices, like in this example:
local.get 4
i32.load
local.get 12
i32.load
local.get 8
i32.load

There are two primary reasons for this:

The latter is much harder to optimize. Determining any kind of relationship between the different indices is hard. Sometimes the second access's index is loaded by the first access (e.g. load(load(...))), which will pretty much defeat any static analysis we throw at it. Even if the indices are each of the form base + a and base + b, and we know that a > b, we have to consider whether overflow occurs in base + a (see the discussion about overflow in the basic constraint-elimination pass section above). Developing a pass to optimize these cases will be difficult, complicated, and prone to unsoundness.

Most of the sequences of back-to-back memory accesses I found in spidermonkey.wasm's hot functions had the same dynamic index. This is great news since these are also the cases we can optimize relatively easily.

To conclude, I don't think it is worth the effort trying to optimize these memory access sequences, where different dynamic indices are accessed.

For posterity, here is the suuuuuper-hacky WASI polyfill I used to get SpiderMonkey running the spidermonkey.wasm benchmark from sightglass (which was forked off from a smaller polyfill that @alexcrichton shared with me, thank you).

const ITERS = 100;

////////////////////////////////////////////////////////////////////////////////

class TextEncoder {
  constructor(enc) {
    if (enc != "utf-8") {
      throw new Error("FITZGEN: unsupported encoding: " + enc);
    }
  }
  encode(s, n) {
    let buf = new Uint8Array(s.length);
    for (let i = 0; i < Math.min(s.length, n); i++) {
      buf[i] = s.charCodeAt(i); // lol
    }
    return buf;
  }
};

class TextDecoder {
  constructor(enc) {
    if (enc != "utf-8") {
      throw new Error("FITZGEN: unsupported encoding: " + enc);
    }
  }
  decode(buf) {
    let buf8 = new Uint8Array(buf);
    let s = "";
    for (let i = 0; i < buf8.length; i++) {
      s += String.fromCharCode(buf8[i]); // lol
    }
    return s;
  }
};

////////////////////////////////////////////////////////////////////////////////

async function ionCompile(wasm) {
  let module = await WebAssembly.compile(wasm);
  while (!wasmHasTier2CompilationCompleted(module)) {
    sleep(1);
  }
  return module;
}

function unimplemented(name) {
  return function (...args) {
    throw new Error(name + " is unimplemented! args = " + args);
  }
}

class ProcExitError extends Error {
  constructor(code) {
    this.code = code;
    this.message = "Program exited with code " + code;
  }
  toString() {
    return "ProcExitError: " + this.message
  }
}

function trace(obj) {
  let proxy = {};
  for (let key of Object.keys(obj)) {
    if (typeof obj[key] != "function") {
      proxy[key] = obj[key];
      continue;
    }
    proxy[key] = function (...args) {
      print("TRACE: " + key + "(" + args + ")");
      let ret = obj[key](...args);
      print("TRACE:   -> " + ret);
      return ret;
    };
  }
  return proxy;
}

////////////////////////////////////////////////////////////////////////////////

async function main() {
  let smWasm = os.file.readFile("/home/nick/sightglass/benchmarks/spidermonkey/benchmark.wasm", "binary");
  let smModule = await ionCompile(smWasm);

  for (let i = 0; i < ITERS; i++) {
    let args = ["js"];
    let mem = null;
    let start = null;
    let unread = true;

    let smInstance = await WebAssembly.instantiate(smModule, {
      bench: {
        start: function () {
          start = monotonicNow()
        },
        end: function () {
          let end = monotonicNow();
          print("ITER: " + (end - start));
        }
      },
      wasi_snapshot_preview1: {
        random_get: function(a, b) {
          while (b > 0) {
            (new DataView(mem.buffer)).setInt8(a, 1, true);
            b -= 1;
            a += 1;
          }
          return 0;
        },
        args_get: function(a, b) {
          for (let arg of args) {
            (new DataView(mem.buffer)).setInt32(a, b, true);
            a += 4;
            for (let c in arg) {
              (new DataView(mem.buffer)).setInt8(b, arg.charCodeAt(c), true);
              b += 1;
            }
            (new DataView(mem.buffer)).setInt8(b, 0, true);
            b += 1;
          }
          return 0;
        },
        args_sizes_get: function(a, b) {
          let len = 0;
          for (let arg of args) {
            len += arg.length + 1;
          }
          (new DataView(mem.buffer)).setInt32(a, args.length, true);
          (new DataView(mem.buffer)).setInt32(b, len, true);
          return 0;
        },
        clock_res_get: function () { return 1; },
        clock_time_get: function(a, b, c) {
          const now = Math.round(performance.now() * 1000000);
          (new DataView(mem.buffer)).setBigInt64(c, BigInt(now), true);
          return 0;
        },
        fd_filestat_get: function() { throw new Error('fd_filestat_get'); },
        fd_read: function(fd, iovecs_ptr, iovecs_len, out_ptr) {
          let mem8 = new Uint8Array(mem.buffer);
          switch (fd) {
          case 4:
            let data = os.file.readFile("/home/nick/sightglass/benchmarks/spidermonkey/default.input.md");
            let k = 0;
            for (let i = 0; i < iovecs_len && k < data.length && unread; i++) {
              let ptr = (new DataView(mem.buffer)).getUint32(iovecs_ptr + i * 8, true);
              let len = (new DataView(mem.buffer)).getUint32(iovecs_ptr + i * 8 + 4, true);
              for (let j = 0; j < len && k < data.length; j++) {
                mem8[ptr + j] = data.charCodeAt(k++);
              }
            }
            unread = false;
            (new DataView(mem.buffer)).setUint32(out_ptr, k, true);
            return 0;
          default:
            return 8;
          }
        },
        fd_seek: function(fd, offset, whence, out_ptr) {
          switch (fd) {
          case 4:
            let len = os.file.readFile("/home/nick/sightglass/benchmarks/spidermonkey/default.input.md").length;
            (new DataView(mem.buffer)).setBigUint64(out_ptr, BigInt(len), true);
            return 0;
          default:
            return 8;
          }
        },
        fd_write: function(a, b, c, d) {
          let s = '';
          let total = 0;
          while (c > 0) {
            let base = (new DataView(mem.buffer)).getInt32(b, true);
            let len = (new DataView(mem.buffer)).getInt32(b + 4, true);
            b += 8;
            c -= 1;

            while (len > 0) {
              let c = new Uint8Array(mem.buffer)[base]
              s += String.fromCharCode(c);
              len -= 1;
              base += 1;
              total += 1;
            }
          }
          // print("fd_write(" + a + "): " + s.trimEnd());
          (new DataView(mem.buffer)).setInt32(d, total, true);
          return 0;
        },
        fd_fdstat_set_flags: function() { throw new Error('fd_fdstat_set_flags'); },
        path_filestat_get: function() { throw new Error('path_filestat_get'); },
        path_open: function(fd, dirflags, path_ptr, path_len, oflags, fs_rights_base, fs_rights_inheriting, fdflags, out_ptr) {
          let buf = new Uint8Array(path_len);
          let mem8 = new Uint8Array(mem.buffer);
          for (let i = 0; i < path_len; i++) {
            buf[i] = mem8[path_ptr + i];
          }
          let path = (new TextDecoder('utf-8')).decode(buf);
          switch (path) {
          case "default.input.md":
            (new DataView(mem.buffer)).setInt32(out_ptr, 4, true);
            return 0;
          default:
            print('denying path_open(' + path + ')');
            return 2;
          }
        },
        path_remove_directory: function() { throw new Error('path_remove_directory'); },
        path_unlink_file: function() { throw new Error('path_unlink_file'); },
        sched_yield: function() { throw new Error('sched_yield'); },
        environ_get: function() { return 0; },
        environ_sizes_get: function(a, b) {
          (new DataView(mem.buffer)).setInt32(a, 0, true);
          (new DataView(mem.buffer)).setInt32(b, 0, true);
          return 0;
        },
        fd_close: function(fd) {
          return 0;
        },
        fd_fdstat_get: function(fd, out_ptr) {
          (new DataView(mem.buffer)).setInt32(out_ptr, 0);
          // type
          switch (fd) {
          case 3:
            (new DataView(mem.buffer)).setInt32(out_ptr + 4, 3, true);
            break;
          case 4:
            (new DataView(mem.buffer)).setInt32(out_ptr + 4, 4, true);
            break;
          default:
            (new DataView(mem.buffer)).setInt32(out_ptr + 4, 0, true);
            break;
          }
          (new DataView(mem.buffer)).setInt32(out_ptr + 8, 0, true);
          (new DataView(mem.buffer)).setInt32(out_ptr + 12, 0, true);
          (new DataView(mem.buffer)).setInt32(out_ptr + 16, 0, true);
          (new DataView(mem.buffer)).setInt32(out_ptr + 20, 0, true);
          return 0;
        },
        fd_prestat_get: function(a, b) {
          // case for preopened "."
          if (a == 3) {
            // discriminant for directory
            (new DataView(mem.buffer)).setInt32(b, 0, true);
            // one entry in directory
            (new DataView(mem.buffer)).setInt32(b + 4, 1, true);
            return 0;
          }
          return 8;
        },
        fd_prestat_dir_name: function(fd, ptr, len) {
          let buf = (new TextEncoder('utf-8')).encode(".", 1);
          let mem8 = new Uint8Array(mem.buffer);
          for (let i = 0; i < Math.min(buf.length, len); i++) {
            mem8[ptr + i] = buf[i];
          }
          mem8[Math.min(buf.length, len)] = 0;
          return 0;
        },
        proc_exit: function(code) { throw new ProcExitError(code); },
      },
    });

    mem = smInstance.exports.memory;

    try {
      smInstance.exports._start();
    } catch (e) {
      if ((e instanceof ProcExitError) && e.code == 0) {
        continue;
      }
      throw e;
    }
  }
  print("Okay!");
}

main().catch(function (e) {
  print("====== ERROR!!! ======");
  print(e);
  print(e.stack);
});

Thanks for summarizing all this @fitzgen; sorry for not getting to it sooner!
I don't think a "remove dominated checks within a basic block" optimization is safe, either, for the same reasons; I want to try to give a little intuition into how the out-of-order execution works to show why and help guide any further discussions.
The main idea behind OoO is restricted dataflow execution: "restricted" because it operates like a sliding window over the program's execution trace, and "dataflow execution" because within that window, instructions are decoded into a true dependency graph and can execute whenever their inputs are ready.
You can think of the CPU has having a frontend that predicts some trace (at each branch, picks a direction; at indirects, predicts a target; at returns, predicts based on an internal predictor callstack); that output is a linear stream of instructions. This stream goes through a "rename" stage that computes dependencies and does a sort of SSA-like thing, assigning new physical registers to instructions' written dests; and an "allocate" stage that puts the instructions into a "reorder buffer" (ROB) as well as the scheduling data structures (think "ready queues" and such). Then there is a "retirement stage" that crawls the reorder buffer from a tail pointer, and at any cycle can take some number of the oldest instructions in-flight and "commit" or "retire".
Speculation is a behavior that arises out of the cooperation of the frontend and backend. The frontend is predicting some path; the OoO engine then treats this linear stream as truth; but if a branch resolves differently than it was predicted, or an instruction traps, we need to flush and resteer. In the simplest designs, this happens when the instruction reaches retirement (is the oldest instruction): we need a consistent architectural machine state to restart from, and the most natural way to get that is to let retirement put things back in-order. More complex aproaches use periodic checkpoints but the effect is the same -- the speculation recovery is delayed a bit, and younger instructions (those "beyond" the misspeculation or trap) were still in the dataflow window for however long and could have executed.
So if we have a program something like:
block0:
jnz oob // if out-of-bounds, go elsewhere
r0 <- cmove0 // check conditions for unsafe_load0
unsafe_load0 r0
r1 <- cmove1 // check conditions for unsafe_load1
unsafe_load1 r1

and let's say the branch is mispredicted as falling through (predicted not-OOB, but actually is). Then the reorder buffer will contain those five instructions in order and execution obeys only true dataflow dependencies. Even if we can resolve the branch relatively early, misspeculation recovery is delayed somewhat (possibly until retirement in simple designs, and imagine there are 100 instructions prior to these in the window, so it could be a while). If it weren't for the cmoves, the loads could execute and there's our side-channel. But unsafe_load0 can't execute until cmove0 produces its result, and likewise unsafe_load1, and even though we predicted this wrong path, the actual dataflow is correct, so the cmoves will pick a NULL pointer and we don't leak anything.
Now say we have
jnz oob
r0 <- cmove0
unsafe_load r0
unsafe_load r1 // assume first inst trapping will guard this one

we get these instructions into the window, and the jnz might be marked in the ROB as mispredicted (resteer at retirement), and the unsafe_load r0 might be marked as faulting (resteer to pagefault handler when it reaches retirement; this will never happen though because the jnz recovery happens first), but hey look, this unsafe_load r1 uses a register that has been available for a while; nothing is stopping us from running it out-of-order (just as it says on the tin; out-of-order execution to the rescue!). And there's our side-channel leak again.
The basic intuition behind the Spectre mitigations, and guidance from all the CPU vendors, is that processors can speculate control-flow and execute instructions in arbitrary order (and this is fundamental to the 10x perf per clock cycle we've gotten in hardware in the last few decades), but they won't speculate dataflow. (Value speculation is a thing in the uarch research literature but AFAIK no one has ever shipped it; and now that Spectre is a thing, likely no one will?). This is why the cmove thing works. Fences also work, because a fence instruction can't enter the OoO window until every older instruction retires (the window is empty). But of course that window flush is quite expensive.
Given that, I think we need to obey the invariant that some cmove based on an OOB condition exists in the computation of every Wasm address we load. It's possible we could do a multi-stage thing: if we access p and later p - 16, the computation of p - 16 could use the cmove-guarded p as input. So something like: v2 = icmp ge v1, bound; v3 = select_spectre_guard v2, v1, nullptr; v4 = load v3+0; v5 = load v3-16. But I suspect that's the best we can do, unless we start playing with fences. My intuition is that this might still be OK if it means we can hoist the one cmove out of a loop -- basically we have a single choke-point where we can "turn off" all the address dataflow. But we'd have to think through it a bit further...

Ah, the tidbit of intuition I forgot to state explicitly as well, in case it helps: we can think of faulting loads/stores as a kind of "branch" as well, given that we use the same recovery mechanism as for mispredicted branches. So instructions beyond one that must fault may still be executed speculatively, because we either haven't executed the to-fault inst yet, or we have but our recovery mechanism hasn't yet let us flush the window and redirect to the trap handler. So the "make the load fault" mitigation protects that load, because it won't result to a value that lets its dependent instructions run; but it doesn't protect younger instructions, because they can run before or after (of course with architectural effects only committed once earlier insts resolve successfully, so, never if earlier fault).
Other tidbit of intuition that might help: it's more accurate to think of an OoO CPU as executing every instruction speculatively, rather than thinking of speculation as a thing that happens when the branch predictor meets a branch. The latter is certainly a form of speculation, which we need to produce the linear stream of instructions into the backend. But strictly speaking every execution of a node in the dataflow graph in the backend when that node is not the oldest is speculation that some condition (trap, misspeculated memory ordering or aliasing (st-to-ld-fwd) condition; etc) will not cause a flush. Effects of this speculative early execution are kept in the ROB (or equivalent structures alongside, such as the store buffer) until retirement.

If we add a knob for tuning the size of the null guard page, and both offset1 and offset2 were within the null guard page and offset1 > offset2, then with the help of the basic constraint-elimination pass, we could do this series of rewrites
Original input, what cranelift-wasm generates now:
addr1_oob = index + offset1 > heap_bound
addr1 = spectre_select_guard(addr1_oob, 0, heap_base + index + offset1)
addr2_oob = index + offset2 > heap_bound
addr2 = spectre_select_guard(addr2_oob, 0, heap_base + index + offset2)

Pull offsets out the spectre guard because they are smaller than the null guard page (or just modify cranelift-wasm to emit them like this in the first place, since that is the location where we know the heap configuration):
addr1_oob = index + offset1 > heap_bound
addr1 = spectre_select_guard(addr1_oob, 0, heap_base + index) + offset1
addr2_oob = index + offset2 > heap_bound
addr2 = spectre_select_guard(addr2_oob, 0, heap_base + index) + offset2

Basic constraint-elimination pass makes addr2_oob an alias of addr1_oob because offset1 > offset2:
addr1_oob = index + offset1 > heap_bound
addr1 = spectre_select_guard(addr1_oob, 0, heap_base + index) + offset1
addr2 = spectre_select_guard(addr1_oob, 0, heap_base + index) + offset2

"GVN" (via e-graphs) the spectre_select_guard operation:
addr1_oob = index + offset1 > heap_bound
base_and_index = spectre_select_guard(addr1_oob, 0, heap_base + index)
addr1 = base_and_index + offset1
addr2 = base_and_index + offset2

Now we only have a single spectre_select_guard!
@cfallin, how does that sound?

I think this looks workable, yep. It's worth defining here precisely what the "replace addr2_oob with addr1_oob" condition is because it's a little nonstandard -- it isn't equivalence but rather implication (addr2_oob --> addr1_oob). It thus wouldn't be a valid substitution on its own, sans any control-flow context (if addr1 and addr2 were just value outputs of the program, we'd be changing addr2 sometimes); but here it is fine because the access of addr1 dominates the access of addr2. Said another way: altering the computed value is fine because such alteration is only observable on a path past a trap.

Could cranelift-wasm track enough information to fully do this transformation itself, instead of implementing a generic optimization in Cranelift?
I think that because wasm only has reducible control flow, the dominator tree is implied by the control-flow instructions in the input wasm. So I think we can keep track of which linear-memory accesses dominate the current instruction while making a single pass over the input program.
If so, then we could maintain a scoped hash map keyed on CLIF Values which have been used as the index to some dominating memory access. (Since local.get and various stack manipulations are all rewritten to SSA values during this phase, this should suffice to normalize a variety of weird patterns in the original wasm.)
These keys would map to a pair of the maximum offset checked so far, and the select_spectre_guard result that was used in that check. If the current access is less than or equal to that offset then the result can be reused. Otherwise we need another bounds-check but can update the map with the new larger offset and new bounds-checked base.
That's pretty simple compared to doing it generically in Cranelift, right?

I think it is possible, yes. The idea of fusing this optimization with an existing pass where we have domination "for free" is definitely tempting.
One hardship I expect we would encounter would be that this runs before the phi-elimination pass and therefore we would likely leave some optimizations on the table, especially around Wasm locals and joining control flow.

That's pretty simple compared to doing it generically in Cranelift, right?

Perhaps?
I don't think the passes I outlined above are too complicated, its mostly just a matter of writing them such that they can be fused with each other and with the egraphs pass (similar to how the alias analysis is written). As long as that is done, then it seems fairly straightforward to me: two passes that each do a single shape of optimization.
In contrast, in cranelift-wasm we are already doing a bunch of translation and such, and adding another optimization responsibility in addition to that feels a bit like conflating concerns.


No ORM function to add a relation on a has_one

There doesn't seem to be a method using the ORM to add a relation to a has_one.
Obviously you can do it like this $data->items()->add($i) but that doesn't work in reverse.
There needs to be something like this http://api.silverstripe.org/3.1/class-HasManyList.html#_add for a has_one class.
It's nice to keep everything tidy and use the ORM where possible so this might be a nice feature.

The way most people do this (the only way I know of) is $object->HasOneID = $item->ID, where “HasOne” is the name of the relation. We have DataObject::getComponent() for extracting one to one relations, perhaps a setComponent() method might be a nice addition? There’s not much ORM magic going on behind the scenes, though, so I’m not sure whether a new API is necessary.

It seems odd that you wouldn't have an ORM function for this instead of assigning IDs directly?
Even if it wouldn't contain much ORM magic it makes sense just to keep things consistent?
I would imagine it being somewhat helpful if the attach function could check you are attaching a model of the correct type so I can't go assigning an Image to a Page relationship etc.

I agree, there should be a function for this. Code should be written so that the knowledge of the DB naming isn't required.

Chiming in here to say that we also need to account for polymorphic has_one and all belongs_to relations, which are 1:1 and do not have an inverse where you can utilize HasManyList or ManyManyList APIs.
Currently in my code I'm having to assign the RelationID and RelationClass fields manually everywhere this needs to be handled and, frankly, for better IDE integration (@method docblock comments aside) it's a heck of a lot simpler for me to to be DRY'er to just write my own method for my specific use case and be done with it :smile: That tells me this needs an API to help simplify refactoring and the development process in general as your code evolves.

p.s. With the change being considered minor, does this make it a candidate for 3.x (i.e. the 3 branch) or does it need to wait for 4? Shouldn't be backward incompatible; we'll be filling a void, here.

Doesn't the API simply support $parent->Child = $childRecord? Is that a more natural syntax?

Chiming in here to say that we also need to account for polymorphic has_one and all belongs_to relations, which are 1:1 and do not have an inverse where you can utilize HasManyList or ManyManyList APIs.

Also, polymorphic has_one supports 1:many through PolymorphicHasManyList, so that's not quite correcty. :P

Doesn't the API simply support $parent->Child = $childRecord? Is that a more natural syntax? @patricknelson have you not tried this for your polymorphic relations?

Ah, doesn't seem that this works... that would be the easiest thing to implement though, right?

Hi @tractorcow regarding this statement:

Chiming in here to say that we also need to account for polymorphic has_one and all belongs_to relations, which are 1:1 and do not have an inverse where you can utilize HasManyList or ManyManyList APIs.

Also, polymorphic has_one supports 1-to-many through PolymorphicHasManyList, so that's not quite correct. :P

I went ahead and restructured my sentence up there a bit. My core point here is that all 1:1 relations (i.e. any that use belongs_to along side a has_one) are affected and it's important to ensure we include compatibility with polymorphic relations which adds the overhead of an extra field as a foreign key. There edited :smile:

What do you think my proposal about using a basic __set-ter for assigning has_one relations?

My immediate intuition when testing for potentially undocumented behavior was to try that generic assignment first just to see if it'd work. However, while I really like the idea of using assignment, my hesitation is that (at a framework level) it may not be consistent. For example, accessing auto-magical relations in SilverStripe always takes the form of a method call, right? So in this case:
// Create relationship through assignment (safer, less likely to cause fatal errors due to
// potential of accessing a method on a non-object or, in this case, undefined relationship).
$parent->Child = $childRecord;

// More consistent with HasMany/ManyMany relationships, but doesn't make sense to use "add"
// since this is the has_one side of the relationship.
$parent->Child()->add($childRecord);

// More intuitive method-based approach, more consistent with Laravel's approach, but
// Laravel already uses methods to define relations (which I am originally used to).
$parent->Child()->save($childRecord);

// Persist on parent? Since ChildID and potentially ChildClass fields live on $parent.
$parent->write();

Also note that I'm not opposed to implementing both the assignment and method approaches. I have some experience of my own in this area since I've actually implemented my own ORM once and modeled it after Eloquent back in 2013 (I believe). What I setup used methods to define relations in the same way and had a query object which was responsible for accessing relations and aggregating objects, etc.

Hmm... Actually, I may be opposed to implementing both styles, because it may limit future backward compatibility extensibility due to some unforeseen enhancements. I think the method-based approach is the most flexible long term which is also reasonably consistent.
As long as we decided between the semantic save vs. add since we need it to be singular, as this is a has_one relation we're dealing with.

// More consistent with HasMany/ManyMany relationships, but doesn't make sense to use "add"
// since this is the has_one side of the relationship.
$parent->Child()->add($childRecord);

The problem with this is that $parent->Child() is the existing mechanism for getting the child object. The object cannot be concurrently its own record, as well as the representation of a relationship with some parent, in the same way that  the RelationList classes are.

@tractorcow that's true. I think in my case once you accessed the has_one relationship, you still had to call ->first() (similar to Laravel again) in order to get an instance of that object to begin handling it. Granted, you're calling ->first() frequently, but you could also potentially do more with the relationship.
So, given the current API, we're likely only limited to the attribute assignment approach. That also suggests that the property then also needs to be available as well (at least via lazy load).

I feel we are trying to force a convention in place where it doesn't really make sense. I don't think the same convention we have for lists of objects (where a separate object represents the list vs the items within) makes the same sense to has_one relation, where there is no natural separation.
I would prefer to make the parent object the container for the "has_one relation" in this case, and have the given behaviour there. E.g.
$parent->Item = null instead of $parent->Item()->remove()
$parent->Item = $obj instead of $parent->Item()->add($obj)
$parent->Item() (or even just $parent->Item) instead of $parent->Item()->first().

Right @tractorcow I agree (see my last comment). I think at this point the part we might want to establish now is:

If $parent->write() is required to retain this newly associated relationship (that is one setup via attribute assignment) and
If we agree that it make sense to allow attribute-based access like $item = $parent->Item (similar to method-based access $item = $parent->Item()) and then have that attribute based access be magic (__get()) and lazily loaded unless specified otherwise.

See what I mean?

... and with those comments, I’m removing the effort/easy label that I mentioned on the Google Group post 😅

@tractorcow’s proposed solution gets a +1 from me. I think intermediary objects would be out of place - add() doesn’t make sense, and set() doesn’t match the convention anyway so probably makes it more confusing.

If $parent->write() is required to retain this newly associated relationship (that is one setup via attribute assignment)

I think this is an interesting question. HasManyList::add() and ManyManyList::add() will write the relations to the database, so if we’re vying for consistency then I suppose the setter should do the same. Is it reasonable for the “magic” approach to perform the write for you, or would that be unexpected? I’m leaning towards it being unexpected... interested to hear what other people think.

If we agree that it make sense to allow attribute-based access like $item = $parent->Item (similar to method-based access $item = $parent->Item()) and then have that attribute based access be magic (__get()) and lazily loaded unless specified otherwise.

Yes I think that makes sense. __set() could write it to the database and store the whole object in-memory, __get() could either fetch it from memory if it’s set or load it from the database if not.

For me, writing on $parent->Child = $childRecord is very much unexpected. Unless you ONLY write the relation. Even then, it gets complicated since $parent could be totally new ($parent->ID == 0). I'd suggest KISS principle applies here. _set(...) magic here only applies to auto-magically defining values for [Relation]ID and [Relation]Class fields on $parent when assignment takes place (so long as this is revealed to be a valid relation when setting).

I'm afraid I'm much more in favour of an intermediary HasOneObject which can have a similar API to a RelationList.
It's a relation and should be treated as such.

I'm afraid I'm much more in favour of an intermediary HasOneObject which can have a similar API to a RelationList.

That's how Laravel does it. That is:
// Actual Laravel style has-one association.
$photo->album()->associate($album)`
$photo->save(); // Important.

// ... would be the inverse of the has-many...
$album->photos()->add($photo)
// No need to save/write.

... and it makes sense.  However, currently the SilverStripe way would be most likely:
// SilverStripe has-one association.
$photo->Album = $album`
$photo->write();

// ...with the inverse being...
$album->Photos()->add($photo)
// Also no need to save/write.

And this is logical as @tractorcow pointed out since $photo->Album() would already be accessible as the method-based getter. That is the only major issue with this method-based ->associate(...) approach for has_one, since it completely changes the API by basically requiring a method like ->first() to fetch that relation first to terminate the class chain. That is, ->Album() can ONLY return either an Album instance or a HasOneList instance (hypothetical name). I think it's probably bad practice to enable the ability to, say, enable the ability to call a Foobar() method on Album like $photo->Album()->FooBar() while still allowing $photo->Album()->associate($album) (i.e. via magic __call(...)). That's some wicked black magic right there :tophat: :rabbit:

@dhensby  I'm in the middle on this because I like the idea of the intermediary HasOneList style approach (a la Eloquent), however we have backward compatibility issues to worry about and it's not a major benefit (as far as I can see) to go with that. Inventing it from scratch I'd say go the intermediary relation route, however, this is where we are :smile:

For me, writing on $parent->Child = $childRecord is very much unexpected. Unless you ONLY write the relation. Even then, it gets complicated since $parent could be totally new ($parent->ID == 0).

DataObject has a mechanism for storing has_one relations for unsaved records. They are stored in the $components property, so that when they are saved the actual foreign key can be generated.
I'm not sure how well it works though, and it may be worth looking at again.


:seedling: test-server: split New/Start/Ready phases

Summary
Decouple the steps to start the test server so we can define a new test-server, start it then call WaitForReady to ensure shard readiness
This is another change decoupled from the shard-standalone-vw branch from @ncdc
Related issue(s)
Follow-up to #2297 and another incremental step towards integrating the the shard-standalone-vw branch from @ncdc
Fixes #

/cc @ncdc @p0lyn0mial next incremental change derived from the shard-standalone-vw branch
Just to clarify - is it correct that when we switch to the standalone VW server, we can derive the availability of the VW server via the shard readyz?
Just wanted to ensure I correctly understood the rationale for this change as I'm not super familiar with the shard/VW bootstrapping atm.

/rename :seedling: test-server: split New/Start/Ready phases

we can derive the availability of the VW server via the shard readyz?

That doesn't seem right - they would be slightly interdependent at startup with e.g. apibinding initializer, but I wouldn't expect the shard -> vw flow.

Just to clarify - is it correct that when we switch to the standalone VW server, we can derive the availability of the VW server via the shard readyz?

The standalone VW server is a distinct binary, so we need to check readyz separately. When the VW server is run in-process it registers a startup hook so checking a shard readyz endpoint is enough.

/lgtm

Just to clarify - is it correct that when we switch to the standalone VW server, we can derive the availability of the VW server via the shard readyz?

The standalone VW server is a distinct binary, so we need to check readyz separately. When the VW server is run in-process it registers a startup hook so checking a shard readyz endpoint is enough.

Ok thanks - I guess this decoupling was added by @ncdc as a step towards that, but I don't see any check of the standalone VW server readyz in the branch, we can add that as a follow-up in the PR which enables standalone VW

/lgtm

/approve


Files wrapped by a filepond instance are locked, error message by FileReader object

Hello, I am trying to implement Filepond mainly as a UI container of files. I am just trying to create a proof of concept by uploading a single file to the server but I am getting a locking error on the FileReader object
This is what my code looks like:
var pond = FilePond.find(document.getElementById('file'));
var pondFiles = pond.getFiles();
var file = pondFiles[0].file;
var reader = new FileReader();
reader.readAsDataURL(file);

reader.onloadend = function () {
  alert('Reader error ::: ' + reader.error.message); // this alert with error: DOMException: The requested file could not be read, typically due to permission problems that have occurred after a reference to a file was acquired.
}

Thanks for any help in advance!

We require the issue template to be filled out on all new issues. The issue template helps us collect all the information we need to address your submission efficiently.
Your issue can be classified as an integration question or a question about specific features of FilePond we ask you kindly to submit it on Stack Overflow.


IOS call failed from terminate state

Hi, Thank you for developing this plugin.
when app in terminated state , call is accepted it will not open the app, after press on called field button app is open , I attached video below.
https://user-images.githubusercontent.com/69109162/173061216-b8c04b5c-b6d8-4cee-a81d-d40a4811282b.mov

IOS 15.4 , plugin version is latest , flutter 3.0

Hi dear, can you share the step to make the call work on IOS?
We can send push notification to IOS but we're not sure how to make the call from our server to the IOS device.

Hello! Also in need of good documentation how to send correct VOIP notification from Node.js and receive VOIP call using this plugin.

Hi, Thank you for developing this plugin.
when app in terminated state , call is accepted it will not open the app, after press on called field button app is open , I attached video below.
https://user-images.githubusercontent.com/69109162/173061216-b8c04b5c-b6d8-4cee-a81d-d40a4811282b.mov

Not sure if this will help, but I stumbled across this issue while seeing similar behavior – in my case, I was trying to connect to the call both by calling FlutterCallkitIncoming.activeCalls() and also in response to the CallEvent.ACTION_CALL_ACCEPT event. In my case, just removing the  FlutterCallkitIncoming.activeCalls() parts fixed it – the app correctly resumes from the terminated state and joins the call.

Have you find any solution for this issue?

@sadaqatdev, I had the same issue, the behaviour was just like in your video. The problem was, that I was calling FlutterCallkitIncoming.endAllCalls() when I didn't really need to. Could it be your case too?

Have you find any solution for this issue?

hello have you fide solution. for this ? @HensiStackApp


Add a deprecated flag and more details of the reason/alternatives for…

fix: https://github.com/kubernetes/website/issues/46722

@pegasas, it helps reviewers if the short PR title makes sense as it is. This one (“Add a deprecated flag and more details of the reason/alternatives for…”) is a big long and was also truncated.
Try something like "Mark foo as deprecated”. Concise is better.

@pegasas, it helps reviewers if the short PR title makes sense as it is. This one (“Add a deprecated flag and more details of the reason/alternatives for…”) is a big long and was also truncated.
Try something like "Mark foo as deprecated”. Concise is better.

Sure. I will correct the title to shorter one.

Current changes seems to be good.
@pegasas, It will be more clear if you can add below information also.

We can point to the official changelog or release announcement where the deprecation was mentioned.
(I hope we mentioned it!)

/sig network
/kind documentation

The upstream changes https://github.com/kubernetes/kubernetes/pull/115160 was closed. Is it a blocker for this PR?
/cc @T-Lakshmi @danwinship
For https://github.com/kubernetes/kubernetes/pull/115160#discussion_r1141382067, it is not first time to deprecate an enum value. i.e. the sig-storage team deprecated the Recycle reclaim policy. before they did that, they sent an email, please see Persistent Volume Recycler Deprecation Annoucement and referenced the email link in the https://github.com/kubernetes/kubernetes/blob/00236ae0d73d2455a2470469ed1005674f8ed61f/pkg/apis/core/types.go#L379. Unfortunately, they forgot to update the k8s.io/api/core/v1/types.go file.
/cc @sftim
FYI @hzxuzhonghu @dprotaso @howardjohn https://github.com/istio/istio/issues/34995 was closed due to inactivity. https://github.com/istio/istio/blob/master/pilot/pkg/serviceregistry/kube/controller/endpointslice.go#L249 may need to be updated to reflect the changes in the upstream.

/close
Closing because the upstream PR was closed.
Feel free to reopen if appropriate.


added changes

We'll leave this on a side branch for now, we want to keep the commit history of main clean. I also noticed your hrefs point to local directories, they won't work remotely, you will have to make them point to relative locations.


Suggestion: less chrome and compatibility with Gw2Hook

Hello,
Excellent tool.
Could you please add an option to turn off the black skin for 'Game Instances' so I can just put the options to the side and not have it overlap my UI?
Compatibility with Gw2Hook would be appreciated. I understand this may be quite difficult though.

Gw2Hook is already compatible FYI, even before v1.7 which introduced an injection system.


GenerateReleaseNotes - The remote server returned an error: (401) Unauthorized.

VSTS Extension you are using
GenerateReleaseNotes
Where are you running it?
on-premise TFS Version 15.117.26714.0
Version of Extension/Task
newest version, just downloaded today
Expected behavior and actual behavior
Expected behavior is for auth to pass. I am getting the following error when using "Use Default Creds" and when not using it.
System.Management.Automation.MethodInvocationException: Exception calling "DownloadString" with "1" argument(s): "The remote server returned an error: (401) Unauthorized." ---> System.Net.WebException: The remote server returned an error: (401) Unauthorized.
My agents are running as "Network Service" could this be the issue?

We would like to use the OAuth token which is available as part of the build process.

You should be able to just tick the box that allows scripts to use the OAuth token of the agent. If you then set the step to not use Default Creds then you should be fine.
If you still have problems then I'll dig into it further.

I have tried that.
System.Management.Automation.MethodInvocationException: Exception calling "DownloadString" with "1" argument(s): "The remote server returned an error: (401) Unauthorized." ---> System.Net.WebException: The remote server returned an error: (401) Unauthorized.
The other tasks are passing it like this.

I'll have to investigate this more then, I was under the impression that you only needed to enable the use of OAuth for this task to work correctly as it pulls that out of the environment variables.
something worth looking at would be the permissions given to the Project Collection Build Service user, that's what is used for the API calls so could be a permission somewhere denying read on various things.
I've got another possible area that could be causing the problem, it might be a few days before I can get into this properly though.

Thanks Chris, I will keep digging also.

I have been out the loop for a while, so sorry for the delay in getting involved. Not sure if you are still having problems.
As I wrote the task I might be able to provide some insight, so a few points I would comment on

The is no support for oAuth in this task, it does not need it as the task is already running in the context of the build/release agent
Checking the 'allow this build to access oAuth' as @ChrisLGardner  says is only needed to make environment variable etc. available to other process e.g. if you want one build to trigger another build.
This task should just work auth wise the build /release agent always has the rights to access the REST API on the server that the build/release was triggered on. To be honest there should be no need to even select the 'Use default creds'. This was only put in for one client strange on prem edge case.
It is interesting your comment that you are running as 'network service', something I would never recommend for any service (TFS or otherwise) as it make cross machine security awkward in a domain and anyway you always want to run services as lowest privileged. For this reason my TFS test box (and all production systems I run) have a mydomain\tfsservice account for the server process and mydomain\tfsbuild for the build agents.

So are you still having any problems?
Have you also looked at my CrossPlatform Generate Release Notes (also in the market place), it is written in Node.JS and I wonder if it has similar issues.

As per my standard practice I am closing this issue as I have heard nothing for a while. If this issue remains a problem please do not hesitate to re-open it


Update DataStore Auth Integration Tests to use V2 Transformer

Describe the bug
Upgrade the schema to use V2 Transform for the tests located in the DataStoreAuthIntegration tests.
These tests were written for V1 Transformer and will not work with V2 Transformer. In V2 Transform, the operations defined on the auth rule indicate which operations are allowed and omitted operations are denied by default. The behavior difference causes the subscription operations for the following V1 schema to fail.
The change is

Update the README.md to indicate use of V2 Transformer and last known CLI version used to test.
Update the schema to include .read for the subscriptions to work.
Run the tests and make sure they are successful and update the provisioning in the pipeline that runs the integration tests.

This is more of an improvement since we still want the older tests. I'll move this over to internal tracking


Replace occurrences of PCM with PCUI in pcluster-manager.yaml and pcluster-manager-cognito.yaml and rename files accordingly

Description
This PR aims to replace occurrences of PCM / PclusterManager / ParallelClusterManager with "ParallelCluster UI" and rename files accordingly.

pcluster-manager.yaml -> parallelcluster-ui.yaml
pcluster-manager-cognito.yaml -> parallecluster-ui-cognito.yaml
In some resources the PclusterManager prefix has been removed since considered not useful. All resources created will still have the stack name as prefix.

How Has This Been Tested?

Successfully deployed stacks on my personal account

References

PR Quality Checklist

[ ] I added tests to new or existing code
[ ] I removed hardcoded strings and used our i18n solution instead (see here)
[ ] I made sure no sensitive info gets logged at any time in the codebase (see here) (e.g. no user info or details, no stacktraces, etc.)
[ ] I checked that infrastructure/update_infrastructure.sh runs without any error
[ ] I checked that npm run build builds without any error
[ ] I checked that clusters are listed correctly
[ ] I checked that a new cluster can be created (config is produced and dry run passes)
[ ] I checked that login and logout work as expected

In order to increase the likelihood of your contribution being accepted, please make sure you have read both the Contributing Guidelines and the Project Guidelines
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

LGTM, let's put this one in Draft. We'll merge it when we have a PR ready to also fix the GH Actions workflows, this way we don't break the workflows for all subsequent PRs


Reloading the page with setRouteLeaveHook changes the location hash on going back in history

Version
2.4.1
2.5.2
2.6.0
Test Case
When using context.router.setRouteLeaveHook, and returning a simple false in hook to prevent the user form navigating away, the hash value in the url is changed to next location. It works as expected if the user comes to the page from an another page, but it doesn't work once the page is reloaded.
Recreated here: https://gopikrishna19.github.io/react-router-issue
Source here: https://github.com/Gopikrishna19/react-router-issue
Steps to reproduce

Go to https://gopikrishna19.github.io/react-router-issue.
Click sub page link.
Click back button -> prevents the navigation, does not change/resets the hash.
Reload the page
Click back button -> prevents the navigation, but sets the hash to next location

Clicking the app link works as expected -- doesn't change the hash nor lets the user navigate.
Expected Behavior
Should retain the hash value
Actual Behavior
Hash value is changed to that of previous state

jic, if anyone is wondering, it has the same effect even when I use a string to show up the confirm dialog and I click cancel, the view remains the same but the hash value changes in the url bar.

Dupe https://github.com/ReactTraining/history/issues/157

@taion I did not look at the history's issues. Thanks


[AIRFLOW-3055] add get_dataset and get_datasets_list to bigquery_hook

Make sure you have checked all steps below.
Jira

[ ] My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

https://issues.apache.org/jira/browse/AIRFLOW-3055
In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.

Description

[ ] Here are some details about my PR, including screenshots of any UI changes:

Tests

[ ] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[ ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters (not including Jira issue reference)
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Documentation

[ ] In case of new functionality, my PR adds documentation that describes how to use it.

When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.

Code Quality

[ ] Passes git diff upstream/master -u -- "*.py" | flake8 --diff

@kaxil,  I need your advice and review..
I have issues with names or if it's okay - I leave it as it is.
Main idea of operators - get possible to check BQ environment before pipeline work start, for example, if dataset not exist - I want to create it or alert support team with detailed issue.  To get such information I need CheckDatasetOperator. And also get possible to check list of datasets, if I have a big env-prerequisites (several datasets using in one DAG).
after logic discussion if all ok - I will add tests and info in docs.
I have some doubts about keeping logic in operator (BigQueryDatasetsListCheckExistOperator)
https://github.com/apache/incubator-airflow/pull/3894/files#diff-6424e11be84d0621ef24935634e3db55R683
If you could, provide some recomendations ok it or not and what are more good place to move it.

Codecov Report

Merging #3894 into master will increase coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3894      +/-   ##
==========================================
+ Coverage   77.53%   77.55%   +0.01%
==========================================
  Files         198      198
  Lines       15847    15847
==========================================
+ Hits        12287    12290       +3
+ Misses       3560     3557       -3

Impacted Files
Coverage Δ

airflow/jobs.py
82.74% <0%> (+0.26%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e68ad5f...bda70f4. Read the comment docs.

@kaxil , I corrected

@kaxil , I very quick.. :D please review

I have updated a few minor changes. Let's wait for the CI to pass. If it passes, I will squash commit and merge.

@kaxil , thanks, as I see - all okay with CI


Unexpected hydration behavior

I copy-paste example and run it:
$rawJson = <<<TEST
{
    "id": 1337,
    "content": "Do you like potatoes?",
    "date": "1957-07-23 13:37:42",
    "answers": [
        {
            "user": "Ella F.",
            "message": "I like potatoes",
            "date": "1957-07-31 15:28:12"
        },
        {
            "user": "Louis A.",
            "message": "And I like tomatoes",
            "date": "1957-08-13 09:05:24"
        }
    ]
}
TEST;

final class Thread
{
    public function __construct(
        public readonly int $id,
        public readonly string $content,
        public readonly DateTimeInterface $date,
        /** @var Answer[] */
        public readonly array $answers,
    ) {}
}

final class Answer
{
    public function __construct(
        public readonly string $user,
        public readonly string $message,
        public readonly DateTimeInterface $date,
    ) {}
}

$result = (new \CuyZ\Valinor\MapperBuilder())
    ->mapper()
    ->map(
        Thread::class,
        new \CuyZ\Valinor\Mapper\Source\JsonSource($rawJson)
    );

I got following result:

But I expect array of  Answer objects in answers property. What am I doing wrong?
P.S. PHP 8.1.1, library version: 0.5.0

Also @var array<Answer> or @var Answer[] not working in property declaration (i.e. without constructor or static methods).

Indeed the example seems incorrect, I'll change it to something actually working.
@roquie it should be working with properties without constructor, could you post here the code that's not working?

Indeed the example seems incorrect, I'll change it to something actually working.

Good :)

@roquie it should be working with properties without constructor, could you post here the code that's not working?

Oh, sorry. Now I double checked this. It's my mistake with camelCase/snake_case notation.
By the way, how to automatically convert case from source to DTO? Maybe via Attributes or "CustomMappingStrategy" existing?


[#2607] Fix collpased accordion sections not opening after editing description

This fix at least ensures that the contents of the accordion are always visible, but it doesn't animate for the first expanding after changing another property. I would welcome thoughts as to a better fix.
Found a better fix that properly stores the height.

[!Note]
I switched from using setTimeout to requestAnimationFrame for the accordion timing here because it felt more appropriate, but that isn't part of the actual bug fix.

I tracked this to a regression caused by the changes here: https://github.com/foundryvtt/dnd5e/pull/2573/files#diff-48f5cd8733c53432fef055aab117363d87bf159c8a3088a567bca60dbf4af90e
If we remove those lines, the issue goes away. I am having trouble reproducing the issue that those lines were meant to fix, so perhaps we could just remove them? Or perhaps address the issue they are fixing in a different way?

I tracked this to a regression caused by the changes here: https://github.com/foundryvtt/dnd5e/pull/2573/files#diff-48f5cd8733c53432fef055aab117363d87bf159c8a3088a567bca60dbf4af90e
If we remove those lines, the issue goes away. I am having trouble reproducing the issue that those lines were meant to fix, so perhaps we could just remove them? Or perhaps address the issue they are fixing in a different way?

If I remove those lines this is what I get:
https://github.com/foundryvtt/dnd5e/assets/19979839/19d09c6c-6cd2-4f3c-9edf-ada8645c8848
Might be a browser-specific issue.

Yeah I don't experience any flash when I tested it, but it is deliberately allowing them to render at full size in order to get the correct height, so I can see why it would happen. It might be related to browsers, or just general FPS and monitor refresh rate. It's fine to solve it with some extra calls to getBoundingClientRect then if it avoids that issue.


pachctl list commit returns size column of 0B - tested on opencv / 2.0.0-alpha.21

On 2.0.0-alpha.21 - OpenCV example
pachctl list commit images, pachctl list commit edges,  and pachctl list commit montage all return empty sizes.

Note: If those sizes can't be updated as part of the list commit, then this column should be removed and exposed with a flag.

This should be addressed in the current alpha.


Sliders not working on mobile

Time and Speed sliders are not working with touch event (jquery UI related).
I propose two solutions for this problem :

Use an input[type=range]

Pro : easy, native
Cons : not supported on IE < 10 without polyfill

Implement a custom slider with L.draggable like kartena/Leaflet.zoomslider

Pro : No dependency
Cons : harder to maintain

Any thought on that @bielfrontera ?

first attempt with solution 2 in #41


Dataset Details (DDN-SIRR)

Hi, according to the paper, for the dataset DDN-SIRR, 9100 pairs of synthesized images and some real-world images are used as the training set, and another 20 pairs as testing set.  I have some questions as follows:

How many real-world images are in the training set? Are the 147 images in the path 'Syn2Real/data/train/derain/real_input/'?
The original dataset  of Fu et al. [9] consists of 9100 training pairs and 4900 testing pairs, did you use the 4900 pairs for testing?
Could you share the data split file for the 9100 training pairs, since the ratio has been updated to 12600:1400 on their official websit.

Thank you very much!

Hi @kylin013 ,

yes, we used 147 real-world images in the training set as unlabelled data set, and they are provided in "./data/train/derain/real_input/"
Yes, the original dataset of Fu et al. [9] consists of 9100 training pairs and 4900 testing pairs, in this publication we didn't use 4900 pairs for testing.
updated dataset of  Fu et al. [9]  consists of 12600:1400 training: testing split. their dataset is created using 14 rain streaks for each image, so using 1000 images a total of 14000 images were created and split into training and testing. Earlier I think images index with 1 to 650 were used for training and rest for testing. In the updated dataset images indexed with 1to 900 are used as training and 901 to 1000 are for testing. So in our training processes we used images indexed with 1to 650 as labelled dataset in our training.

Hi @rajeevyasarla , thanks so much for your reply! I still have a question, did you use the 20 images in "./data/test/derain/SIRR_test/" for both validation and testing, and which 10 of them are dense (or sparse)?

Hi @kylin013 , Yes I use 20 images "./data/test/derain/SIRR_test/" for testing where 10 are dense and 10 are sparse

Hi @rajeevyasarla , which 10 images are dense? the images index with 1 to 10?

Hi @kylin013 , images indexed from 1 to 10 are dense, and 11-20 are sparse


Introduced JWT based authentication for communication with registry

Reason for this PR
The instance registry will require authentication based on JSON Web Tokens (JWTs) soon (as described here). The components of the Delphi system have to be able to generate valid JWTs based on a secret that is either passed in an environment variable or in the configuration file. They have to add the encoded token in the Authorization header of every HTTP request issued to the registry. For more information see the respective issue.
Changes in this PR

The WebApi is now able to generate valid JWTs based on a secret specified in an environment variable (using HS256 signature algorithm)
The WebApi sends an encoded valid JWT with every request issued to the registry
A bug has been fixed that prevented the application from shutting down when trying to report shutdown to the registry

Codecov Report

Merging #41 into develop will decrease coverage by <.01%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           develop     #41      +/-   ##
==========================================
- Coverage      1.4%   1.39%   -0.01%
==========================================
  Files           18      19       +1
  Lines          427     429       +2
  Branches        31      33       +2
==========================================
  Hits             6       6
- Misses         421     423       +2

Impacted Files
Coverage Δ

...ain/scala/de/upb/cs/swt/delphi/webapi/Server.scala
0% <0%> (ø)
:arrow_up:

...la/de/upb/cs/swt/delphi/webapi/Configuration.scala
0% <0%> (ø)
:arrow_up:

...swt/delphi/webapi/authorization/AuthProvider.scala
0% <0%> (ø)

...t/delphi/instancemanagement/InstanceRegistry.scala
0% <0%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4dbb9df...90cc6e3. Read the comment docs.


Add test framework generation

add mocha chai unit testing support (unit testing folders and config files)

Yes, sounds like a great idea. It can be added as a flag for users to choose the framework (for when the next person wants jest for example). Like maybe --test=mocha or something.

Wonderful idea, had the same thought for passport authentication. @dougwilson  Maybe I could take this up once I'm done learning about testing. Could add support for mocha and jest framework


Bosch vacuum robot cause adapter to crash

the Bosch vacuum robot (BCR1ACDE/03) cause the adapter to crash in loop until iobroker detects loop restarts
This always happens if a active program ist running.
I use this device in ioBroker only for vis, no scripts are involved.
Adapter version 1.4.1
`

host.iobroker
2024-10-23 20:21:15.719
info
Restart adapter system.adapter.homeconnect.0 because enabled

host.iobroker
2024-10-23 20:21:15.718
error
instance system.adapter.homeconnect.0 terminated with code 6 (UNCAUGHT_EXCEPTION)

host.iobroker
2024-10-23 20:21:15.718
error
Caught by controller[1]: at /opt/iobroker/node_modules/iobroker.homeconnect/main.js:665:24

host.iobroker
2024-10-23 20:21:15.718
error
Caught by controller[1]: at Homeconnect.setState (/opt/iobroker/node_modules/@iobroker/js-controller-adapter/src/lib/adapter/adapter.ts:7720:21)

host.iobroker
2024-10-23 20:21:15.718
error
Caught by controller[1]: at Homeconnect._setState (/opt/iobroker/node_modules/@iobroker/js-controller-adapter/src/lib/adapter/adapter.ts:7749:29)

host.iobroker
2024-10-23 20:21:15.718
error
Caught by controller[1]: at Validator.validateSetStateObjectArgument (/opt/iobroker/node_modules/@iobroker/js-controller-adapter/src/lib/adapter/validator.ts:422:19)

host.iobroker
2024-10-23 20:21:15.718
error
Caught by controller[1]: Error: The state contains the forbidden properties x, y!

host.iobroker
2024-10-23 20:21:15.717
error
Caught by controller[0]: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:

homeconnect.0
2024-10-23 20:21:15.002
warn
Terminated (UNCAUGHT_EXCEPTION): Without reason

homeconnect.0
2024-10-23 20:21:15.000
info
terminating

homeconnect.0
2024-10-23 20:21:14.969
error
The state contains the forbidden properties x, y!

homeconnect.0
2024-10-23 20:21:14.968
error
Error: The state contains the forbidden properties x, y! at Validator.validateSetStateObjectArgument (/opt/iobroker/node_modules/@iobroker/js-controller-adapter/src/lib/adapter/validator.ts:422:19) at Homeconnect._setState (/opt/iobroker/node_modules/@iobroker/js-controller-adapter/src/lib/adapter/adapter.ts:7749:29) at Homeconnect.setState (/opt/iobroker/node_modules/@iobroker/js-controller-adapter/src/lib/adapter/adapter.ts:7720:21) at /opt/iobroker/node_modules/iobroker.homeconnect/main.js:665:24

homeconnect.0
2024-10-23 20:21:14.799
error
unhandled promise rejection: The state contains the forbidden properties x, y!

homeconnect.0
2024-10-23 20:21:14.798
error
Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch().

homeconnect.0
2024-10-23 20:21:12.799
info
BOSCH-BCR1ACDE-68A40E5666A9/programs/active: There is no program active.

homeconnect.0
2024-10-23 20:21:12.789
info
BOSCH-BCR1ACDE-68A40E5666A9/programs/active/options: There is no program active.

homeconnect.0
2024-10-23 20:21:12.326
info
No array: selected

homeconnect.0
2024-10-23 20:21:12.319
info
No array: active

homeconnect.0
2024-10-23 20:21:11.226
info
Found 2 devices

homeconnect.0
2024-10-23 20:21:09.512
info
starting. Version 1.4.1 in /opt/iobroker/node_modules/iobroker.homeconnect, node: v20.18.0, js-controller: 6.0.11

host.iobroker
2024-10-23 20:21:05.461
info
instance system.adapter.homeconnect.0 in version "1.4.1" started with pid 85038

host.iobroker
2024-10-23 20:21:04.499
info
"system.adapter.homeconnect.0" enabled

`

plaese add version info of
js-controller
node

node v20.18.0
js-controller 6.0.11

Please test GitHub version

GitHub Version reports following errors, but don't crash in loop

homeconnect.0
2024-10-24 09:23:53.878
info
State value to set for "homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.programs.active.options.ConsumerProducts_CleaningRobot_Option_TargetAreas" has to be one of type "string", "number", "boolean" but received type "object"

homeconnect.0
2024-10-24 09:23:53.485
info
BOSCH-BCR1ACDE-68A40E5666A9/programs/available/ConsumerProducts.CleaningRobot.Program.Cleaning.CleanMap: Program currently not available.

homeconnect.0
2024-10-24 09:23:53.179
info
BOSCH-BCR1ACDE-68A40E5666A9/programs/available/ConsumerProducts.CleaningRobot.Program.Basic.GoHome: Program currently not available.

homeconnect.0
2024-10-24 09:23:53.168
info
BOSCH-BCR1ACDE-68A40E5666A9/programs/available/ConsumerProducts.CleaningRobot.Program.Cleaning.CleanAll: Program currently not available.

homeconnect.0
2024-10-24 09:23:52.900
error
Error: The state contains the forbidden properties x, y!

homeconnect.0
2024-10-24 09:23:52.899
error
Value: '[object Object]'

homeconnect.0
2024-10-24 09:23:52.898
error
failed set state BOSCH-BCR1ACDE-68A40E5666A9.programs.selected.options.GoTo.ConsumerProducts_CleaningRobot_Option_TargetSpot

homeconnect.0
2024-10-24 09:23:52.448
error
Error: The state contains the forbidden properties x, y!

homeconnect.0
2024-10-24 09:23:52.447
error
Value: '[object Object]'

homeconnect.0
2024-10-24 09:23:52.446
error
failed set state BOSCH-BCR1ACDE-68A40E5666A9.programs.selected.options.GoTo.ConsumerProducts_CleaningRobot_Option_TargetSpot

homeconnect.0
2024-10-24 09:23:52.432
info
State value to set for "homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.programs.active.options.ConsumerProducts_CleaningRobot_Option_TargetAreas" has to be one of type "string", "number", "boolean" but received type "object"

homeconnect.0
2024-10-24 09:23:52.306
info
No array: selected

homeconnect.0
2024-10-24 09:23:52.302
info
No array: active

homeconnect.0
2024-10-24 09:23:46.764
info
Found 2 devices

homeconnect.0
2024-10-24 09:23:46.152
info
starting. Version 1.4.1 (non-npm: iobroker-community-adapters/ioBroker.homeconnect#777887fd606b9561e180fd4dedad21bbe7dd80bf) in /opt/iobroker/node_modules/iobroker.homeconnect, node: v20.18.0, js-controller: 6.0.11

Please reinstall again

reinstalled and now got only some warnings.

homeconnect.0
2024-10-24 11:18:53.050
info
BOSCH-BCR1ACDE-68A40E5666A9/programs/available/ConsumerProducts.CleaningRobot.Program.Cleaning.CleanAll: Program currently not available.

homeconnect.0
2024-10-24 11:18:52.320
info
BOSCH-BCR1ACDE-68A40E5666A9/programs/available/ConsumerProducts.CleaningRobot.Program.Cleaning.CleanMap: Program currently not available.

homeconnect.0
2024-10-24 11:18:52.137
info
BOSCH-BCR1ACDE-68A40E5666A9/programs/available/ConsumerProducts.CleaningRobot.Program.Basic.GoHome: Program currently not available.

homeconnect.0
2024-10-24 11:18:51.757
info
State value to set for "homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.programs.active.options.BSH_Common_Option_ElapsedProgramTime" has to be type "number" but received type "string"

homeconnect.0
2024-10-24 11:18:51.589
warn
State "homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.programs.selected.options.GoTo".ConsumerProducts_CleaningRobot_Option_TargetSpot" has no existing object, this might lead to an error in future versions

homeconnect.0
2024-10-24 11:18:51.581
info
No array: selected

homeconnect.0
2024-10-24 11:18:51.579
info
No array: active

homeconnect.0
2024-10-24 11:18:51.516
warn
State "homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.programs.selected.options.GoTo".ConsumerProducts_CleaningRobot_Option_ReferenceMapId" has no existing object, this might lead to an error in future versions

homeconnect.0
2024-10-24 11:18:51.486
warn
Used invalid characters: homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.programs.selected.options.GoTo".ConsumerProducts_CleaningRobot_Option_TargetSpot changed to homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.programs.selected.options.GoTo_.ConsumerProducts_CleaningRobot_Option_TargetSpot

homeconnect.0
2024-10-24 11:18:51.478
warn
Used invalid characters: homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.programs.selected.options.GoTo" changed to homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.programs.selected.options.GoTo_

homeconnect.0
2024-10-24 11:18:51.476
warn
Used invalid characters: homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.programs.selected.options.GoTo".ConsumerProducts_CleaningRobot_Option_ReferenceMapId changed to homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.programs.selected.options.GoTo_.ConsumerProducts_CleaningRobot_Option_ReferenceMapId

homeconnect.0
2024-10-24 11:18:51.452
warn
Used invalid characters: homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.programs.selected.options.GoTo" changed to homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.programs.selected.options.GoTo_

homeconnect.0
2024-10-24 11:18:51.429
warn
State "homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.programs.selected.options.GoTo".ConsumerProducts_CleaningRobot_Option_TargetSpot" has no existing object, this might lead to an error in future versions

homeconnect.0
2024-10-24 11:18:51.417
warn
State "homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.programs.selected.options.GoTo".ConsumerProducts_CleaningRobot_Option_ReferenceMapId" has no existing object, this might lead to an error in future versions

homeconnect.0
2024-10-24 11:18:51.343
info
State value to set for "homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.status.ConsumerProducts_CleaningRobot_Status_Lifted" has to be type "boolean" but received type "string"

homeconnect.0
2024-10-24 11:18:51.341
info
State value to set for "homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.status.ConsumerProducts_CleaningRobot_Status_Lost" has to be type "boolean" but received type "string"

homeconnect.0
2024-10-24 11:18:51.339
info
State value to set for "homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.status.ConsumerProducts_CleaningRobot_Status_DustBoxInserted" has to be type "boolean" but received type "string"

homeconnect.0
2024-10-24 11:18:51.311
info
State value to set for "homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.status.BSH_Common_Status_BatteryLevel" has to be type "number" but received type "string"

homeconnect.0
2024-10-24 11:18:51.304
warn
Used invalid characters: homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.programs.selected.options.GoTo".ConsumerProducts_CleaningRobot_Option_TargetSpot changed to homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.programs.selected.options.GoTo_.ConsumerProducts_CleaningRobot_Option_TargetSpot

homeconnect.0
2024-10-24 11:18:51.278
warn
Used invalid characters: homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.programs.selected.options.GoTo" changed to homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.programs.selected.options.GoTo_

homeconnect.0
2024-10-24 11:18:51.276
warn
Used invalid characters: homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.programs.selected.options.GoTo".ConsumerProducts_CleaningRobot_Option_ReferenceMapId changed to homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.programs.selected.options.GoTo_.ConsumerProducts_CleaningRobot_Option_ReferenceMapId

homeconnect.0
2024-10-24 11:18:51.261
warn
Used invalid characters: homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.programs.selected.options.GoTo" changed to homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.programs.selected.options.GoTo_

homeconnect.0
2024-10-24 11:18:51.242
info
State value to set for "homeconnect.0.BOSCH-BCR1ACDE-68A40E5666A9.programs.active.options.BSH_Common_Option_ElapsedProgramTime" has to be type "number" but received type "string"

homeconnect.0
2024-10-24 11:18:49.867
info
Found 2 devices

homeconnect.0
2024-10-24 11:18:49.295
info
starting. Version 1.4.1 (non-npm: iobroker-community-adapters/ioBroker.homeconnect#08311a0c0ced2a39c20bc5a6ab0dc182ead81700) in /opt/iobroker/node_modules/iobroker.homeconnect, node: v20.18.0, js-controller: 6.0.11

host.iobroker
2024-10-24 11:18:45.749
info
instance system.adapter.homeconnect.0 in version "1.4.1" (non-npm: iobroker-community-adapters/ioBroker.homeconnect#08311a0c0ced2a39c20bc5a6ab0dc182ead81700) started with pid 99557

please reinstall again and check if the device is controllable

Everything looks good. All devices (dishwasher and robot) are controllable and can't see any errors and warnings anymore.


HIVE-26832: Implement SHOW PARTITIONS for Iceberg Tables

Implement SHOW PARTITIONS for Iceberg Tables
What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

Yes.
How was this patch tested?

q tests.

Yes Attila.

Will reopen a new PR on the latest master branch.


[Bug] Undiagnosed parse error

Reported on Discord
[entry]
bg = "waves"
text = "Hello World"
options = [ ["(T)est", "card"] ]

Causes
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "_toml.py", line 349, in load
  File "_toml.py", line 343, in loads
  File "_toml.py", line 170, in __init__
  File "_toml.py", line 298, in _parse_line
TOMLError: Malformed string, check out your quotes in line 4

Solved by 681197f7d2b3774f7d13513d8b7c4aeadbd89ad7


Chinese v0.2

Chinese Translation for v0.2

Add: the Evo Tree 61 should be Fish boosts "FISH BODY" by ^...


seg fault trying to modify global attribute

Hi, I am trying to modify a global attribute ("loess_span") using:
ncputatt(filename, "loess_span", Dict("loess_span", loess_span))
in an existing netcdf file:
netcdf my_chl_qggrid_file {
dimensions:
        lon = 301 ;
        lat = 61 ;
        time = UNLIMITED ; // (3104 currently)
variables:
        double lon(lat, lon) ;
                lon:_FillValue = 9999. ;
        double lat(lat, lon) ;
                lat:_FillValue = 9999. ;
// global attributes:
                :loess_span = 0. ;

but I get the following seg fault:
julia> include("make_chl_loess.jl")

signal (11): Segmentation fault
NC4_put_att at /opt/lib/libnetcdf.so (unknown line)
nc_put_att_float at /opt/lib/libnetcdf.so (unknown line)
_nc_put_att_float_c at /home/emason/.julia/v0.3/NetCDF/src/netcdf_c_wrappers.jl:116
jlcall__nc_put_att_float_c_20661 at  (unknown line)
jl_apply_generic at /opt/julia/julia/usr/bin/../lib/libjulia.so (unknown line)
_nc_put_att at /home/emason/.julia/v0.3/NetCDF/src/netcdf_helpers.jl:102
jlcall__nc_put_att_20659 at  (unknown line)
jl_apply_generic at /opt/julia/julia/usr/bin/../lib/libjulia.so (unknown line)
putatt at /home/emason/.julia/v0.3/NetCDF/src/NetCDF.jl:149
jlcall_putatt_20629 at  (unknown line)
jl_apply_generic at /opt/julia/julia/usr/bin/../lib/libjulia.so (unknown line)
putatt at /home/emason/.julia/v0.3/NetCDF/src/NetCDF.jl:160
jl_apply_generic at /opt/julia/julia/usr/bin/../lib/libjulia.so (unknown line)
ncputatt at /home/emason/.julia/v0.3/NetCDF/src/NetCDF.jl:172
jl_apply_generic at /opt/julia/julia/usr/bin/../lib/libjulia.so (unknown line)
unknown function (ip: 402299928)
unknown function (ip: 402296441)
unknown function (ip: 402363022)
unknown function (ip: 402365696)
jl_load at /opt/julia/julia/usr/bin/../lib/libjulia.so (unknown line)
include at ./boot.jl:245
jl_apply_generic at /opt/julia/julia/usr/bin/../lib/libjulia.so (unknown line)
include_from_node1 at ./loading.jl:128
jl_apply_generic at /opt/julia/julia/usr/bin/../lib/libjulia.so (unknown line)
unknown function (ip: 402299928)
unknown function (ip: 402296441)
unknown function (ip: 402363022)
jl_f_top_eval at /opt/julia/julia/usr/bin/../lib/libjulia.so (unknown line)
eval_user_input at REPL.jl:53
jlcall_eval_user_input_20293 at  (unknown line)
jl_apply_generic at /opt/julia/julia/usr/bin/../lib/libjulia.so (unknown line)
anonymous at task.jl:95
jl_handle_stack_switch at /opt/julia/julia/usr/bin/../lib/libjulia.so (unknown line)
julia_trampoline at /opt/julia/julia/usr/bin/../lib/libjulia.so (unknown line)
unknown function (ip: 4199581)
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 4199635)
unknown function (ip: 0)
Segmentation fault

Does NetCDF.jl allow this operation on global attributes, or is my call to ncputatt wrong?
Thanks

Hmm, I haven|t seen this before, in general this should work. What is the type of your attribute and which julia version do you use?

I guess it's a float 64; the nc file where the attribute was first set was created with python.netcdf4 with the line:
nc.loess_span = 0.
and in my Julia script I have:
loess_span = 0.1
just before the call to ncputatt()
I am using Julia Version 0.3.12-pre+5 Commit 24138e7* (33 days old release-0.3)

I think your DIct is not constructed correctly, could you please try the syntax:
ncputatt(filename, "loess_span", {"loess_span" => loess_span})

and test if it works. There might still be an issue with the package, it should never segfault, but can you try if this works for you?

Perfect, it worked:
:loess_span = 0.1f ;
Can you explain why the syntax I used failed?  (I've only been using Julia for about a week.)
One other comment,  in the documentation it says to use ncsync as:
ncsync( [ filename ] )
but I found it would not accept the filename: instead I used just:
ncsync()

Is it necessary to use ncclose() when using the high level interface?
Thanks for your help.

The Dict constructor you have been using accepts an iterable of keys as the first and an iterable of values as the second argument. So it just created a Dict containing a single pair
julia> Dict("loess_span", 0.1)
Dict{Any,Any} with 1 entry:
  'l' => 0.1

This is quite unfortunate and the Dict construction syntax is going to be changed in julia v0.4.
You are right with ncsync(filename) I forgot to implement this so far, but ncsync() should do what you want. I will add this to the new version.
Regarding ncclose, if you open your files in write mode, you should definitely close them to make sure the data gets actually written to disk. If you are in read mode and you don't leave millions of files open it should be ok if you don't ncclose, files will be closed when you quit julia.

Ok, great, thanks again for your help.  Evan


Fix .apkg

Currently results in error and users need to manually change, maybe create a whole new version instead also adding more fields as described in #1

https://www.ankiapp.com/support/solutions/f6fd2c9f/export-decks-from-anki-apkg-for-use-with-ankiapp/

Added new deck, still needs to be tested

Seems to be working fine, might need to adjust things later


How to detect an empty array

I'm struggling to come up with an expression to determine if an array is empty. ex: dependsOn: [].

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 77c58a78-a567-e99a-9eb7-62dddd1b90b6
Version Independent ID: 680a79bc-11de-39fc-43e3-e07dc762db18
Content: Expressions - Azure Pipelines
Content Source: docs/pipelines/process/expressions.md
Product: devops
Technology: devops-cicd
GitHub Login: @juliakm
Microsoft Alias: jukullam

Hi @poltj18, we'll need to check with the product team to find out more about this. For now, you can reach out to the Azure DevOps Developer Community, which is monitored by the product team to see if you can get a solid answer there!

@juliakm , Thanks for your reply. I'm interested to hear what they come back with.

This isn't something that is supported. You can make a feature request to the product team at the Azure DevOps Developer Community. As a workaround, you might look into solving this at runtime with bash or another scripting language.

@juliakm , Thanks for the update. Unfortunately, I don't think the runtime solution will work for my use case. Specifically looking to allow users to provide a list of tasks via bringYourOwnSteps: [] which would override my default list of tasks in my YAML template. For now, it seems possible to declare bringYourOwnSteps: '' and check if bringYourOwnSteps is an empty string. Somewhat surprisingly, the caller can still specify a list of steps.