StartContainerExecAsync fails with DockerContainerNotFoundException

Output of dotnet --info:
.NET Command Line Tools (1.0.4)

Product Information:
 Version:            1.0.4
 Commit SHA-1 hash:  af1e6684fd

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  16.04
 OS Platform: Linux
 RID:         ubuntu.16.04-x64
 Base Path:   /usr/share/dotnet/sdk/1.0.4

What version of Docker.DotNet?:
2.124.3

Steps to reproduce the issue:

sudo docker ps -a

CONTAINER ID        IMAGE                           COMMAND                  CREATED             STATUS                           PORTS                                              NAMES
25b933366a88        jrottenberg/ffmpeg              "ffmpeg -i https:/..."   34 minutes ago      Exited (0) 34 minutes ago                                                           wonderful_cray
25c05dbf7e65        ravendb/ravendb:ubuntu-latest   "/bin/sh -c /opt/r..."   42 minutes ago      Up 42 minutes                    <IP_ADDRESS>:8080->8080/tcp, <IP_ADDRESS>:38888->38888/tcp   quizzical_poincare
2e04b598816d        ravendb/ravendb:ubuntu-latest   "/bin/sh -c /opt/r..."   About an hour ago   Exited (137) 45 minutes ago                                                         mystifying_lalande
89aecfa6122f        ravendb/ravendb:ubuntu-latest   "/bin/sh -c /opt/r..."   About an hour ago   Exited (137) About an hour ago                                                      hopeful_bardeen
9d650c177226        ravendb/ravendb:ubuntu-latest   "/bin/sh -c /opt/r..."   About an hour ago   Exited (255) About an hour ago   <IP_ADDRESS>:8080->8080/tcp, <IP_ADDRESS>:38888->38888/tcp   eloquent_minsky
2906902d11a5        ravendb/ravendb:ubuntu-latest   "/bin/sh -c /opt/r..."   3 hours ago         Exited (137) About an hour ago                                                      brave_volhard
bc4f15e3cbc4        ravendb/ravendb:ubuntu-latest   "/bin/sh -c /opt/r..."   3 hours ago         Exited (137) 3 hours ago                                                            nervous_almeida

grab first id of container
run the following code

  const string id = "25b933366a88";
  var client = new DockerClientConfiguration(new Uri("unix:///var/run/docker.sock")).CreateClient();
  await client.Containers.StartContainerExecAsync(id, default(CancellationToken));

What actually happened?:
Got exception:
Docker.DotNet.DockerContainerNotFoundException: Docker API responded with status code=NotFound, response={"message":"No such exec instance '25b933366a88' found in daemon"}

   at Docker.DotNet.ContainerOperations.<>c.<.cctor>b__29_0(HttpStatusCode statusCode, String responseBody)
   at Docker.DotNet.DockerClient.HandleIfErrorResponse(HttpStatusCode statusCode, String responseBody, IEnumerable`1 handlers)
   at Docker.DotNet.DockerClient.<MakeRequestAsync>d__33.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Aperture.Controllers.HomeController.<Index>d__5.MoveNext() in /home/maqe/Projects/aperture/src/Controllers/HomeController.cs:line 47
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeActionMethodAsync>d__27.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextActionFilterAsync>d__25.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextResourceFilter>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ResourceExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.<Invoke>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.<Invoke>d__7.MoveNext()

What did you expect to happen?:
I expected the container to be started.
Additional information:

This is expected.  The method you are calling is for starting an exec instance that has already been created, so it expects an exec ID not a container ID (which is why the error from the daemon is "No such exec instance").  If your intent is to start a process inside of a container, you need to start with ExecCreateContainerAsync on a running container.  If you are just trying to start a container, the method you are looking for is StartContainerAsync.  For reference, see the docker engine remote API documented here: https://docs.docker.com/engine/api/v1.24/.  Since Docker.DotNet is designed to wrap this engine API, the endpoints and objects match this documentation more than what folks are used to from the docker CLI.

@swernli Ok. I'm trying to run do a docker run with some additional parameters.
If I understand correctly from the documentation I need to create the container, start the container and then attach to the container. I also need to add some additional parameters to the run command. I think this is done by passing CreateContainerParameters(config) to the CreateContainerAsync method like this:
var parameters = new Config {
  Image = "d5db001b2b8a",
  ArgsEscaped = true,
  AttachStderr = true,
  AttachStdin = true,
  AttachStdout = true,
  Cmd = new string[] { " -i http://foobar.com -c:a copy -f mjpeg -preset ultrafast -an -c:v mjpeg -vf \"select='not(mod(n\\,29))',yadif,scale=480:-1,crop=iw:270,tile=1x50\" -q:v 20 -vframes 1 pipe:1" }
};

CreateContainerResponse response = await client.Containers.CreateContainerAsync(new CreateContainerParameters(parameters));

When the container is created I start and attach to it like this:
Task task = client.Containers.StartContainerAsync(response.ID, new ContainerStartParameters());

var config = new ContainerAttachParameters {
  Stream = true,
  Stderr = true,
  Stdin = false,
  Stdout = true,
  Logs = "1"
};

var buffer = new byte[1024];
using (var stream = await client.Containers.AttachContainerAsync(response.ID, false, config, default(CancellationToken)))
{
    var result = await stream.ReadOutputAsync(buffer, 0, buffer.Length, default(CancellationToken));
    do
    {
        Console.Write(Encoding.UTF8.GetString(buffer, 0, result.Count));
    }
    while (!result.EOF);
}

The container seems to start by the Cmd does not run, probably because it's not the correct way of adding additional parameters to the run command?
The complete command I'm trying to run using the remote API is this: docker run jrottenberg/ffmpeg -i http://foobar.com -c:a copy -f mjpeg -preset ultrafast -an -c:v mjpeg -vf "select='not(mod(n\\,29))',yadif,scale=480:-1,crop=iw:270,tile=1x50" -q:v 20 -vframes 1 - > scrub.jpg
Have I missed something obvious?


Support disabling default setup of shim logger.

Before this change, the v2 runtime shim setup code was hardcoded to always
configure logrus to write logs to the "log" FIFO present in the current working
directory. This only happens in the "default" action codepath
(i.e. not shim start or shim delete).
This is problematic for shims that execute outside the current working
directory of a bundle. For example, it often doesn't make sense for shims that
manage multiple containers to execute in a single bundle directory. Additionally,
shim processes that require being pre-created, i.e. spun up before tasks they
will handle are actually created, won't have a log FIFO to write to until a task
is created.
This change leaves the default behavior as is but introduces a Binary Config
field that will optionally disable automatic configuration of logrus to use the
"log" FIFO. This allows shims to configure their own logger if necessary while
still re-using the rest of the shim helper code in containerd.
Signed-off-by: Erik Sipsma<EMAIL_ADDRESS>

Codecov Report

Merging #3239 into master will decrease coverage by 4.76%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master    #3239      +/-   ##
==========================================
- Coverage   44.63%   39.86%   -4.77%
==========================================
  Files         113       76      -37
  Lines       12161    10104    -2057
==========================================
- Hits         5428     4028    -1400
+ Misses       5898     5510     -388
+ Partials      835      566     -269

Flag
Coverage Δ

#linux
?

#windows
39.86% <0%> (-0.01%)
:arrow_down:

Impacted Files
Coverage Δ

runtime/v2/shim/shim.go
0% <0%> (ø)
:arrow_up:

snapshots/native/native.go
1.79% <0%> (-41.26%)
:arrow_down:

archive/tar.go
16.99% <0%> (-26.8%)
:arrow_down:

metadata/snapshot.go
21.53% <0%> (-24.28%)
:arrow_down:

cio/io.go
1.52% <0%> (-21.38%)
:arrow_down:

content/local/writer.go
56.86% <0%> (-0.99%)
:arrow_down:

gc/scheduler/scheduler.go
66.34% <0%> (-0.97%)
:arrow_down:

oci/spec_opts.go
30.33% <0%> (-0.25%)
:arrow_down:

mount/temp_unix.go

sys/reaper_linux.go

... and 35 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2d780a7...48f4651. Read the comment docs.

LGTM


import package issue

The generated code is still referring to casualjim repository:
  "github.com/casualjim/go-swagger"
   "github.com/casualjim/go-swagger/middleware"
   "github.com/casualjim/go-swagger/spec"
   "github.com/casualjim/go-swagger/strfmt"

Moreover, the middleware package is actually under github.com/casualjim/go-swagger/middleware/httpkit/middleware.
In some other imports there is a reference to validation package
"github.com/casualjim/go-swagger/validation"

I guess that it should refer to validate package?
Could you please point me to the right place in the code so I can update it. I would be happy to contribute and help.

I thnk they moved here: https://github.com/go-swagger/go-swagger/tree/master/httpkit/validate

you could also try to pull in the templates for this from the gen-client branch. i fixed those things on there and the templates for the server shouldn't have changed.

Having the same issues even by modifying the templates. Could it be that the generator is referring to external templates? Because even when I removed the templates completely, the server was generated with the wrong package paths.

try running go generate ./generator  that should update the embedded templates.
Sorry don't have any docs about this stuff yet

Thanks, this was helpful!

@casualjim ^ did that for you : )

Tagged operations still have casualjim in there tho. I am not sure where that came from.


about collapseWhitespace

when i use pug-loader load .jade   get result without collapseWhitespace, but i need result with collapseWhitespace, help?

i found the arg     haha
function Compiler(node, options) {
  this.options = options = options || {};
  this.node = node;
  this.bufferedConcatenationCount = 0;
  this.hasCompiledDoctype = false;
  this.hasCompiledTag = false;
  this.pp = true;
  if (this.pp && typeof this.pp !== 'string') {
    this.pp = '    ';
  }
  this.debug = false !== options.compileDebug;
  this.indents = 0;
  this.parentIndents = 0;
  this.terse = false;
  this.mixins = {};
  this.dynamicMixins = false;
  this.eachCount = 0;
  if (options.doctype) this.setDoctype(options.doctype);
  this.runtimeFunctionsUsed = [];
  this.inlineRuntimeFunctions = options.inlineRuntimeFunctions || false;
  if (this.debug && this.inlineRuntimeFunctions) {
    this.runtimeFunctionsUsed.push('rethrow');
  }
};

this.pp

oh my good  the documentation there is nothing about pretty

What is collapseWhitespace? IIRC it is something from EJS, not Pug...
Also

They are all mapped directly to Pug options, unless pointed out otherwise.


Add compatibility to electron remote

This PR add ability to use oauth in a renderer process (web page).
https://github.com/electron/electron/blob/master/docs/api/remote.md
Is usefull with multiples oauth services.

I can confirm this works. Would be great to get it merged.

Sorry this one slipped under my radar 😞
Just published the PR as 2.4.1


bump doh-go

Codecov Report

Merging #57 (b6c78c1) into master (50473f0) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #57   +/-   ##
=======================================
  Coverage   42.01%   42.01%
=======================================
  Files           5        5
  Lines         557      557
=======================================
  Hits          234      234
  Misses        302      302
  Partials       21       21

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 50473f0...b6c78c1. Read the comment docs.


can't reproduce the same result between sc.tl.tsne and scikit-learn.TSNE

from the doc of sc.tl.tsne, I knew it was implemented by scikit, but I can't get the same result between sc.tl.tsne and scikit.TSNE，my test code is below
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""

from sklearn import datasets
import scanpy as sc
import numpy as np
import random
import matplotlib.pyplot as plt
from sklearn.manifold import TSNE
np.random.seed(1)
random.seed(1)

iris = datasets.load_iris()
X = iris.data
label=iris.target

adata=sc.AnnData(X)
adata.obs["celltype"]=label.astype(int).astype(str)
sc.tl.tsne(adata,random_state=0,use_fast_tsne=False)
axis=sc.pl.tsne(adata,color=["celltype"],size=100,show=False)
sc_tsne=adata.obsm["X_tsne"]
#print(ax)

print(np.min(sc_tsne[:,0]))
print(np.max(sc_tsne[:,0]))
print(np.min(sc_tsne[:,1]))
print(np.max(sc_tsne[:,1]))
print("====================")

# import pickle
# #with open()
# file = open('/Users/xiaokang/Desktop/data/tsne.pkl', 'rb')
# tsne2=pickle.load(file)

target=label
tsne = TSNE(learning_rate=1000,init='random', random_state=0)
X_transformed = tsne.fit_transform(X)
fig=plt.figure()
for label in np.unique(target):
    plt.scatter(X_transformed[label==target,0], X_transformed[label==target,1],label=label)
plt.legend(loc="upper left")
plt.show()
#print(X_transformed)
print(np.min(X_transformed[:,0]))
print(np.max(X_transformed[:,0]))
print(np.min(X_transformed[:,1]))
print(np.max(X_transformed[:,1]))

print("==================")
params_sklearn = dict(
    perplexity=30,
    random_state=0,
    verbose=False,
    early_exaggeration=12,
    learning_rate=1000,
)
from sklearn.manifold import TSNE
# unfortunately, sklearn does not allow to set a minimum number
# of iterations for barnes-hut tSNE
tsne3 = TSNE(**params_sklearn)
X_transformed=tsne3.fit_transform(X)
print(np.min(X_transformed[:,0]))
print(np.max(X_transformed[:,0]))
print(np.min(X_transformed[:,1]))
print(np.max(X_transformed[:,1]))

I get the result

I can't figure it out why this happen. Thank you for your help

Hello, could you write what version of sklearn you have installed?

Sorry,I forget to mention it , this is my sessioninfo:

Hey, I found the issue. AnnData automatically casts your input data from a float64 to a float32 format
If you generate it using:
adata = sc.AnnData(X, dtype='float64')
your results will be reproducible


Investigate a proper boilerplate code generator

The compiler code currently has two major uses of "code generation" in the sense of "code to generate part of the compiler source code":

The shader "standard library" is built at runtime out of a combination of static strings and C++ logic to generate repetitious boilerplate (e.g., all the various Texture* types and their members)

Various compiler data structures - most notably the AST nodes themselves - are defined in the various *-defs.h files via macros, that then get expanded in many different ways to generate declarations and definitions for processing them (e.g., the entire "visitor" setup")

Both of these work well enough for now, but they aren't easy to understand, and don't produce great error messages when things go wrong.
Now that we can package up releases via CI, it might be okay to accept a bit of extra build complexity by using/building a slightly more "real" metaprogramming tool to generate all this. My ideal would be something like a "template engine" where the files can look mostly like the target language (instead of the current macro mess).
Maybe build something small and simple in Lua?
A key point here is that the deployed "source release" can include the generated code, so that builds for users can stay just as simple as they are today.

There has been work done on this front, but the situation is still very messy. Instead of writing the standard library code as C++ that builds strings of Slang, we now write the code in *.meta.slang files that are processed by the tool slang-generate to produce .h files with... C++ code that builds strings of Slang. Still, this is a reasonable improvement for cases where the stdlib code has large blocks of raw Slang.
The big issue here is that actually parsing all those generated strings is getting to be heavy-weight, and so we will eventually want a solution where the Slang code is generated and compiled offline, and then embedded into the library. Using C++ as the language for the generation part is maybe overkill in this case.
I'm considering switching to Lua as a generator language, simply because we can include the source code under externals/ and wrap it up in our build without too much fuss, but I wouldn't consider that a slam dunk either.

At this point we have many different code synthesis strategies all vying for their place. There is no need to track this as a distinct gap.
Closing.


datastreams: add subjects

:heart: Thank you for your contribution!
Description
Adds the subject datastreams to match https://github.com/inveniosoftware/invenio-vocabularies/pull/355.
Checklist
Ticks in all boxes and 🟢 on all GitHub actions status checks are required to merge:

[x] I'm aware of the code of conduct.
[x] I've created logical separate commits and followed the commit message format.
[x] I've added relevant test cases.
[x] I've added relevant documentation.
[x] I've marked translation strings.
[x] I've identified the copyright holder(s) and updated copyright headers for touched files (>15 lines contributions).
[x] I've NOT included third-party code (copy/pasted source code or new dependencies).

If you have added third-party code (copy/pasted or new dependencies), please reach out to an architect.

Reminder
By using GitHub, you have already agreed to the GitHub’s Terms of Service including that:

You license your contribution under the same terms as the current repository’s license.
You agree that you have the right to license your contribution under the current repository’s license.

This PR has a dependency on https://github.com/inveniosoftware/invenio-vocabularies/pull/355 which has to be merged first.

This is replaced by https://github.com/inveniosoftware/invenio-app-rdm/pull/2790


Connect timeout does not always work

Expected behavior
We configure our Bootstrap with ChannelOption.CONNECT_TIMEOUT_MILLIS to set the connect timeout to 2 seconds. We expect, that after 2 seconds the connect attempt will be canceled, e.g. due to a firewall blocking the connection attempt.
Actual behavior
In most cases the timeout works as expected, however in some environments it does not work. After running our application for several hours in one of our clients environment, there were delays of up to 140 minutes before the timeout was handled.
Steps to reproduce
I narrowed the problem down to the point where tasks are scheduled on an event loop group. We use an NioEventloopGroup with default size (i.e. double number of CPUs, in our case 12 threads). As connect attempts and scheduled timeouts use the same event loop group, there is a chance that the timeout task is scheduled to the same event loop as the connect task. In case that the connect task blocks, the timeout task is not executed as expected.
It is merely impossible to reproduce this behavior in an network environment, but I was able to setup  a small example using NioEventLoopGroup with two threads that simulates the problem.
It just starts a long running task and afterwards schedules 10 tasks to be executed after 2 seconds.
The blocking thread starts immediately on one event loop and the scheduled threads are distributed in a round robin manner on the two event loops. This leads to 5 tasks being executed after 2 seconds and the other 5 tasks to be executed after the long running task has terminated.
This example forces the behavior that we determined in our customers logs. We do not know exactly what causes one thread to block and to prevent connect timeout tasks to be executed. It might be - as I assume - a blocking connection attempt.  But it might be as well some other code of our own.
Anyways, the result is that the connect timeout task is not executed as it should be.
Therefore the question is, shouldn't there be a separate EventLoopGroup for the connection timeouts so that these are guaranteed to be executed?
Minimal yet complete reproducer code (or URL to code)
package test;

import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;

import io.netty.channel.EventLoopGroup;
import io.netty.channel.nio.NioEventLoopGroup;

public class NioTest {

    private static int SCHEDULED_TASK_COUNT = 10;

    private static long start;

    private static long diff() {

        return (System.nanoTime() - start) / 1000_000;
    }

    public static void main(String[] args) throws InterruptedException {

        // use a small event loop group to force the unexpected behavior
        EventLoopGroup loopGroup = new NioEventLoopGroup(2);

        CountDownLatch cdl = new CountDownLatch(SCHEDULED_TASK_COUNT + 1);

        start = System.nanoTime();
        System.out.println("start: " + diff());

        // submit a task, that is executed immediately
        loopGroup.submit(() -> {

            System.out.println(String.format("blocking task started    at %6d on thread %6d. Count down %d", diff(), Thread.currentThread().getId(), cdl.getCount()));
            System.out.println("waiting for 10 seconds ...");
            try {
                // wait for 5 seconds to simulate a blocking operation
                Thread.sleep(5000);
            } catch (InterruptedException e) {
            }
            System.out.println(String.format("blocking task finished  at %6d. Count down %d", diff(), cdl.getCount()));
            cdl.countDown();
        });
        System.out.println(String.format("blocking task submitted  at %6d. Count down %d", diff(), cdl.getCount()));

        for (int i = 0; i < SCHEDULED_TASK_COUNT; i++) {

            final int fi = i; // necessary to be accessible in closure ...

            // schedule tasks to be executed in 2 seconds
            loopGroup.schedule(() -> {
                String message = String.format("task %3d      started   at %6d on thread %6d. Count down %d",
                    fi, diff(), Thread.currentThread().getId(), cdl.getCount());

                System.out.println(message);
                cdl.countDown();
            }, 2, TimeUnit.SECONDS);

            System.out.println(String.format("task %3d      submitted at %6d. Count down %d", i, diff(), cdl.getCount()));
        }

        cdl.await();
        System.out.println(String.format("All tasks completed at %6d", diff()));
        loopGroup.shutdownGracefully();
    }
}

The result is
start: 0
blocking task submitted  at      8. Count down 11
blocking task started  at      8 on thread     13. Count down 11
waiting for 10 seconds ...
task   0      submitted at     12. Count down 11
task   1      submitted at     12. Count down 11
task   2      submitted at     12. Count down 11
task   3      submitted at     12. Count down 11
task   4      submitted at     13. Count down 11
task   5      submitted at     13. Count down 11
task   6      submitted at     13. Count down 11
task   7      submitted at     13. Count down 11
task   8      submitted at     13. Count down 11
task   9      submitted at     13. Count down 11
task   0      started   at   2014 on thread     14. Count down 11
task   2      started   at   2015 on thread     14. Count down 10
task   4      started   at   2015 on thread     14. Count down 9
task   6      started   at   2016 on thread     14. Count down 8
task   8      started   at   2016 on thread     14. Count down 7
blocking task finished  at   5010. Count down 6
task   1      started   at   5010 on thread     13. Count down 5
task   3      started   at   5010 on thread     13. Count down 4
task   5      started   at   5010 on thread     13. Count down 3
task   7      started   at   5011 on thread     13. Count down 2
task   9      started   at   5011 on thread     13. Count down 1
All tasks completed at   5011

Netty version
4.1.23
JVM version (e.g. java -version)
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
OS version (e.g. uname -a)
Linux myhost 4.4.180-102-default #1 SMP Mon Jun 17 13:11:23 UTC 2019 (7cfa20a) x86_64 x86_64 x86_64 GNU/Linux

@A-Willems this is completely expected... if you block the EventLoop all bets are off. You MUST never block the EventLoop.

All connect attempts done by Netty are non-blocking.

Thank you for the reply. I think I found the weak spot in our code.


Thirty-Fifth Community Call

Next call will be on Tue Mar 15th at 6pm CEST (1700 UTC, 12pm EST, 9am PST). We will aim to include more projects from America this time.
Agenda:

wasmd 0.24
CosmWasm academy
CosmWasm doc working group
Open Q&A

Zoom link: https://zoom.us/j/361222561?pwd=ZmpIQUppYks2MDVtSjRGQVpTaVlKUT09

thanks for joining and watching the replay video https://vimeo.com/688545834


fix(xp-management): Make XP Management chart use service account selectively

Motivation
Similar to one of the changes in PR #334, a tiny fix is introduced to the XP Management chart such that the service account name is only injected into the XP Management Service deployment only when we are expecting one to be created or if its name is specified explicitly.
Modification

charts/xp-management/templates/deployment.yaml - Addition of logical check before the service account name is added
charts/xp-management/templates/_helpers.tpl - Simplification of the definition for serviceAccountName (there is no way the "default" name will ever be used, since serviceAccountName is never used unless .Values.deployment.serviceAccount.create or .Values.deployment.serviceAccount.name is true; see the fix above)

Checklist

[x] Chart version bumped
[x] README.md updated

Thanks for the quick review! Merging this now! :D


Loose versions

@mambocab to review. Depends on https://github.com/pcmanus/ccm/pull/548
CI at http://cassci.datastax.com/view/Dev/view/ptnapoleon/job/ptnapoleon-trunk-dtest/18/testReport/
http://cassci.datastax.com/view/Dev/view/ptnapoleon/job/ptnapoleon-trunk-upgrade-upgrade/1/

Code LGTM. Optional nit: LooseVersion knows how to compare itself to strings:
In [2]: from distutils.version import LooseVersion

In [3]: LooseVersion('3.10') > '3.9'
Out[3]: True

In [4]: LooseVersion('3.10') > '3.11'
Out[4]: False

In [5]: LooseVersion('3.10') > '3.1'
Out[5]: True

So if you want you can change relevant lines to just compare against strings. Up to you.
Feel free to merge if CI looks good after fixing the unused imports Travis found.

Re-testing here: http://cassci.datastax.com/view/Dev/view/ptnapoleon/job/ptnapoleon-test-ccm-dtest/2/


No action when nodes crash during mission

When I kill the mission_control_node or waypoint_node or qr_code_scanner_node, the fcc bridge doesn't notice that. Therefore, also no RTH gets triggered. The expected behaviour would be to monitor the heartbeat of these nodes and trigger RTH if one fails.

Crafting truly groundbreaking safety features, like the one we discussed, requires a significant investment of time and resources. @J0las fully recognizes this, and we share that commitment.  As he develops this and other crucial safety improvements, we kindly ask for your patience. Your understanding is highly valued.

implemented by @J0las a while ago


JSDoc / Documentation

The code isn't even commented yet.... would like to setup JSDoc

Sure, let me see if I can help. Two questions:

You want the JSDoc comments in this file? https://github.com/accraze/machine-translator/tree/master/src
Do you want basic descriptions or the full documentation tags?

Hey @hillaryfraley,

Yes comments in that file would be great.
Just basic descriptions are fine for now

OK...I was able to do a little. I'm not that great with JavaScript, so I had some trouble figuring out what different parts did. I basically used the comments you already had in there, so there's still a lot missing.

This looks great for now @hillaryfraley! Thanks!
closed by #98


parametise the licensing config file to accept database authentication

Context
In AWS, the licensify apps use documentDB which compulsory requires a username and password.
Deci sions

add the option to configure the app to use a specified username and password for the database if the database authentication is enabled in the config.

Data part of PR: https://github.com/alphagov/govuk-secrets/pull/780

Looks good but I think we should leave the database hostnames in the clear, since this is the current pattern elsewhere in the codebase and it makes it easier for us to see what's going on.

Looks good but I think we should leave the database hostnames in the clear, since this is the current pattern elsewhere in the codebase and it makes it easier for us to see what's going on.

I talked with Sebastian who advised to put into secrets

Looks good but I think we should leave the database hostnames in the clear, since this is the current pattern elsewhere in the codebase and it makes it easier for us to see what's going on.

I talked with Sebastian who advised to put into secrets

Ok no probs :)


Storage spaces for DB files

This would be great to include or explicitly  define if storage spaces or NVME configuration for primary sites.  This  is still is based on old RAID configurations from my understanding

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: e67ddd45-90df-32cd-3cbe-9e1443b2540f
Version Independent ID: dee7fcb8-a7b7-e5a1-9c04-c113805be0da
Content: Recommended hardware - Configuration Manager
Content Source: sccm/core/plan-design/configs/recommended-hardware.md
Product: configuration-manager
GitHub Login: @aczechowski
Microsoft Alias: aaroncz

Thanks @rdoloto for the feedback. You can find more information on the topic in the perf and scale whitepaper. In the future I hope to incorporate that content directly into the core docs, but leverage that resource for now.

Thanks again for the feedback. We recently moved this content to a new GitHub repository, so this issue is no longer tracked with the original article. I am tracking the work on the perf/scale whitepaper in an internal item, so I'll link to this for future reference. But am going to close this issue.


Setup lab through a background job

It would be nice to have an easy way to setup a lab in a background job. Will need to make sure there are no prompts for things like installing nuget.

Added this feature to Unattend-Lab in v4.3.0


Keras to tflite. Multiple outputs all given name Identity

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below):2.0-rc0
Python version: 3.6
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version: 10.0/7.6
GPU model and memory: RTX 2070

Describe the current behavior
When I convert a keras model to a tflite model all outputs are named Identity. When you have a bunch of outputs that all have the same output size it becomes impossible to tell which one is which.
Describe the expected behavior
I would expect the names to match the layer names that the output comes from.
Code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate the problem.
import tensorflow as tf

inputs = []
outputs = []

x = tf.keras.layers.Input(shape=(10), batch_size=1, name='input1')
inputs.append(x)
x = tf.keras.layers.Dense(10, name='dense1')(x)
outputs.append(x)
x = tf.keras.layers.Dense(10, name='dense2')(x)
outputs.append(x)

model = tf.keras.Model(inputs=inputs, outputs=outputs)

model.summary()

Model: "model_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input1 (InputLayer)          [(1, 10)]                 0
_________________________________________________________________
dense1 (Dense)               (1, 10)                   110
_________________________________________________________________
dense2 (Dense)               (1, 10)                   110
=================================================================
Total params: 220
Trainable params: 220
Non-trainable params: 0

model.outputs

[<tf.Tensor 'dense1/Identity:0' shape=(1, 10) dtype=float32>,
 <tf.Tensor 'dense2/Identity:0' shape=(1, 10) dtype=float32>]

# Convert and save the model.
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()
open("./save/simple.tflite", "wb").write(tflite_model)

# Load TFLite model and allocate tensors.
interpreter = tf.lite.Interpreter(model_path="./save/simple.tflite")
interpreter.allocate_tensors()

# Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

output_details

[{'name': 'Identity',
  'index': 0,
  'shape': array([ 1, 10], dtype=int32),
  'dtype': numpy.float32,
  'quantization': (0.0, 0)},
 {'name': 'Identity_1',
  'index': 1,
  'shape': array([ 1, 10], dtype=int32),
  'dtype': numpy.float32,
  'quantization': (0.0, 0)}]

Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

I would like to work on this issue!  Please assign

Thumbs up! Right now it is impossible to deploy models to tflite if they have multiple outputs. Is anybody working on this?

@Cospel One work around I found was using https://lutzroeder.github.io/netron/ to view the tflite model to visually figure out what index each output corresponded to. This issue should still be fixed but this work around should help people that are currently stuck on this problem.

@marno1d Thank you for the idea. However this is not very friendly and scalable solution :).
I hoped that creating tf.keras.layers.Lambda with the name would work:
output1 = tf.keras.layers.Lambda(lambda k: k, name='output1')(x)

but it's just creates another Identity....

@Cospel I agree that it isn't a scalable solution. I think all outputs are automatically wrapped in an Identity layer according to the source code. I'm not familiar enough with the code to know why this is there / how to fix it.
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/keras/engine/base_layer.py#L774
# Wrap Tensors in "outputs" in "tf.identity" to avoid
 # circular dependencies.
 outputs = base_layer_utils.mark_as_return(outputs, acd)

@marno1d yes, making a change like this fixes the issue (and probably creates another issues 😄)
https://github.com/tensorflow/tensorflow/blob/v2.1.0-rc2/tensorflow/python/framework/auto_control_deps.py#L167
-tensor = array_ops.identity(tensor)
+# convert "model/tf_op_layer_output/output:0" -> "output"
+tensor = array_ops.identity(tensor, tensor.name.split("/")[-1].split(":")[0])

Is anybody working on this ? This problem still appears in the newer version of TF.

Hi, I have found (temporal) solution for this problem. It seems that the invalid output names are generated within this function:
tensorflow_core.python.framework.func_graph.func_graph_from_py_func

It has a default argument add_control_dependencies set to True. Switching it to False before model conversion leads to correct names when combined with keras.Lambda and tf.identity e.g:
output = Lambda(lambda x: tf.identity(x, name="output"), name="logits")(outputs)
keras.Model(input_image, output, name="export")

gives me following output details:
Output details:
{ 'dtype': <class 'numpy.float32'>,
  'index': 192,
  'name': 'export/logits/output',
  'quantization': (0.0, 0),
  'shape': array([  1, 4], dtype=int32)}

My shitfix for this is to use Monkey patching and create tflite_converter_shitfix.py file which will set add_control_dependencies to False at runtime just before model conversion:
import weakref
import tensorflow as tf
from tensorflow.python.eager import def_function as _def_function
from tensorflow.python.eager import function as _function
from tensorflow.python.framework import func_graph as func_graph_module
from tensorflow.python.framework import ops
from tensorflow.python.keras.saving import saving_utils as _saving_utils
from tensorflow.python.util import tf_decorator

keras = tf.keras

def from_keras_model(model: keras.Model) -> tf.lite.TFLiteConverter:
    """Creates a TFLiteConverter object from a Keras model.

    Args:
      model: tf.Keras.Model

    Returns:
      TFLiteConverter object.
    """
    input_signature = None
    # If the model's call is not a `tf.function`, then we need to first get its
    # input signature from `model_input_signature` method. We can't directly
    # call `trace_model_call` because otherwise the batch dimension is set
    # to None.
    # Once we have better support for dynamic shapes, we can remove this.
    if not isinstance(model.call, _def_function.Function):
        # Pass `keep_original_batch_size=True` will ensure that we get an input
        # signature including the batch dimension specified by the user.
        input_signature = _saving_utils.model_input_signature(
            model, keep_original_batch_size=True
        )

    func = _saving_utils.trace_model_call(model, input_signature)
    func._defun_with_scope = lambda s: _defun_with_scope(func, s)
    concrete_func = func.get_concrete_function()
    return tf.lite.TFLiteConverter([concrete_func])

def _defun_with_scope(self, scope):
    """Creates a defun wrapped inside a variable creator scope."""

    weak_wrapped_fn = None

    def wrapped_fn(*args, **kwds):
        """Wraps `self._python_function` in a variable creator scope."""
        with ops.get_default_graph()._variable_creator_scope(scope, priority=50):
            return weak_wrapped_fn().__wrapped__(*args, **kwds)

    weak_wrapped_fn = weakref.ref(wrapped_fn)

    fun = self._defun(tf_decorator.make_decorator(self._python_function, wrapped_fn))

    fun._create_graph_function = lambda args, kwargs: _create_graph_function(
        fun, args, kwargs
    )
    return fun

def _create_graph_function(self, args, kwargs, override_flat_arg_shapes=None):
    """Create a `ConcreteFunction` from `args` and `kwargs`."""

    self.tracing_count += 1
    if self.input_signature is None:
        arglen = len(args)
    else:
        arglen = len(self.input_signature)
    base_arg_names = self._function_spec.arg_names[:arglen]
    num_missing_args = arglen - len(self._function_spec.arg_names)
    missing_arg_names = [self._function_spec.vararg_name] * num_missing_args
    # Produce a list of missing args of the form ["arg_0", "arg_1", ...],
    # where arg is based on the self._function_spec.vararg_name.
    missing_arg_names = ["%s_%d" % (arg, i) for i, arg in enumerate(missing_arg_names)]
    arg_names = base_arg_names + missing_arg_names

    graph_function = _function.ConcreteFunction(
        func_graph_module.func_graph_from_py_func(
            self._name,
            self._python_function,
            args,
            kwargs,
            self.input_signature,
            autograph=self._autograph,
            autograph_options=self._autograph_options,
            arg_names=arg_names,
            override_flat_arg_shapes=override_flat_arg_shapes,
            capture_by_value=self._capture_by_value,
            add_control_dependencies=False,
        ),
        self._function_attributes,
        # Tell the ConcreteFunction to clean up its graph once it goes out of
        # scope. This is not the default behavior since it gets used in some
        # places (like Keras) where the FuncGraph lives longer than the
        # ConcreteFunction.
        shared_func_graph=False,
    )
    return graph_function

And then use:

from tflite_converter_shitfix import from_keras_model
converter = from_keras_model(model)
converter.experimental_new_converter = True
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.convert()

Hi, I've run into this today and figured out that I can still use the old converter code with disabled eager execution:
tf.compat.v1.disable_eager_execution()

converter = tf.compat.v1.lite.TFLiteConverter.from_keras_model_file(path_model)

return converter.convert()

this will fix it for my use case. I hope it helps others too.

The problem persists in TF2.3 nightly as well.

I also encountered this problem。

This also happens on tensorrt conversion.
I tracked down the cause to 'canonicalize_signatures' method in signature_serialization.py
Seems like only the inputs are being saved in the signature and the outputs are being converted to "Identity" named tensors.

I run into the problem too。is it solved？ tflite for multiple output

Same issue here, All outputs ordered after their defined names but shows as Identity

I got this also...  all my outpus were re-ordered

@caegomezji
SignatureDef provides meaningful/generic names for inputs/outputs which doesn't rely on specific model details. More on SignatureDefs here.
If your saved model has a defined signatureDef then it will be exported during conversion to TFLite and then you can use the Signature inputs/outputs for inference and not relying on inputs/outputs order or tensor names.
SignatureDef support is new thing available in nightly and will be available in 2.5
Then use the nightly for converting to tflite. The generated tflite file will have the SignatureDef details.
Using Python API (as an example) you can do something like
my_signature = interpreter.get_signature_runner("my_method")
results = my_signature(input_1=input_tensor_1, input_2=input_tensor_2)
print(results["my_output"])

Any update on this issue?
It is possible to rename a TF model's outputs for using coremltools but not `tensorflow?

Looking forward to updates on this issue! For extreme use cases, every byte counts and would be great if we had the option to remove some of the unnecessary metadata and tensor names from the exported C array.

@kdonbekci @maxpv
Please see https://github.com/tensorflow/tensorflow/issues/32180#issuecomment-772140542 for the solution.

@abattery
If I'm not wrong, the issue is not solved at C++ level ?
Is there any way to solve it with tflite C++ API ?
Thanks

@n-moussa There is a C++ API for using Signatures

To get list of signatures available
interpreter.signature_def_names();
To get inputs/output of a single Signature:
TfLiteTensor* input_1_tensor = interpreter.input_tensor_by_signature_name("input1", "mySignature");
TfLiteTensor* input_2_tensor = interpreter.input_tensor_by_signature_name("input2", "mySignature");
const TfLiteTensor* output_tensor = interpreter.output_tensor_by_signature_name("output", "mySignature");

Example,
interpreter.AllocateTensors()
autp* input1 = interpreter.input_tensor_by_signature_name("input1", "mySignature");
// Set the data in the tensor normally e,g,: input1->data.f[0] = 1.0f;
autp* input2 = interpreter.input_tensor_by_signature_name("input2", "mySignature");
// Set the data in the tensor normally e,g,: input1->data.f[0] = 1.0f;
// Run
interpreter.Invoke();
// Fetch outputs
const TfLiteTensor* output_tensor = interpreter.output_tensor_by_signature_name("output", "mySignature");

Maybe i'm missing something obvious here but I don't understand how this was fixed? I still have the same problem with nightly 2.5.0-dev20210307
my signature def is intact in the saved model. and lists my outputs correctly
['serving_default']
{'myoutputname1': TensorSpec(shape=(None, 8), dtype=tf.float32, name='myoutputname1'), 'myoutputname2': TensorSpec(shape=(None, 1), dtype=tf.float32, name='myoutputname2'), 'myoutputname3': TensorSpec(shape=(None, 2), dtype=tf.float32, name='myoutputname3')}
then during conversion the serving_default is ignored and i get generic names:
name: serving_default_input_1:0
shape: [ 1 80 80  1]
type: <class 'numpy.float32'>
quantization_parameters: (0.0, 0)
0= 0.0
255= 0.0
== Output details ==
name: StatefulPartitionedCall:2
shape: [1 8]
type: <class 'numpy.float32'>
quantization_parameters: (0.0, 0)
0= 0.0
255= 0.0
== Output details ==
name: StatefulPartitionedCall:1
shape: [1 1]
type: <class 'numpy.float32'>
quantization_parameters: (0.0, 0)
0= 0.0
255= 0.0
== Output details ==
name: StatefulPartitionedCall:0
shape: [1 2]
type: <class 'numpy.float32'>
quantization_parameters: (0.0, 0)
0= 0.0
255= 0.0

Love the signature runner API! I noticed that when I convert a concrete_function instead of a saved_model to tflite using tf.lite.TFLiteConverter.from_concrete_functions() the signature def is missing from the model after I load it into the interpreter. Is this behaviour by design or am I missing something during the conversion (for example, some attributes that I need to add to the concrete function, so it persists in the signature def)?
Sorry for adding to a closed thread. I thought this was relevant to the discussion here.

@avroshk Thanks for your comment! Could you file another issue separately for the concrete function's signature def support? This can be a good one for the feature request. :-)

Currently the Signature is supported through saved model. As Jaesung said please file another issue for a feature request.
Thanks

Another this I realised is the weights and biases are getting modified in this procedure and graph structure is also changing when visualised through netron and the same changed structure is reflecting when checked via scripting to  cross validate the netron output. Conv2_block1_2 for example had no BN layer before Relu. But If want to pull out the Relu output then it adds BN layers at every node which is being pulled out

@manp-git I am not sure i understand what you refer to by "this procedure"
Please file separate issue with sample and reproduce steps if you're having an issue.
Thanks

this issue is partially back in the latest latest nightly and the official tf2.5.0
it works in nightly 2.5.0-dev20210307
When doing a straight conversion to tflite it works as expected, with input and output naming carried over as it should
But when quantizing a model through converter.optimizations = [tf.lite.Optimize.DEFAULT]
the signature is empty

@LIFEPANTS Thanks for the report. We are working on a fix for signature getting dropped. It is work in progress now.
Sorry for the trouble.

@MeghnaNatraj FYI

@karimnosseir
Was this ever fixed? any release i should try?

You can try out the nightly version. See also https://www.tensorflow.org/lite/guide/signatures

@caegomezji
SignatureDef provides meaningful/generic names for inputs/outputs which doesn't rely on specific model details. More on SignatureDefs here. If your saved model has a defined signatureDef then it will be exported during conversion to TFLite and then you can use the Signature inputs/outputs for inference and not relying on inputs/outputs order or tensor names.
SignatureDef support is new thing available in nightly and will be available in 2.5
Then use the nightly for converting to tflite. The generated tflite file will have the SignatureDef details. Using Python API (as an example) you can do something like
my_signature = interpreter.get_signature_runner("my_method")
results = my_signature(input_1=input_tensor_1, input_2=input_tensor_2)
print(results["my_output"])

you saved my life !!! so cool. thanks!!!


pypi install-ability

Hello,
Turns out there was more to be done. The tl;dr; is that we still need a shim setup.py, and a few other things changed.
When I tried to install pywfa for a package feature I'm testing, the github action that tests my package wasn't able to successfully install pywfa==0.4.2. I was getting an error that pywfa was named "UNKNOWN-0.0.0"

full error
WARNING: Generating metadata for package pywfa produced metadata for project name unknown. Fix your #egg=pywfa fragments.
Discarding https://files.pythonhosted.org/packages/5d/92/366a9f0a0a4b34e2650199861a377eddf7e9f24ecc5298efb81352dc7cee/pywfa-0.4.2.tar.gz#sha256=065c782aa917da52c947a7ea1ed9d0e0153325153f7fcc32cca588f08e58136b (from https://pypi.org/simple/pywfa/) (requires-python:>=3.7): Requested unknown from https://files.pythonhosted.org/packages/5d/92/366a9f0a0a4b34e2650199861a377eddf7e9f24ecc5298efb81352dc7cee/pywfa-0.4.2.tar.gz#sha256=065c782aa917da52c947a7ea1ed9d0e0153325153f7fcc32cca588f08e58136b has inconsistent name: filename has 'pywfa', but metadata has 'unknown'

I also needed to fix the MANIFEST.in to include the _custom_build.py and Makefile in pywfa/WFA2_lib/*. Also, the new _custom_build.py has a check to ensure that the make command works. However, the make command never exited non-zero due to only returning the status of the final command (was cd pywfa/WFA2_lib; make .. ; cd ../).
I believe these changes should work better because I was able to get python3 setup.py sdist to create a valid, usable python3 -m pip install dist/pywfa-0.5.0.dev1.tar.gz.  And by having the shim, things like pip install -e seem to work. Furthermore, I successfully tested by editing my github action to pull directly via: python3 -m pip install 'pywfa @ git+https://github.com/ACEnglish/pywfa'. Which is great, but I don't know if this is a great emulation of what pypi will make.
Finally, I bumped the version to 0.5.0-dev. At least once trying  to install 0.4.2 failed and pip slyly installed 0.4.1 instead (which had caused the old undefined symbol: wavefront_align). I'm guessing that bumping the minor version instead of the patch may prevent this.
I don't know your workflow for distributing to pypi, but if possible, could you try a test.pypi.org upload before pushing to regular pypi? If there's more problems, we'll at least not be filling it up with my mistakes lol.
Have a great day,
~/Adam English

Thanks again, it seems there are so many traps to fall in to. I tested this PR on testPypi and it seemed to work fine. Happy to merge and put on pypi

Looks like it works! Thanks for putting this out there.


Get rid of the panic stack trace for the favicon:

If you use a browser to download content from S3Proxy, it results in a
panic stack trace in the S3Proxy server logs. This is caused by Chrome
(possibly others) doing a subsequent request to get the favicon
(/favicon.ico).
Because of the catch-all nature of the regex, S3Proxy tries to lookup a
bucket with that name.
The simple fix is to 404 on the favicon URL

This looks good, go ahead and merge this in.


Move cross-compile scripts to scripts/es2k

Move make-cross-ovs.sh and config-cross-recipe.sh to the scripts/es2k directory. This helps declutter the root directory and makes it clear that these two scripts are specific to the ES2K target.

This PR will require corresponding change(s) to the MEV build scripts.

Yes Derek, These changes will break eisting ACC builds. This needs a change in ACC build scripts as well.

can this PR wait?
We have a cyclic dependency here.
ACC build as part of OD will fail when we update manifest with SHA which has changed script location.
If we go ahead and update ACC to use a script from new file, since manifest is not updated with latest SHA even that will fail.
We have a solution here, where we can use the same branch name for both manifest and ACC script. We might have issues when updating this in release branch.
Is it possible to have a transition,
we will have script in both locations, and once ACC build script is updated with new location then we can go ahead and remove the file from older directory


open3d>=0.16.0 conflict problem

Hi I was wondering if I can use instruct-nerf2nerf with only a open3d dependency of 0.13.0. I am unable to use open3d>=0.16.0 at the moment, and my most recent version is 0.13.0. I am able to download nerfstudio successfully by modifying the pyproject.toml, but it seems that I cannot download instruct-nerf2nerf...
The conflict is caused by:
    nerfstudio 0.3.2 depends on open3d>=0.16.0
    nerfstudio 0.3.1 depends on open3d>=0.16.0
    nerfstudio 0.3.0 depends on open3d>=0.16.0

This can be solved by installing nerfstudio first (with open3d=0.13.0) and then installing instruct-nerf2nerf.

@hjhyunjinkim I encountered the same issue. Could you please share in detail how you resolved it?


Irrecoverable errors when running AST parser in parallel

Summary
I'm working on some static analysis tools that leverage RustPython's AST parser. The basic setup is that I enumerate a bunch of Python files, then in parallel (via Rayon), read them from disk, parse them with RustPython, and perform various operations on the AST.
This generally works great! However, I've noticed that the parallelized parsing can lead to irrecoverable errors (panics, segmentation faults, etc.) when the source Python files themselves contain certain contents, especially nested f-strings.
Here's an example snippet that fails for me occasionally, maybe one in twenty times, typically with Trace/BPT trap: 5, though depending on how exactly I structure the code, I can also get a Segmentation fault: 11:
use std::fs;
use std::path::Path;

use rayon::prelude::*;
use rustpython_parser::ast::Suite;
use rustpython_parser::parser;

fn main() {
    [
        Path::new("resources/test/broken/__init__.py").to_path_buf(),
        Path::new("resources/test/broken/make_string.py").to_path_buf(),
    ]
    .par_iter()
    .map(|path| {
        let contents = fs::read_to_string(path).unwrap();
        parser::parse_program(&contents).unwrap()
    })
    .collect::<Vec<Suite>>();
}

...where __init__.py is empty and make_string.py looks like (sorry, weirdly specific example, but it's been difficult to pinpoint the exact issue):
where_statement = f"""
{' OR '.join([f'column_name = "{field}"' for field in ['plate', 'well']])}
"""

A few observations:

On the larger Python codebase I'm using for development, I got a stack overflow 100% of the time until I removed the nested f-strings from two specific files (now it works without error every time). If helpful, I can probably come up with an example that errors consistently, but it may be more involved.
If I remove Rayon, and do everything serially, the code never errors.
If I run Rayon over a single file (e.g., remove the __init__.py from the above snippet), the code never errors. (I don't know if this is due to a Rayon optimization to not spin out a thread in that case, or something else.)

Here's a self-contained example that fails ~75% of the time for me:
use rayon::prelude::*;
use rustpython_parser::ast::Suite;
use rustpython_parser::parser;

fn main() {
    let mut all_contents: Vec<&str> = vec![];
    for _ in 1..100 {
        all_contents.push("");
    }
    all_contents.push(
        "where_statement = f\"\"\"
{' OR '.join([f'column_name = \"{field}\"' for field in ['plate', 'well']])}
\"\"\"",
    );

    all_contents
        .par_iter()
        .map(|contents| parser::parse_program(*contents).unwrap())
        .collect::<Vec<Suite>>();
}

Example errors include Bus error: 10, Segmentation fault: 11, and sometimes something like:
thread '<unnamed>' panicked at 'index out of bounds: the len is 0 but the index is 8', /Users/crmarsh/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.3/src/sleep/mod.rs:357:28
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Rayon: detected unexpected panic; aborting
Abort trap: 6

To embed python code, you can do it like:
    all_contents.push(
        r#"where_statement = f"""
{' OR '.join([f'column_name = "{field}"' for field in ['plate', 'well']])}
""""#,
    );

reproducible with:
use rayon::prelude::*;
use rustpython_parser::{ast::Suite, parser};

fn main() {
    let mut all_contents: Vec<&str> = vec![];
    for _ in 1..10000 {
        all_contents.push("");
    }
    all_contents.push(
        r#"f"{f'{1}'}""#,
    );

    all_contents
        .par_iter()
        .map(|contents| parser::parse_program(*contents).unwrap())
        .collect::<Vec<Suite>>();
}

Thanks, that's a much better repro.

I tried if I could reproduce this with https://github.com/RustPython/RustPython/issues/4099#issuecomment-1219821507, but with the rustpython_parser version from crates.io I get
error[E0277]: the trait bound `Vec<Vec<Located<StatementType>>>: FromParallelIterator<rustpython_parser::ast::Program>` is not satisfied
    --> src/main.rs:16:10
     |
16   |         .collect::<Vec<Suite>>();
     |          ^^^^^^^ the trait `FromParallelIterator<rustpython_parser::ast::Program>` is not implemented for `Vec<Vec<Located<StatementType>>>`
     |
     = help: the trait `FromParallelIterator<T>` is implemented for `Vec<T>`
note: required by a bound in `rayon::iter::ParallelIterator::collect`
    --> /home/bjorn/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.3/src/iter/mod.rs:2044:12
     |
2044 |         C: FromParallelIterator<Self::Item>,
     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ required by this bound in `rayon::iter::ParallelIterator::collect`

For more information about this error, try `rustc --explain E0277`.
error: could not compile `rustpython_sigsegv` due to previous error

and with the one you are currently using for ruff, I'm getting
error[E0061]: this function takes 2 arguments but 1 argument was supplied
  --> src/main.rs:15:25
   |
15 |         .map(|contents| parser::parse_program(*contents).unwrap())
   |                         ^^^^^^^^^^^^^^^^^^^^^ --------- supplied 1 argument
   |                         |
   |                         expected 2 arguments
   |
note: function defined here
  --> /home/bjorn/.cargo/git/checkouts/rustpython-80c0a11a8fa8e175/1613f6c/compiler/parser/src/parser.rs:19:8
   |
19 | pub fn parse_program(source: &str, source_path: &str) -> Result<ast::Suite, ParseError> {
   |        ^^^^^^^^^^^^^

For more information about this error, try `rustc --explain E0061`.
error: could not compile `rustpython_sigsegv` due to previous error

if I specify an empty source_path, it runs just fine. Valgrind also reports no issues.

(I think you have to use the RustPython version from GitHub, the version on crates.io is quite old.)

That is what I did in the second case. Ruff uses a pinned git commit.

Oh sorry, yes, misread! I think that signature is newer than the issue. (Passing an empty string is fine, I believe it's just used for the error message.)

I tried passing an empty string, but it didn't crash. Nor did valgrind memcheck report anything.

Yeah I'm no longer seeing this crash either on my end.


SB width by CSS

Hi, is it possibile to set sb-width by the CSS?
Regards,
Antonio

Yup, just set the width of sb-slidebar .sb-left or .sb-right after you include slidebars.css.
https://jsfiddle.net/epxdusnf/

Thank you, it workds!


Configuring path schemes in TOML

Following Michael's idea, we can implement paths schemes customisation like thiss:
[path_schemes.cpg_google_storage]
default = "gs://cpg-{dataset}-{main/test}"
tmp = "gs://cpg-{dataset}-{main/test}-tmp"
analysis = "gs://cpg-{dataset}-{main/test}-analysis"
web = "gs://cpg-{dataset}-{main/test}-web"
archive = "gs://cpg-{dataset}-archive"
web_url = 'https://{main/test}-web.populationgenomics.org.au/{dataset}'

# default path scheme name
path_scheme = 'cpg_google_storage'

Any suggestions with regards to design and details? :)
This would be followed up by a scheme for Azure, and with some PRs in production-pipelines and AIP.

format() can take multiple arguments that aren't in the format string, e.g.
>>> '{this}'.format(this=1, that=2)
'1'

so groups can substitute in whichever pre-formatting they want, and cpg_utils will only embed dataset and main/test if present. e.g. gs://matts-bucket-{dataset} will be totally valid?

@MattWellie, sorry, not following, is there a bug somewhere? E.g. archive category is like your example: archive = "gs://cpg-{dataset}-archive" doesn't depend on main/test but should be valid.

@MattWellie, sorry, not following, is there a bug somewhere?

No bug! I'm really just talking to myself 😆 I was just having a play to see how tightly coupled output_path would be to the config templates

CPG infrastructure decides this as {path_prefix}{dataset}-{type}
https://github.com/populationgenomics/cpg-infrastructure/blob/97d72179727f27a991a318ffae3e7b30990fe92c/stack/cpg.toml#L3
Maybe we should bind the cpg.toml onto everything else for cpg-utils to access and decide the path from here. The TOML doesn't decide the categories, but the DRIVER does, so whether the driver should bake those values in at "deploy" time and have the analysis-runner just pull from a set of defined configs, that also seems reasonable.
Maybe then you only bind azure paths on if the dataset is actually deployed to that place.

Oh nice, I like the idea of binding to cpg.toml. @illusional do you think we can make sure that get_config() would load cpg.toml, passed from analysis-runner?

I think we should move some of this heavy lifting to analysis-runner, the analylsis-runner should take the cpg.toml and prepare the config for that run that gives something like:
[buckets]
default = "cpg-fewgenomes-main"
tmp = "cpg-fewgenomes-main-tmp"
web = "cpg-fewgenomes-main-web"

Or maybe something more specific for GCP and Azure:
[gcp.buckets]
default = "cpg-fewgenomes-main"
tmp = "cpg-fewgenomes-main-tmp"
web = "cpg-fewgenomes-main-web"

Which then the dataset_path / output_path should read. Gives the advantage that:

anyone can just prepare the final config and pass it to the pipeline, OR
anyone that deploys our analysis-runner sees those bonuses without having to modify cpg-utils

AND anyone that uses our cpg-infrastructure gets everything for free. Not saying we should be actively promoting people using our infrastructure, but it would be a bonus if it's not much work (and actually what we submitted to ABACBS this year).

Again, for workflows like seqr-loader, output_path and dataset_path is called for multiple datasets within one invocation (e.g. dataset=seqr but input_datasets=['acute-care', ...]), so hardcoding dataset in "cpg-fewgenomes-main" is not an option, unless we want to make a larger refactoring

Ah sorry, I missed that bit. Could it be:
[gcp.buckets]
# default is whatever dataset you ran it on
default = "cpg-seqr-main"
tmp = "cpg-seqr-main-tmp"
web = "cpg-seqr-main-web"

# include all "dependent" datasets here?
[gcp.buckets.fewgenomes]
default = "cpg-fewgenomes-main"
tmp = "cpg-fewgenomes-main-tmp"
web = "cpg-fewgenomes-main-web"

Such auto-generated config would be very long for large cohorts, so I'm not sure anyone would construct it manually, this there wouldn't be any point of running things outside of cpg-infra. In this case we might as well make cpg-infra a package and merge it with cpg-utils, so output_path could generate paths dynamically :)

Updated following the infra PR: https://github.com/populationgenomics/cpg-infrastructure/pull/20
Assumes web_url is one of the categories, which is not implemented in cpg-infra. We can discuss if we put it elsewehere.

Thanks Michael! I added a flag into dataset_path instead, should be good?

I don't undersatnd why PathSchemes are being removed from here

They were only there to support generating paths for different clouds, but it's all covered in infra now

Could be worth catching AttributeErrors and returnining a more friendly error, like "The category you selected was not available in this configuration".

Agree, I'll add something. But globally I think we should have a more general check in the config module

@lgruen merging is currently blocked because of your requested changes.


az vm repair create can fail due to policy restrictions

Describe the bug
It appears that the default repair VM config uses premium SSD disks, with no way to alter this.  In scenarios where an organization has put restrictions on potentially costly infrastructure, such as premium ssd, 'az vm repair create' can fail in ways that cannot be mitigated.
Related command
az vm repair create -n rhel9an -g litterbox
Errors
Do not have the call trace / error as this was during SR troubleshooting, but it was clear that policy was restricting the creation of the VM.
Issue script & Debug output
not available
Expected behavior
Would have preferred the default repair VM config to be using the least costly disk available, perhaps with an option to create higher tier disks if desired, much like public IPs can be optional.
Environment Summary
{
"experimental": false,
"extensionType": "whl",
"name": "vm-repair",
"path": "/home/pagienge/.azure/cliextensions/vm-repair",
"preview": false,
"version": "1.0.8"
}
Additional context
Premium disks may be more useful for Windows nested Hyper-V scenarios, but for Linux there is no gain from premium disk in the repair VM.
We are also using 'az vm repair' with ALAR in 'fixit' buttons in the Azure portal, this is one failure mode that could be disruptive in these automated use-cases.

Thank you for opening this issue, we will look into it.

@pagienge Thanks for the feedback! We are currently assessing this for our roadmap going forward.


cannot reconnect please reload the window

Does this issue occur when all extensions are disabled?: Yes

VS Code Version:
Version: 1.83.1
Commit: f1b07bd25dfad64b0167beb15359ae573aecd2cc
Date: 2023-10-10T23:45:31.402Z
Electron: 25.8.4
ElectronBuildId: 24154031
Chromium: 114.0.5735.289
Node.js: 18.15.0
V8: <IP_ADDRESS>-electron.0
OS: Linux x64 5.15.77-amd64-desktop
OS Version: Linux x64 5.15.77-amd64-desktop
remote OS Version:
ubuntu (docker container: Linux huaweiyun20220320211 3.10.0-1160.25.1.el7.x86_64 #1 SMP Wed Apr 28 21:49:45 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux)
openssh-client: 1:7.6p1-4ubuntu0.7
openssh-server: 1:7.6p1-4ubuntu0.7
openssh-sftp-server: 1:7.6p1-4ubuntu0.7

Steps to Reproduce:
It's always good before this bug happend. VScode open a window that showed "cannot reconnect please reload the window", and VScode show the windows again after I pushed the "reload window"  button. Sometimes I can open one remote windows, opening another remote window reproduced the bug. the possibility of bug is above 95% since it first happened.
my final solution is downgrading the remote ssh-server version to 1:7.6p1-4ubuntu0.5, and remove the vscode-server on the remote.
openssh-client: 1:7.6p1-4ubuntu0.5
openssh-server: 1:7.6p1-4ubuntu0.5
openssh-sftp-server: 1:7.6p1-4ubuntu0.5
vscode local log (~/.config/Code/logs/20231020T102530/window3/renderer.log):
2023-10-20 10:27:53.259 [warning] Via 'product.json#extensionEnabledApiProposals' extension 'ms-dotnettools.dotnet-interactive-vscode' wants API proposal 'languageConfigurationAutoClosingPairs' but that proposal DOES NOT EXIST. Likely, the proposal has been finalized (check 'vscode.d.ts') or was abandoned.
2023-10-20 10:27:53.277 [warning] Via 'product.json#extensionEnabledApiProposals' extension 'github.copilot-chat' wants API proposal 'terminalContextMenu' but that proposal DOES NOT EXIST. Likely, the proposal has been finalized (check 'vscode.d.ts') or was abandoned.
2023-10-20 10:27:54.180 [info] Invoking resolveAuthority(ssh-remote)...
2023-10-20 10:27:54.180 [info] [LocalProcess0][resolveAuthority(ssh-remote,1)][0ms] obtaining proxy...
2023-10-20 10:27:54.181 [info] [LocalProcess0][resolveAuthority(ssh-remote,1)][0ms] invoking...
2023-10-20 10:27:54.811 [error] [Extension Host] (node:13223) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
(Use `exe --trace-deprecation ...` to show where the warning was created)
2023-10-20 10:27:54.825 [info] [LocalProcess0][resolveAuthority(ssh-remote,1)][645ms] returned WebSocket(<IP_ADDRESS>:36945)
2023-10-20 10:27:54.825 [info] resolveAuthority(ssh-remote) returned 'WebSocket(<IP_ADDRESS>:36945)' after 646 ms
2023-10-20 10:27:54.827 [info] Creating a socket (renderer-Management-610f9fde-bfc2-4fb5-96af-3e58f79fa6ed)...
2023-10-20 10:27:54.828 [info] Creating a socket (renderer-ExtensionHost-fc80840b-0553-4a88-90fe-2af136981023)...
2023-10-20 10:27:54.924 [info] Creating a socket (renderer-Management-610f9fde-bfc2-4fb5-96af-3e58f79fa6ed) was successful after 97 ms.
2023-10-20 10:27:54.949 [info] Creating a socket (renderer-ExtensionHost-fc80840b-0553-4a88-90fe-2af136981023) was successful after 120 ms.
2023-10-20 10:28:00.207 [info] [perf] Render performance baseline is 55ms
2023-10-20 10:28:00.610 [error] [Extension Host] (node:502358) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
2023-10-20 10:28:01.883 [info] [remote-connection][Management   ][610f9…][reconnect] received socket close event (wasClean: false, code: 1006, reason: ).
2023-10-20 10:28:01.884 [error] {"isTrusted":true}
2023-10-20 10:28:01.884 [info] [remote-connection][Management   ][610f9…][reconnect] starting reconnecting loop. You can get more information with the trace log level.
2023-10-20 10:28:01.886 [info] [remote-connection][Management   ][610f9…][reconnect] resolving connection...
2023-10-20 10:28:01.887 [info] Invoking resolveAuthority(ssh-remote)...
2023-10-20 10:28:01.887 [info] [LocalProcess0][resolveAuthority(ssh-remote,2)][0ms] obtaining proxy...
2023-10-20 10:28:01.887 [info] [LocalProcess0][resolveAuthority(ssh-remote,2)][1ms] invoking...
2023-10-20 10:28:01.901 [info] [remote-connection][ExtensionHost][fc808…][reconnect] received socket close event (wasClean: false, code: 1006, reason: ).
2023-10-20 10:28:01.901 [error] {"isTrusted":true}
2023-10-20 10:28:01.901 [info] [remote-connection][ExtensionHost][fc808…][reconnect] starting reconnecting loop. You can get more information with the trace log level.
2023-10-20 10:28:01.902 [info] [remote-connection][ExtensionHost][fc808…][reconnect] resolving connection...
2023-10-20 10:28:02.522 [info] [LocalProcess0][resolveAuthority(ssh-remote,2)][636ms] returned WebSocket(<IP_ADDRESS>:40525)
2023-10-20 10:28:02.522 [info] resolveAuthority(ssh-remote) returned 'WebSocket(<IP_ADDRESS>:40525)' after 636 ms
2023-10-20 10:28:02.523 [info] [remote-connection][Management   ][610f9…][reconnect] connecting to WebSocket(<IP_ADDRESS>:40525)...
2023-10-20 10:28:02.523 [info] Creating a socket (renderer-Management-610f9fde-bfc2-4fb5-96af-3e58f79fa6ed)...
2023-10-20 10:28:02.524 [info] [remote-connection][ExtensionHost][fc808…][reconnect] connecting to WebSocket(<IP_ADDRESS>:40525)...
2023-10-20 10:28:02.524 [info] Creating a socket (renderer-ExtensionHost-fc80840b-0553-4a88-90fe-2af136981023)...
2023-10-20 10:28:02.625 [info] Creating a socket (renderer-Management-610f9fde-bfc2-4fb5-96af-3e58f79fa6ed) was successful after 103 ms.
2023-10-20 10:28:02.653 [info] Creating a socket (renderer-ExtensionHost-fc80840b-0553-4a88-90fe-2af136981023) was successful after 130 ms.
2023-10-20 10:28:02.700 [info] [remote-connection][Management   ][610f9…][reconnect] received socket close event (wasClean: false, code: 1006, reason: ).
2023-10-20 10:28:02.700 [error] {"isTrusted":true}
2023-10-20 10:28:02.703 [error] [remote-connection][Management   ][610f9…][reconnect][WebSocket(<IP_ADDRESS>:40525)] received error control message when negotiating connection. Error:
2023-10-20 10:28:02.703 [error] Error: Connection error: Unknown reconnection token (never seen)
    at x (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:676:23194)
    at u.value (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:676:14766)
    at c.z (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:87:1902)
    at c.fire (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:87:2119)
    at r.fire (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:616:14393)
    at b.z (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:616:17492)
    at u.value (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:616:16863)
    at c.z (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:87:1902)
    at c.fire (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:87:2119)
    at u.acceptChunk (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:616:12167)
    at vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:616:11454
    at u.value (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:676:4916)
    at c.z (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:87:1902)
    at c.fire (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:87:2119)
    at j.onload (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:676:3300)
2023-10-20 10:28:02.704 [error] [remote-connection][Management   ][610f9…][reconnect] A permanent error occurred in the reconnecting loop! Will give up now! Error:
2023-10-20 10:28:02.704 [error] Error: Connection error: Unknown reconnection token (never seen)
    at x (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:676:23194)
    at u.value (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:676:14766)
    at c.z (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:87:1902)
    at c.fire (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:87:2119)
    at r.fire (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:616:14393)
    at b.z (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:616:17492)
    at u.value (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:616:16863)
    at c.z (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:87:1902)
    at c.fire (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:87:2119)
    at u.acceptChunk (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:616:12167)
    at vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:616:11454
    at u.value (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:676:4916)
    at c.z (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:87:1902)
    at c.fire (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:87:2119)
    at j.onload (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:676:3300)
2023-10-20 10:28:02.714 [error] Error: Unable to read file 'vscode-remote://ssh-remote+<IP_ADDRESS>/data1/heran5/ChatGLM-6B/.vscode/settings.json' (Canceled: Canceled)
    at r.F (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:645:16628)
    at r.D (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:645:16423)
    at async r.C (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:645:15735)
    at async Object.factory (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:645:15664)
2023-10-20 10:28:02.714 [error] Error: Unable to read file 'vscode-remote://ssh-remote+<IP_ADDRESS>/data1/heran5/ChatGLM-6B/.vscode/tasks.json' (Canceled: Canceled)
    at r.F (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:645:16628)
    at r.D (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:645:16423)
    at async r.C (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:645:15735)
    at async Object.factory (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:645:15664)
2023-10-20 10:28:02.715 [error] Error: Unable to read file 'vscode-remote://ssh-remote+<IP_ADDRESS>/data1/heran5/ChatGLM-6B/.vscode/launch.json' (Canceled: Canceled)
    at r.F (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:645:16628)
    at r.D (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:645:16423)
    at async r.C (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:645:15735)
    at async Object.factory (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:645:15664)
2023-10-20 10:28:02.716 [error] Error: Unable to read file 'vscode-remote://ssh-remote+<IP_ADDRESS>/data1/heran5/ChatGLM-6B/.vscode/settings.json' (Canceled: Canceled)
    at r.F (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:645:16628)
    at r.D (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:645:16423)
    at async r.C (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:645:15735)
    at async Object.factory (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:645:15664)
2023-10-20 10:28:02.716 [error] Error: Unable to read file 'vscode-remote://ssh-remote+<IP_ADDRESS>/data1/heran5/ChatGLM-6B/.vscode/tasks.json' (Canceled: Canceled)
    at r.F (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:645:16628)
    at r.D (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:645:16423)
    at async r.C (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:645:15735)
    at async Object.factory (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:645:15664)
2023-10-20 10:28:02.716 [error] Error: Unable to read file 'vscode-remote://ssh-remote+<IP_ADDRESS>/data1/heran5/ChatGLM-6B/.vscode/launch.json' (Canceled: Canceled)
    at r.F (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:645:16628)
    at r.D (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:645:16423)
    at async r.C (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:645:15735)
    at async Object.factory (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:645:15664)
2023-10-20 10:28:02.763 [info] [remote-connection][ExtensionHost][fc808…][reconnect] received socket close event (wasClean: false, code: 1006, reason: ).
2023-10-20 10:28:02.764 [error] {"isTrusted":true}
2023-10-20 10:28:02.764 [error] [remote-connection][ExtensionHost][fc808…][reconnect][WebSocket(<IP_ADDRESS>:40525)] received error control message when negotiating connection. Error:
2023-10-20 10:28:02.765 [error] Error: Connection error: Unknown reconnection token (never seen)
    at x (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:676:23194)
    at u.value (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:676:14766)
    at c.z (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:87:1902)
    at c.fire (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:87:2119)
    at r.fire (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:616:14393)
    at b.z (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:616:17492)
    at u.value (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:616:16863)
    at c.z (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:87:1902)
    at c.fire (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:87:2119)
    at u.acceptChunk (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:616:12167)
    at vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:616:11454
    at u.value (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:676:4916)
    at c.z (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:87:1902)
    at c.fire (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:87:2119)
    at j.onload (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:676:3300)
2023-10-20 10:28:02.765 [error] [remote-connection][ExtensionHost][fc808…][reconnect] A permanent error occurred in the reconnecting loop! Will give up now! Error:
2023-10-20 10:28:02.765 [error] Error: Connection error: Unknown reconnection token (never seen)
    at x (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:676:23194)
    at u.value (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:676:14766)
    at c.z (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:87:1902)
    at c.fire (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:87:2119)
    at r.fire (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:616:14393)
    at b.z (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:616:17492)
    at u.value (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:616:16863)
    at c.z (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:87:1902)
    at c.fire (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:87:2119)
    at u.acceptChunk (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:616:12167)
    at vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:616:11454
    at u.value (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:676:4916)
    at c.z (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:87:1902)
    at c.fire (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:87:2119)
    at j.onload (vscode-file://vscode-app/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js:676:3300)

My solution above is not a final solution.. This bug also shows, but not as frequent anymore. Another remote docker container with openssh-server: 1:7.6p1-4ubuntu0.4 doesn't show this bug, but I can't downgrade the bug remote to this version of openssh-server

potentially a duplicate of this issue here, no answers yet from vscode team: https://github.com/microsoft/vscode/issues/195617

My solution is downgrading the jupyter extention to v2023.10.1002931100 (but I dont know my last jupyter version, just downloaded the second version from vscode office website), that works for me.


Add translog test to DataTypeTestCase and extend to more datatypes

Summary of the changes / Why this improves CrateDB
Ensure that we have coverage of translog compatibility for all data types.

Another option might be to split this in two, adding the translog test in one commit and extending test coverage to the various data types in a second.

@Mergifyio squash title+body


Rails - Unpermitted parameters

Hi, I'm trying to use ng-token-auth with devise_token_auth gem.
I got a problem when trying to register with angulajs. I got these two line:
Unpermitted parameters: confirm_success_url, config_name, registration
Unpermitted parameters: confirm_success_url, config_name, registration
Any idea ?
This is the logs :
Started POST "/auth" for <IP_ADDRESS> at 2014-10-15 14:17:57 +0200
Processing by DeviseTokenAuth::RegistrationsController#create as HTML
Parameters<EMAIL_ADDRESS>"password"=>"[FILTERED]", "password_confirmation"=>"[FILTERED]", "confirm_success_url"=>"http://localhost:3000/", "config_name"=>"default"<EMAIL_ADDRESS>"password"=>"[FILTERED]", "password_confirmation"=>"[FILTERED]", "confirm_success_url"=>"http://localhost:3000/", "config_name"=>"default"}}
Unpermitted parameters: confirm_success_url, config_name, registration
Unpermitted parameters: confirm_success_url, config_name, registration
(0.1ms)  begin transaction
(0.3ms)  SELECT COUNT(*) FROM "users"  WHERE "users"."provider" = 'email' AND "users"."email" =<EMAIL_ADDRESS>User Exists (0.3ms)  SELECT  1 AS one FROM "users"  WHERE LOWER("users"."email") =<EMAIL_ADDRESS>LIMIT 1
(0.1ms)  rollback transaction
Completed 403 Forbidden in 309ms (Views: 1.3ms | ActiveRecord: 6.4ms)
Thanks for your work.

I'm experiencing the same issue, and haven't been able to resolve it, I'm unclear as to what the solution here is.
I tried configuring ApplicationController to allow these parameters like described in devise docs, but that didn't resolve anything.

@milos89 - can you explain your issue exactly? Please include:

version of ng token auth module
version of devise token auth gem
body of the request to API
status code of the response from the API
body of the response from the API

Okay. The issue is exactly the same as @m2omou had.
This is server output:
Started POST "/api/auth" for ::1 at 2015-01-13 18:40:01 +0100
Processing by DeviseTokenAuth::RegistrationsController#create as HTML
  Parameters<EMAIL_ADDRESS>"password"=>"[FILTERED]", "password_confirmation"=>"[FILTERED]", "confirm_success_url"=>"http://localhost:3000/#/register", "config_name"=>"default"<EMAIL_ADDRESS>"password"=>"[FILTERED]", "password_confirmation"=>"[FILTERED]", "confirm_success_url"=>"http://localhost:3000/#/register", "config_name"=>"default"}}
Can't verify CSRF token authenticity
Unpermitted parameters: confirm_success_url, config_name, registration
Unpermitted parameters: confirm_success_url, config_name, registration
   (0.3ms)  BEGIN
   (0.3ms)  SELECT COUNT(*) FROM "users" WHERE "users"."provider" = $1 AND "users"."email" = $2  [["provider", "email"], ["email"<EMAIL_ADDRESS>   (0.3ms)  ROLLBACK
Completed 403 Forbidden in 121ms (Views: 0.4ms | ActiveRecord: 2.8ms)

This is registration form:
  <form ng-submit="registerUser()">

      Email
      <input ng-model="user.email" type="text" autofocus class="form-control" required="required"<EMAIL_ADDRESS>

      Password
      <input ng-model="user.password" type="password" class="form-control" required="required" placeholder="Password">

      Password Confirmation
      <input ng-model="user.password_confirmation" type="password" class="form-control" required="required" placeholder="Password Confirmation">

      <button type="submit" class="btn btn-primary btn-lg">Register</button>

  </form>

I'm using rails 4.2, and newest versions of ng-token-auth, devise-token-auth and angular (just installed them from bower).

I think I have probably messed up configuration on the server side, but this issue seemed to look the most to mine, and previous user seems to have resolved it.

This could be the problem: Can't verify CSRF token authenticity
Try setting protect_from_forgery to :null_session, like this:
class ApplicationController < ActionController::Base
  protect_from_forgery with: :null_session
  include DeviseTokenAuth::Concerns::SetUserByToken
  # ...
end

Didn't work. Don't have time to work this out today, but thanks for the help and the great work you have done on this plugin. Will post here if I resolve my issue.

@milos89 - please provide the following:

body of the request to API
body of the response from the API

These can be found in the network tab of your browser's web inspector.

Found what was wrong. Turns out my password was too short. I must have missed that in the console.
Thanks.

Hi,
I am trying to create(signup) a user from curl by making a post request. I am getting the same error i.e.
"errors": [
"Missing confirm_success_url param."
]
Since I am not using a browser, I don't see how that can be set. Could you please tell me how to signup a user from curl request. Since the front end of my application is not rails, I cannot use views of devise and have to resolve to provide apis for all the user management related queries.
Thanks.

Did you try passing any value as confirm_success_url?

This worked for me:
curl http://localhost:3000/api/v1/auth --data<EMAIL_ADDRESS>

Thanks guys for quick response. I tried using confirm_success_url as part of my post request data along with email, password and password_confirmation. also I added it to devise_parameter_sanitizer for sign up. I got following error:
Processing by DeviseTokenAuth::RegistrationsController#create as /
Parameters<EMAIL_ADDRESS>"password"=>"[FILTERED]", "password_confirmation"=>"[FILTERED]", "confirm_success_url"=>"foo"}
Completed 500 Internal Server Error in 89ms
NameError (undefined local variable or method `confirm_success_url=' for #User:0x00000002899c00):

Finally got it working. Here is what I did:

removed confirmable from User model
removed confirm_success_url from deserializer block in application controller.
removed confirm_success_url from post request as well.
It started working.
Thanks @jfields @callmeahab

having the same issue here as well. since unpermitted parameters aint the problem. can someone help me out with how to solve this?
Started POST "/api/auth" for <IP_ADDRESS> at 2015-04-14 21:47:05 +0800
Processing by DeviseTokenAuth::RegistrationsController#create as HTML
Parameters<EMAIL_ADDRESS>"password"=>"[FILTERED]", "password_confirmation"=>"[FILTERED]", "confirm_success_url"=>"http://localhost:9006/#/sign_up", "config_name"=>"default"<EMAIL_ADDRESS>"password"=>"[FILTERED]", "password_confirmation"=>"[FILTERED]", "confirm_success_url"=>"http://localhost:9006/#/sign_up", "config_name"=>"default"}}
Unpermitted parameters: confirm_success_url, config_name, registration
WARNING: Can't mass-assign protected attributes for User: email, password, password_confirmation
Unpermitted parameters: confirm_success_url, config_name, registration
(0.2ms)  BEGIN
(0.4ms)  SELECT COUNT(*) FROM users  WHERE users.provider = 'email' AND users.email IS NULL
(0.1ms)  ROLLBACK
Completed 403 Forbidden in 15ms (Views: 0.8ms | ActiveRecord: 0.7ms)
my routes.rb
namespace :api do
mount_devise_token_auth_for 'User', at: 'auth'
end
my usersessions.js
$scope.handleRegBtnClick = function() {
console.log("im here now")
$auth.submitRegistration($scope.registrationForm).then(function(resp) {
console.log($scope.registrationForm)
$auth.submitLogin({
email: $scope.registrationForm.email,
password: $scope.registrationForm.password
});
});
};

@boyfunky
In case you're still working on this, you need to stop passing this piece:
<EMAIL_ADDRESS>"password"=>"[FILTERED]", "password_confirmation"=>"[FILTERED]", "confirm_success_url"=>"http://localhost:9006/#/sign_up", "config_name"=>"default"}
And make sure that any passwords you try to pass in are at least 8 characters.

@vikashvikram
I believe I am having the same issues with creating a user via POST.  I also receive the "Missing confirm_success_url param".  And when I add "confirm_success_url"  to the devise_parameter_sanitizer that error disappears and I get "unknown attribute 'confirm_success_url' for User.  So I think I'm at mostly the same place you were at.
I just don't understand how to go about step 2 of your last post listing 3 steps that fixed the problem for you.  Could you explain how to go about  "2) removed confirm_success_url from deserializer block in application controller"?  Do you mean removing it from the devise_parameter_sanitizer?
Thanks.

Ok.  I updated the gem to -
gem "devise_token_auth", ">= 0.1.32.beta9"

and success...
I can get registration working and an email confirmation sent... if in devise_token_auth.rb I set -
   config.default_confirm_success_url = "confirmed"

However - including confirm_success_url as a parameter in the POST throws the same error as always - "unknown attribute 'confirm_success_url" and a "Completed 500 Internal Server Error" in the development.log.
I guess I will just have to stick to this setup.

I had same issue as @ericgaut . After checking this on clear application (it worked like need), I understood that problem in my application behavior. In the end problem was in custom errors interrupter on application controller level. It broken standard  Rails behavior, and cause this errors. I can be something like this:
ActionController::Parameters.action_on_unpermitted_parameters = :raise

  # Showing errors about unknown params
  rescue_from(ActionController::UnpermittedParameters) do |pme|
    render json: { error:  { unknown_parameters: pme.params } },
           status: :bad_request
  end

Hi,
i had same issue:
Started POST "/auth" for ::1 at 2016-03-07 11:36:24 +0100
Processing by DeviseTokenAuth::RegistrationsController#create as JSON
  Parameters<EMAIL_ADDRESS>"password"=>"[FILTERED]", "password_confirmation"=>"[FILTERED]", "confirm_success_url"=>"http://localhost:3000/"}
Unpermitted parameters: email, confirm_success_url, format
Unpermitted parameters: email, confirm_success_url, format
Unpermitted parameters: email, confirm_success_url, format
   (0.1ms)  begin transaction
   (0.1ms)  SELECT COUNT(*) FROM "users" WHERE "users"."provider" = ? AND "users"."email" IS NULL  [["provider", "email"]]
   (0.1ms)  rollback transaction
Completed 403 Forbidden in 106ms (Views: 2.4ms | ActiveRecord: 0.2ms)

and response JSON:

{
  "status": "error",
  "data": {
    "id": null,
    "provider": "email",
    "uid": "",
    "telephone": null,
    "name": null,
    "image": null,
    "email": null,
    "coins": null,
    "birthday": null,
    "address": null,
    "housenumber": null,
    "zipcode": null,
    "floor": null,
    "city_id": null,
    "created_at": null,
    "updated_at": null
  },
  "errors": {
    "email": [
      "can't be blank",
      "is not an email"
    ],
    "full_messages": [
      "Email can't be blank",
      "Email is not an email"
    ]
  }
}

So, if i want sign in with something other than their email address????
Thanks

Sorry,
the problem is ":authentication_keys => [:telephone]" in:
 class User < ActiveRecord::Base
  # Include default devise modules.
  devise :database_authenticatable, :registerable,
          :recoverable, :rememberable, :trackable, :validatable,
          :confirmable, :omniauthable, :authentication_keys => [:telephone]
  include DeviseTokenAuth::Concerns::User
end


[Impeller] ToColor converts an SkColor instead of a DlColor.

It likely doesn't matter, but the DisplayList should only contain DlColor objects yet the ToColor converter in Impeller takes an SkColor as an argument. It turns out these two types can be implicitly converted between each other with no work, but the ToColor method should probably be declared to take a DlColor (and a search done to make sure that there are no instances of SkColor left that it needs to convert).

I'm going to take this issue to test if my dev environment is setup correctly 😸