MXNet 2.0 up to 10x slower than 1.x on Windows

I am building master branch (v2.0) from commit fabcd145cd496628791f9f2ea813048360ac33ca and 1.x branch (v1.6) from commit 6eec9da55c5096079355d1f1a5fa58dcf35d6752 and our internal benchmarks show a severe performance loss of v2.0 on Windows.
 I would consider this to be a bug in the CMake code because one shouldn't need extra stuff besides USE_OPENMP=ON to fully leverage openMP. Also, Linux and Mac need the analogous of those flags (-fopenmp on Linux, -Xpreprocessor -fopenmp on Mac) on both v1.6 and v2.0 to get optimal performance. This is not evidence that the v2.0 slowdown problem on Windows is due to openMP but I think it's worth mentioning.

@leezu I haven't originally, but now I have built with openblas instead of MKL and I'm getting the same problem.
However, I have discovered that benchmarks of v2.0 builds (both MKL and OpenBLAS) are insensitive to setting MXNET_ENGINE_TYPE="NaiveEngine": performance doesn't get better or worse. On the other hand, with v1.6 + naive engine I see a slowdown as expected, and performance becomes more similar to v1.6 (although still up to 2.5x faster in some cases).
So it seems there is something wrong with multithreading in v2.0, perhaps with OpenMP? Do I have some bad setting in my OpenMP build flags for v2.0?

@anko-intel is this something your team could help take a look?

@matteosal would you be able to share one workload that we can try to reproduce the results with?

@szha this is the python equivalent of one of our cases (a single large GRU and a few other operations):
net.zip
import mxnet as mx
import time

print(mx.__version__)

batch_size = 64
n_iter = 20

sym = mx.symbol.load('net.json')
shapes = {'Input': [batch_size, 100, 16], ':FusedWeights': [3201024], 'State': [64, 1024]}
def gen_arrays():
	return {name: mx.nd.ones(shape) for (name, shape) in shapes.items()}
if(mx.__version__ == '2.0.0'):
	ex = sym._bind(mx.cpu(), gen_arrays(), args_grad=gen_arrays())
else:
	ex = sym.bind(mx.cpu(), gen_arrays(), args_grad=gen_arrays())

input_data = [mx.ndarray.random.uniform(0, 1, [batch_size, 100, 16]) for i in range(n_iter)]
outgrad = [mx.nd.ones([batch_size, 100, 1024]), mx.nd.ones([batch_size, 1024])]

start = time.time()
for i in range(n_iter):
	ex.arg_dict['Input'] = input_data[i]
	ex.forward()
	ex.backward(outgrad)
	mx.ndarray.waitall()
end = time.time()

inputs_per_sec = batch_size * n_iter / (end - start)

print(inputs_per_sec)

With v1.6 I get around 70 inputs/sec and only 22 with v2.0.
With v2.0 I have used static allocation to stay as close as possible to v1.6 (although it didn't seem to matter much), but I am on fabcd145cd496628791f9f2ea813048360ac33ca and _bind doesn't expose static allocation there (it was added later in https://github.com/apache/incubator-mxnet/pull/20462), so I had to manually pass flags=[("static_alloc", "true")] here.
Also, back to MXNET_ENGINE_TYPE="NaiveEngine": I have tried to run the benchmark with this setting again and this time I have found to difference in both v1.6 and v2.0, so I'm not sure that's a thing anymore.

@matteosal
I tried to reproduce the slowdown with your configuration on a machine with I9-7940X CPU but the performance loss wasn't as severe as in your case.
I got ~153 inputs/sec with mxnet 2.0 and ~165 inputs/sec with mxnet 1.6 using the script you provided.
Could you please share more details on the hardware config you used in the benchmarks, so I can try to reproduce the slowdown?

@RafLit I am on an i9-9980HK, but I don't think this is hardware dependent because I have double checked the same build on a different machine with an i7-4770K and I still see the slowdown (~20 inputs/sec vs ~40 inputs/sec).
So I still think this is due to the build settings. I have shared my build here so that you can try this out and maybe get some insight from inspecting it: https://drive.google.com/drive/folders/1STQ2Md3OIj7RBxJYTyvqetoGAh9st5tT?usp=sharing
Could you share yours so I can do the same?

@matteosal this is my MXNet 2.0 build: https://drive.google.com/drive/folders/1KmrjQ6cePc4WCUdFF__MldIiXYty4vTc?usp=sharing.
I have checked your build on my setup and there was no significant difference in performance compared to my mxnet 2.0 build (and 1.6 as well).
From what I've seen slowdowns like this can happen whenever there are 2 versions of OpenMP used at the same time. Could you try to compile MXNet using the Visual Studio OpenMP (vcomp) only and check whether the slowdown is present?

@RafLit thanks, we will do that and report back. In the meantime, I have to say that your 2.0 build is as fast as my 1.6 build on my machine so it doesn't have any problem. Moreover, there are two clear differences between our builds:

dumpbin /dependents shows an explicit dependency on libiomp5md.ldd for your build while it doens't for mine
Your library file is around 95MB while mine is 75MB

This is clear evidence that (at least part of) the problem lies with the build procedure and not the runtime environment. And given all this, I find it very weird that my build doens't run slower on your setup.

@RafLit while trying to build with Visual Studio OpenMP we discovered that building with VC2019 instead of VC2017 makes our build as fast as yours. This seems to be expected because /openmp:experimental is only supported on VC2019 (https://docs.microsoft.com/en-us/cpp/build/reference/openmp-enable-openmp-2-0-support?view=msvc-170). But there is no failure for using  /openmp:experimental with VC2017 so we never realized it. What VC version have you used to build? Should the CMake script fail explicitly for VC2017 and below?
Also, using VC2019 makes our build as fast as yours, but still not as fast as mxnet 1.6 for some models. The worst slowdown went from about 10x with VC2017 to about 3x with VC2019, and it's for this case (symbol attached):
net.zip
import mxnet as mx
import time

print(mx.__version__)

batch_size = 64
n_iter = 20

sym = mx.symbol.load('net.json')
arg_shapes = sym.infer_shape(**{'.Inputs.Input': (batch_size, 1, 64, 64)})[0]

def gen_arrays(shapes):
	return [mx.nd.ones(shape) for shape in shapes]

if(mx.__version__ == '2.0.0'):
	ex = sym._bind(mx.cpu(), gen_arrays(arg_shapes), args_grad=gen_arrays(arg_shapes))
else:
	ex = sym.bind(mx.cpu(), gen_arrays(arg_shapes), args_grad=gen_arrays(arg_shapes))

input_data = [mx.ndarray.random.uniform(0, 1, [batch_size, 1, 64, 64]) for i in range(n_iter)]
outgrad = [mx.nd.ones([batch_size, 1])]

start = time.time()
for i in range(n_iter):
	ex.arg_dict['.Inputs.Input'] = input_data[i]
	ex.forward(is_train=True)
	ex.backward(outgrad)
	mx.ndarray.waitall()
end = time.time()

inputs_per_sec = batch_size * n_iter / (end - start)

print(inputs_per_sec)

My results are:
MXNet 2.0 + VC2017: 12 inputs/s
MXNet 2.0 + VC2019 (and your build): 55 inputs/s
MXNet 1.6: 140 inputs/s
Do you reproduce a similar performance loss (55/140)?

@matteosal

What VC version have you used to build?

I used VC2019.

Should the CMake script fail explicitly for VC2017 and below?

I'm not sure, I think the script just sets the compile flags.
I ran the tests with the model you provided and got similar results:
MXNet 2.0: 95 inputs/s
MXNet 1.6: 250 inputs/s
Good find with the Visual Studio version! I will try to investigate the problem now that I can reproduce the slowdown and get back to you once I find something.

I'm not sure, I think the script just sets the compile flags.

I mean, do you think that the script should be changed to check the MSVC version and fail if it's older than VC2019? I can open a PR adding this check.

@matteosal
I found that the slowdown is caused by enabling large tensor support (the -DUSE_INT64_TENSOR_SIZE cmake flag). There are several things that can be done to get rid of the slowdown:

Switching from ilp to lp and disabling the large tensor support (if you don't need it).
Compiling with onednn enabled - it is a lot faster and doesn't suffer from the slowdown.
The individual naive operators used by you, like deconvolution, can be looked into and possibly optimized with large tensor support enabled.

I ran some tests with configurations proposed by me and got the following results for mxnet 2.0:
large tensor support enabled, ilp, USE_ONEDNN=OFF: 95 inputs/s
large tensor support disabled, lp, USE_ONEDNN=OFF: 231 inputs/s
large tensor support enabled, ilp, USE_ONEDNN=ON: 1050 inputs/s
large tensor support disabled, lp, USE_ONEDNN=ON: 1154 inputs/s

I mean, do you think that the script should be changed to check the MSVC version and fail if it's older than VC2019? I can open a PR adding this check

I think it's a good idea, but it would probably be better to just display a warning about a potential slowdown instead of failing.

@RafLit
Thanks for the analysis! Unfortunately we can't switch to LP64, but we are surely going to try out oneDNN as it was on our list anyway. It will take a while but we are surely going to try it out.

@RafLit thank you for investigating the issue. Would you like to open a PR to error out on the known bad build settings or at least not use it by default?
For example, you can change the default by modifying the
https://github.com/apache/incubator-mxnet/blob/9fa75b470b8f0238a98635f20f5af941feb60929/CMakeLists.txt#L352
or error out via
if((NOT USE_ONEDNN) AND USE_INT64_TENSOR_SIZE)
  message(FATAL_ERROR "")
endif()

Does the observation that VC2017 is much slower than VC2019 still hold, or was the root-cause here also the ILP/onednn setting?
cc @barry-jin

@leezu it was 2 independent slowdowns one on top of the other: for the case of this toy model, using VC2017 introduced a ~4.5x slowdown (55/12) while INT64 was around 2.5x (140/55) on my machine.
As for the cmake changes, I think the combination (NOT USE_ONEDNN) AND USE_INT64_TENSOR_SIZE should produce a warning instead of a failure because that's a genuine combination that someone might still want to use for some reason (e.g. in an environment that requires MKL ILP64), even if slower.
VC2017 on the other hand might fail explicitly because the compiler version is generally not a constraint.


Added RadioPlayer Repo & Voice Chat

Please resolve the conflict (just force push)

Thanks!

Thank you also sir!


How to use gcov to trace the whole project?

Hi, I want to use gcov to see the coverage information of the project through the following building commands:
YACC=byacc CFLAGS="-fprofile-arcs -ftest-coverage" CXXFLAGS="-fprofile-arcs -ftest-coverage" LDFLAGS="-fprofile-arcs -ftest-coverage" ./autogen.sh && \
    YACC=byacc CFLAGS="-fprofile-arcs -ftest-coverage" CXXFLAGS="-fprofile-arcs -ftest-coverage" LDFLAGS="-fprofile-arcs -ftest-coverage" ./configure --disable-shared --enable-static --prefix=$PWD/install --sysconfdir=/etc --with-privsep-user=root --with-bgplgd-user=root && \
    YACC=byacc CFLAGS="-fprofile-arcs -ftest-coverage" CXXFLAGS="-fprofile-arcs -ftest-coverage" LDFLAGS="-fprofile-arcs -ftest-coverage" make && make install

However, since the openbsd is the submodule, I found I cannot trace the coverage information of this submodule. Do you have any suggests? Thanks.

autogen.sh copies all relevant files out of the openbgpd-openbsd repo. During compile no file is referenced there anymore.
I never looked a gcov but it is well possible it trips over the chroots in bgpd. At least with gprof that is normally the number one issue (with number 2 being the issue that bgpd forks and execs into multiple processes).
Please provide more information. Especially configure output and make V=1 output.

autogen.sh copies all relevant files out of the openbgpd-openbsd repo. During compile no file is referenced there anymore. I never looked a gcov but it is well possible it trips over the chroots in bgpd. At least with gprof that is normally the number one issue (with number 2 being the issue that bgpd forks and execs into multiple processes).
Please provide more information. Especially configure output and make V=1 output.

So that the openbsd  directory is no more needed after autogen.sh?

Yes, it is no longer needed after autogen.sh ran.


Missing: upgrade-insecure-request

The option to set an upgrade insecure request is missing, it would be nice to have that option :)

I notice you've fixed this already in dev-master, just not released yet.
👍


Getting started example is not working

Description
The getting started example of Go client API is not working
Steps to reproduce the issue:

wget https://github.com/containerd/containerd/releases/download/v1.2.7/containerd-1.2.7.linux-amd64.tar.gz
tar -zvf containerd-1.2.7.linux-amd64.tar.gz
creating the base configuration file
adding containerd/bin to PATH
go build main.go
sudo main

Describe the results you received:
Got this error:
2019/07/31 16:35:51 runtime "io.containerd.runc.v2" binary not installed "containerd-shim-runc-v2": file does not exist: unknown

Describe the results you expected:
Something like:
1:C 04 Aug 20:41:37.682 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 04 Aug 20:41:37.682 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 04 Aug 20:41:37.682 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
1:M 04 Aug 20:41:37.682 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
1:M 04 Aug 20:41:37.682 # Server can't set maximum open files to 10032 because of OS error: Operation not permitted.
1:M 04 Aug 20:41:37.682 # Current maximum open files is 1024. maxclients has been reduced to 992 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
1:M 04 Aug 20:41:37.683 * Running mode=standalone, port=6379.
1:M 04 Aug 20:41:37.683 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 04 Aug 20:41:37.684 # Server initialized
1:M 04 Aug 20:41:37.684 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:M 04 Aug 20:41:37.684 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 04 Aug 20:41:37.684 * Ready to accept connections
1:signal-handler<PHONE_NUMBER>) Received SIGTERM scheduling shutdown...
1:M 04 Aug 20:41:40.791 # User requested shutdown...
1:M 04 Aug 20:41:40.791 * Saving the final RDB snapshot before exiting.
1:M 04 Aug 20:41:40.794 * DB saved on disk
1:M 04 Aug 20:41:40.794 # Redis is now ready to exit, bye bye...
redis-server exited with status: 0

Output of containerd --version:
containerd github.com/containerd/containerd v1.2.6 894b81a4b802e4eb2a91d1ce216b8817763c29fb

Any other relevant information:
Running the same image using ctr works
Ubuntu 16.04.6
runc version 1.0.0-rc8
go version go1.12.7 linux/amd64

Just as a small update, if I create v2 just by copying v1 it works
cp containerd-shim-runc-v1 containerd-shim-runc-v2

From this and your other issue about LogFile, you don't have the current daemon installed.  You are using a master containerd client running a 1.2.6 daemon do it has some discrepancies

@crosbymichael is there some way to tell the client to downgrade to the version of the containerd daemon? The rationale is that the containerd installation might come from the OS packages, which usually lack behind. But in order to get the containerd client compiling at all, atm we need to use github.com/containerd/containerd@master. So there's plainly a catch-22 here...! :(

@TheDiveO This can be resolved by adding the line
containerd.WithRuntime("io.containerd.runtime.v1.linux", nil),
to the client.NewContainer call in the example which will allow it to run against earlier daemon versions.

Just as a small update, if I create v2 just by copying v1 it works
cp containerd-shim-runc-v1 containerd-shim-runc-v2

Thanks it worked but you didn't mentioned path for this file, it is there in  /usr/bin
my docker is not staring from 2 days with error
docker: Error response from daemon: runtime "io.containerd.runc.v2" binary not installed "containerd-shim-runc-v2": file does not exist: unknown. , I tried different ways nothing worked , but this above solution worked thank you sooo much

I found that When Re-installing docker in AWS Amazon Linux 2 it just reinstalled Docker but when Installing Fresh Docker all Other dependencies Like contained, runc are installed with Docker, Therefore, Reinstallaling containerd resolved all the issues.
sudo yum reinstall containerd docker -y
This will resolve it. No need to Copy-Paste them ( i.e. cp containerd-shim-runc-v1 containerd-shim-runc-v2 )


Too sensitive to inotify?

I have this .envrc:
cp gemset.nix tmp-gemset
bundix --gemset=tmp-gemset
if diff tmp-gemset gemset.nix; then
  echo No change to gemset.nix
  rm tmp-gemset
else
  echo Updating gemset.nix
  mv tmp-gemset gemset.nix
fi
use lorri
watch_file Gemfile.lock

The idea is that, if Gemfile.lock changes, I want to run bundix to generate a new gemset.nix, which my shell.nix depends on.
What I'm finding is that lorri runs nearly every prompt. I think that what happens is that the gemset.nix gets read in the process which generates an inotify event? But I'm not 100% on this particular and not sure how to investigate.

Nope: I think I found it. I'm doing work on a Rubygem, and as a result the bundix generated gemset.nix includes path = ./.; which I gather will induce Lorri to watch the whole directory.


reduce required privileges

right now when the script asks for authorization it asks for pretty much pwnage-level access to the user's entire google account.
reduce the security threat surface -- find a way to make this work with OnlyCurrentDoc.
https://developers.google.com/apps-script/guides/services/authorization
the risk is that a malicious commit on the legalese codebase will embed undesirable content in an xml template file which then runs with user permissions with access to all the user's docs. this is clearly undesirable.
a functionally equivalent man-in-the-middle attack would intercept the UrlFetch() operation and return a malicious XML template file, either attacking obtainTemplate or INCLUDE(Available Templates).
lodging the XML templates inside the app itself is a seemingly attractive alternative, but it reduces to the same threat scenario because that data has to populate from somewhere in the first place.
we should require that all committers with access to GitHub must have 2FA.
ideally we would reduce the authorization scope of this script to only the current doc.
but we need a way to share the resulting PDF with the user without access to everything in Drive!

the google security model may have matured since this issue was created, making it easier to work on this issue.


Change ParameterData#getJavadoc to return a list of strings

Currently, MappingDataContainer.ParameterData#getJavadoc returns a single String. This precludes the possibility of multi-line javadocs for parameters, which is certainly possible both by javadoc and Enigma, the current mapping interface tool used by Parchment.
I plan to change that method to return a List<String> instead, to support the possibility of multi-line javadocs for parameters officially. This has the following consequences:

The change in the API interface is a breaking API change and will lead to a bump of Feather's version from 1.0.1 to 2.0.0.

The JSON format of the Mapping Data Container specification will be updated to update the javadoc key for parameters from a string to an array of strings. This necessitates a bump in the format version from the current 1.0.0 to 2.0.0.

The parsers and writers for the MDC format (in the io-gson and io-moshi subprojects) will be updated to support 2.0.0 of the MDC JSON format, consistent with the change in interface and Feather version.

Support will be added to allow reading the v1.0.0 MDC format. This will likely be a default enabled option in the current parsers.

Developers which use Feather will be informed of the breaking change sometime before Parchment exports are produced with the new format, giving them a period to update their applications to use the new Feather version and test compatibility. Consumers of their applications also need to be informed to update.

I will also look over the codebase for any API-breaking changes which can get rolled into the 2.0.0 version update.

Why not take the Tiny v2 route and simply embed an escaped \n into the single string?


Fix nfs location

Fixes NFS location bug introduced in #54
Changes proposed in this pull request:

Replace /net/nfs with /net/nfs.cirrascale. The former doesn't actually work for nfs, tested here. I have tested that using /net/nfs.cirrascale instead works here.

Before submitting

[x] I've read and followed all steps in the Making a pull request
section of the CONTRIBUTING docs.
[x] I've updated or added any relevant docstrings following the syntax described in the
Writing docstrings section of the CONTRIBUTING docs.
[ ] If this PR fixes a bug, I've added a test that will fail without my fix.
[ ] If this PR adds a new feature, I've added tests that sufficiently cover my new functionality.

Huh, this should work. Running ls /net/nfs/allennlp/hamishi/.hf/ directly on allennlp-cirrascale-01 works fine. @codeviking any idea what's wrong here?

No idea. Do we know what host this failed on? It's possible the generic NFS paths are missing on that machine, because of some sort of configuration error.

It was on allennlp-cirrascale-01.reviz.ai2.in, which definitely has the generic NFS path

Right, as you noted, it seems to work:
sams@allennlp-cirrascale-01:~$ ls /net/nfs/allennlp/hamishi/.hf
datasets  hub  modules  transformers

Maybe there was a network hiccup (though NFS is supposed to absorb those). Is it possible the .hf directory didn't exist yet, and something else created it?


Overloaded constructors: 'TValue' not assignable to 'string'

Bug Report
🔎 Search Terms
constructor, overload, TValue, jsdoc
🕗 Version & Regression Information

This changed between versions "recent"? and ~1.77.1-ish
I was unable to test this on prior versions because PR #52577 was not implemented (there was no reliable way to define overloads for class constructors)

Recent PR fixed the use of jsdoc-defined overloads for class constructors

Bug report: #52477
Bug fix (for jsdoc @overload): #52577

That seems to have made it in to recent VS Code release; I'm using:

Version: 1.77.1 (user setup)
Commit: b7886d7461186a5eac768481578c1d7ca80e2d21
Date: 2023-04-04T23:21:11.906Z
Electron: 19.1.11
Chromium: 102.0.5005.196
Node.js: 16.14.2
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.19045
Sandboxed: Yes

⏯ Playground Link
(Updated to exhibit error): Playground link
💻 Code
// @ts-check

/**
 * @template TValue
 * @type {StringKeyMap<TValue>}
 */
export class StringKeyMap extends Map {
    /**
     * @constructor
     * @overload
     * @param {Iterable<[string, TValue]>} iterable
     * @param {string} form
     *//**
     * @constructor
     * @overload
     * @param {string} form
     *//**
     * @constructor
     * @overload
     * @param {Iterable<[string, TValue]>} iterable
     *//**
     * @constructor
     * @overload
     */
    constructor(A, B) {
        super()
        // ...
    }

    /**
     * @override
     * @param {string} key
     * @param {TValue} value
     * @returns {this}
     */
    set(key, value) {
        // normalize key (removed for brevity)
        return super.set(key, value)
    }
}

/** @type {StringKeyMap<number>} */
const foo = new StringKeyMap() // <-- code hint error appears here

🙁 Actual behavior
In VS Code editor, if I try and instantiate my class using code like this:
class foo {
    // ...

    /** @type {StringKeyMap<string>} */
    #tags

    /**
     * @param {Iterable<string>} tags
     */
    constructor(tags) {
        this.#tags = new StringKeyMap() // <-- error shows here
    }

    // ....
}

This happens:

Hovering the error shows:

The code still works fine when used in node, the issue is constrained to the VS Code editor which seems to think TValue is a type rather than a template.
Possibly related to comment here about type checking being broken: https://github.com/microsoft/TypeScript/issues/52477#issuecomment-1414165166
🙂 Expected behavior
TValue is a template and thus should adopt the @type {StringKeyMap<string>} defined for #tags .

I can't repro this in TS 5.0.2. or nightly. Can you try installing the TS/JS nightly extension and seeing if that fixes it?

I can't repro this in TS 5.0.2. or nightly. Can you try installing the TS/JS nightly extension and seeing if that fixes it?

I've installed that extension and restarted VS Code and still getting the error. I'll try and make a smaller test case exhibiting the error.

I've updated code sample in OP to minimal code that reproduces the error.

Simpler:
/**
 * @template TValue
 */
export class StringKeyMap {
    /**
     * @constructor
     * @overload
     */
    constructor() {
    }

    /**
     * @param {TValue} value
     */
    bar(value) {
    }
}

/** @type {StringKeyMap<number>} */
let foo;
foo = new StringKeyMap() // <-- code hint error appears here

Removing the @overload line makes the error go away, but this shouldn't have any effect


Restore blender addon to export quad gltf.

Previously Facebook made an addon to allow quads to pass through from Blender.
I may be able to revive it and publish the extension as https://github.com/omigroup/gltf-extensions

If its just quads I don't think an extra extension is necessary. Like, if it's not too much effort it'd be better than the current solution by a mile cause you could show the problematic faces, but in most cases the recreation by merging vertices should work (you have to actively try to break it, like disconnecting a face so vertices overlap)

Because of the other pr of enabling loop subdivide for trimeshes, this has less importance.


Crashed: com.google.fira.worker

Step 1: Describe environment
Xcode version: 12.4
Firebase SDK version: 6.26.0
Firebase Component: (Performance, Crashlytics, Analytics, DynamicLinks, Core)
Installation method: `CocoaPods
Step 2: Describe the problem
Crashlytics is reporting crashes. The crash rate is so high, so it is difficult to reproduce it. Can you help sort it out, since tool is needed for us?
Steps to reproduce:
I have not been able to reproduce it so far.
//stacktrace
Crashed: com.google.fira.worker
0  libobjc.A.dylib                0x1bd284de0 objc_release + 16
1  libobjc.A.dylib                0x1bd286574 AutoreleasePoolPage::releaseUntil(objc_object**) + 200
2  libobjc.A.dylib                0x1bd286450 objc_autoreleasePoolPop + 208
3  libdispatch.dylib              0x1a8b37260 _dispatch_last_resort_autorelease_pool_pop + 40
4  libdispatch.dylib              0x1a8ae0908 _dispatch_lane_invoke$VARIANT$mp + 520
5  libdispatch.dylib              0x1a8aea338 _dispatch_workloop_worker_thread + 712
6  libsystem_pthread.dylib        0x1f16335a4 _pthread_wqthread + 272
7  libsystem_pthread.dylib        0x1f1636874 start_wqthread + 8
com.apple.main-thread
0  libsystem_kernel.dylib          0x1d4db2644 mach_msg_trap + 8
1  libsystem_kernel.dylib          0x1d4db1a48 mach_msg + 72
2  CoreFoundation                  0x1a8e7e0fc __CFRunLoopServiceMachPort + 376
3  CoreFoundation                  0x1a8e78570 __CFRunLoopRun + 1176
4  CoreFoundation                  0x1a8e77ba0 CFRunLoopRunSpecific + 572
5  GraphicsServices                0x1bfbb5598 GSEventRunModal + 160
6  UIKitCore                       0x1ab7673d8 -[UIApplication _run] + 1052
7  UIKitCore                       0x1ab76c958 UIApplicationMain + 164
8  ConsumerMobileAppNonClinical    0x1006ed80c main + 35 (AppDelegate.swift:35)
9  libdyld.dylib                   0x1a8b56568 start + 4

I found a few problems with this issue:

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.
This issue does not seem to follow the issue template. Make sure you provide all the required information.

Thanks for the update . could you please tell me the what are required information to help your investigation.

Thanks for reporting, @perumal1216. Could you try updating your SDK to the latest one and see if the issue still occurs?

Related issue at #6802. This may be related to issues that have been fixed in FirebasePerformance.

Thanks for reporting, @perumal1216. Could you try updating your SDK to the latest one and see if the issue still occurs?

thanks for your update @rizafran   .  we are already having Firebase SDK version: 7.6.0 and  Firebase/Performance (7.6.0)

The current release is 7.11.0

Do we have a any potential fix on release 7.11.0 version related to above defined issue . if yes please provide details that could be great help to us.

@perumal1216 this kind of crash can also be caused by memory corruption in your app, consider debugging your app with some memory diagnostic tools (like asan, zombies) enabled.


[List] Auto-expand SelectableList

Resolves #2895, #3023.

@mbrookes Really cool feature :100:!

Thanks! :smile:
I'm glad you said no to the easy option! :wink: I really didn't think I'd pull it off when I first looked at the code for this HOC (which is why I took the the alternative approach), but with a little bit of tinkering it came together in the end!

@mbrookes Can you squash down?


SAS URL Export Hangs

If the issue is to do with Azure CLI 2.0 in-particular, create an issue here at Azure/azure-cli

Extension name (the extension in question)
image copy extension
Description of issue (in as much detail as possible)
A SAS URL has been generated for this snapshot for export. You can cancel the export by clicking the button below. This will revoke the SAS URL and may cancel any in-progress transfers if the snapshot is currently being downloaded to another location.
After the image is created and moved to the new resource group, the snapshot has a hung export SAS Url which has to be manually stopped in order for the resource group to be cleaned up.

image triaged.


Python: Loops- break

Description 📜
Here, I explained about the python's break statement in the audio format along with ppt containing relevant information and examples.
Fixes # 561

Type of change 📝

[x] Audio (Should be in mp3 format Includes speech clarity, Concise ,Low distortion)

Domain of Contribution 📊

[x] Python

Checklist ✅

[x] I follow Contributing Guidelines & Code of conduct of this project.
[x] I have performed a self-review of my own code or work.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] My changes generates no new warnings.
[x] I'm GWOC'21 contributor

Screenshots / Gif (Optional) 📸

@kumarankm
Please do review this and let me kniw if any changes are required.

@achsahxavier I guess you have made pr by seeing my previous video. But it is not a right way
Close this pr and create a new one by following this video.
https://youtu.be/Ud6MIDHTzhc
Sorry for the inconvenience!


I don't understand what is mean this results

Hi!
I need your help.
I received this output. What's mean this results? How I could get drone_id data from this results?
Found burst @ 587225372 / 587225371
7c3ff9feefc6eeebfffc60f5f57db7c1e38febffafbd5417dfffde7fbcSending message with af27952 elements (7e27952) starting at 587225371
f4Found burst @ 587253384 / 587253383
7b34b9bfcfdaec37bfffb9669deedffde8aef6ffbe65b5056dbf7bf76f17beb5fc1d8a8cffef7bafc6f6f7ff6f17fff7fb3033d97fdf4bddc95fffe6acfc1ffe77e5ed273fbfffc52891fbfd979af5efed5f9c95e5bffbfee376737b743779bfffd9fd6d415fdfecd93bc6e7efff297773f7dfa33ebbfddfb665aeed7fde4ee62f7befeeff5a3b74b77958396dfffd99
CRC Check Failed!
Sending message with 27952 elements (27952) starting at 587253383
ZC start index: 9761, est start index: 3025
Sending message with 27952 elements (27952) starting at 591369583
Found burst @ 591397563 / 591397562
Sending message with 27952 elements (27952) starting at 591397562
Found burst @ 591427851 / 591427850
Sending message with 27952 elements (27952) starting at 591427850
Found burst @ 591490373 / 591490372
f68ffffdefff57fc877ff46ac7fffbefaff72eff75b5ddfb7febfefbffd9ff7fbfed6f7cdb7bfd703becdf3b2fbffdb7eb7ee87ff7c5f9fdb9ff5fe7ed0e7ffc9fdffacb8ffffbb6ddf7bfSending message with 27952 elements (27952) starting at 591490372
fddd4fcc7dffff56b6efddf7fbfdfd3fefbbe99fe7bfbde1e9fe77cfb57e6dfffe6ffad3bbfff7febdacaeff7fefe663efbade97dacdfffbf57eb5f8f7fd59c3fffff777767ecfb5ffeb3bb77f9f57f55ddaf5dfbc63ff7a7f3e13bdaf7bcfdbe777aff47e
CRC Check Failed!
Found burst @ 591547527 / 591547526
Sending message with 27952 elements (27952) starting at 591547526
Found burst @<PHONE_NUMBER> /<PHONE_NUMBER>
Sending message with 27952 elements (27952) starting at<PHONE_NUMBER>
35d77a616024d9daf4c571e6b6afb784068787d71b363667777a0c1a50f73efe0a7f78fb6095903f7c7161f9ea3b2be847d36be5a7d105c8accf684557a1d457e113969fFound burst @ f2b41ac8471f6c4d6714119062905751 / eae9d2a5541190629056b17ff4
32394d6b6f9c4325b97d8dc9a1758f6de6646fbfeb975493dab0beb416a42c4e771c476be35a2a50d05ece30898deb47892165a6d9f7f48033afbccfa6e917afd7df8889d07f9e37043cdb3f1a526a8bdcfb5a907fef3c09ad
CRC Check Failed!
Sending message with 27952 elements (27952) starting at<PHONE_NUMBER>

If you Google gr-droneid YouTube you should find three videos that’ll walk through taking the capture from the air into decoded format. Hope that helps.

@alphafox02 Unfortunately I have seen this  video but it didn't help. In this video version of dii_droneid library is so old. Output is different. I'm not sure but as I can see in latest version library gr-droneid could decode frame in realtime.
Please, tell me how it have to work?

The same steps can apply. I tested it. Still in the latest repo if gr-drone for gr10 there’s still bursts saved to /tmp along with what you’re seeing in the bottom left corner of gnuradio. I take those bursts and combine them into one and run through octave and get the frames. I then run through python script for results.
I would have to check the real time output vs the info I get from the bursts in temp, but I can say the basic process shown in the few videos works in latest DragonOS (has the newer dji droneid and gr-droneid). Check and see if you’re getting bursts in /tmp and then use those with octave to verify against the live output from gnuradio.

 Instead of running each burst one by way through the octave script which I have to edit manually, I instead took the burst_* files and then (I’m having a memory lapse) I think just like this is fine..
cat file1 file2 > file3
but do all the burst files. Someone correct me if I’m wrong, then I configure the script file per the video pointing it at the one large file I just made. At the end, if all goes well, it should say found FRAME xxx zzz and have lots info. I then copy and paste the info into a text file per number 3 of 3 videos on YouTube and use the python script inside DragonOS to get the decoded results.

@alphafox02 Thank's for your answer it helped me. I have just understood what's my problem. I have Mavic Air 2S and I'm using it for my tests. But I'm not sure that it has Ocusync 2 protocol. What can you tell about this?
And one more question. What about Mavic 3? I can't get drone_id using this tools?


Arrow_builder_size

[ ] I added an entry to CHANGELOG.md if knowledge of this change could be valuable to users.

Here is a brief summary of what I did:

 Arrow Builder Byte Size wird nicht richtig berechnet


[3.x] Only rotate Android sensor values for sensors that need them rotated

3.x version of #55183.

I've added the missing break statements and added the validation of the float values[] parameter.

Thanks!


Update content in candidate mailers about deferrals

Context
We noticed that when a candidate receives an offer email, and emails after they have accepted an offer, half the email content talks about deferring the offer. we don't want to encourage candidates to do this, so we've simplified the content to make it shorter. We'll still tell candidates about the option of deferring, but it now doesn't take up half the email.
We also made a change to a page within the interface where the content is the same as the emails.
Changes proposed in this pull request
Updating content in new offer emails and accepted offer emails
Updating content once a candidate receives an offer before they accept it
Link to Trello card
https://trello.com/c/lIsLoOXQ/1003-review-our-offer-emails-to-try-and-remove-encouragement-for-deferrals
Things to check

[ ] If the code removes any existing feature flags, a data migration has also been added to delete the entry from the database
[ ] This code does not rely on migrations in the same Pull Request
[ ] If this code includes a migration adding or changing columns, it also backfills existing records for consistency
[ ] If this code adds a column to the DB, decide whether it needs to be in analytics yml file or analytics blocklist
[ ] API release notes have been updated if necessary
[ ] Required environment variables have been updated added to the Azure KeyVault

 Review our offer emails to try and remove encouragement for deferrals


tf-nightly-2.0-preview failed to install on my Windows 7/64

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template
System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 7/64
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): pip
TensorFlow version: 2.0
Python version: 3.6
Installed using virtualenv? pip? conda?: pip
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:

Describe the problem
Provide the exact sequence of commands / steps that you executed before running into the problem
Any other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

For those who can't upgrade their OS (company policies, ...) and still want to access nightly previews (issue has been solved in alpha version pip install tensorflow==2.0.0-alpha0):

Download the latest wheel matching their configuration from pypi:
-- GPU version here; https://pypi.org/project/tf-nightly-gpu-2.0-preview/#history
-- CPU version here: https://pypi.org/project/tf-nightly-2.0-preview/#history
Open the whl with a zip application (like 7-zip)
Edit the 2 folders names at root level of the archive to remove both the "_preview" and "dev2019XXXX." string in their names
Extract the RECORD file in the root subfolder ending in .dist-info
Edit the extracted RECORD file and remove all "_preview" and "dev2019XXXX." strings
Put the modified RECORD file back into the archive
Install the wheel with pip using command pip install <path_to_modified.whl>

@IkabongoCS thanks for the update and the workaround. After following your instructions, however, I keep running into a directory error:
C:\Users\asus>pip install "C:\Users\asus\Downloads\tf_nightly_gpu_2.0_preview-2.0.0.dev20190327-cp36-cp36m-win_amd64.whl"
Processing c:\users\asus\downloads\tf_nightly_gpu_2.0_preview-2.0.0.dev20190327-cp36-cp36m-win_amd64.whl
Requirement already satisfied: protobuf>=3.6.1 in c:\users\asus\anaconda3\lib\site-packages (from tf-nightly-gpu-2.0-preview==2.0.0.dev20190327) (3.6.1)
Requirement already satisfied: six>=1.10.0 in c:\users\asus\anaconda3\lib\site-packages (from tf-nightly-gpu-2.0-preview==2.0.0.dev20190327) (1.10.0)
Requirement already satisfied: google-pasta>=0.1.2 in c:\users\asus\anaconda3\lib\site-packages (from tf-nightly-gpu-2.0-preview==2.0.0.dev20190327) (0.1.4)
Requirement already satisfied: numpy<2.0,>=1.14.5 in c:\users\asus\anaconda3\lib\site-packages (from tf-nightly-gpu-2.0-preview==2.0.0.dev20190327) (1.15.4)
Requirement already satisfied: tb-nightly<1.15.0a0,>=1.14.0a0 in c:\users\asus\anaconda3\lib\site-packages (from tf-nightly-gpu-2.0-preview==2.0.0.dev20190327) (1.14.0a20190301)
Requirement already satisfied: keras-applications>=1.0.6 in c:\users\asus\appdata\roaming\python\python36\site-packages (from tf-nightly-gpu-2.0-preview==2.0.0.dev20190327) (1.0.6)
Requirement already satisfied: termcolor>=1.1.0 in c:\users\asus\anaconda3\lib\site-packages (from tf-nightly-gpu-2.0-preview==2.0.0.dev20190327) (1.1.0)
Collecting tensorflow-estimator-2.0-preview (from tf-nightly-gpu-2.0-preview==2.0.0.dev20190327)
  Downloading https://files.pythonhosted.org/packages/ed/d2/7d9931a45b1085c5f74132fd6178982fc8481ee10e2016a7973696da0c92/tensorflow_estimator_2.0_preview-1.14.0.dev2019040100-py2.py3-none-any.whl (352kB)
    100% |████████████████████████████████| 358kB 1.2MB/s
Requirement already satisfied: grpcio>=1.8.6 in c:\users\asus\anaconda3\lib\site-packages (from tf-nightly-gpu-2.0-preview==2.0.0.dev20190327) (1.16.1)
Requirement already satisfied: astor>=0.6.0 in c:\users\asus\anaconda3\lib\site-packages (from tf-nightly-gpu-2.0-preview==2.0.0.dev20190327) (0.7.1)
Requirement already satisfied: absl-py>=0.7.0 in c:\users\asus\anaconda3\lib\site-packages (from tf-nightly-gpu-2.0-preview==2.0.0.dev20190327) (0.7.1)
Requirement already satisfied: gast>=0.2.0 in c:\users\asus\anaconda3\lib\site-packages (from tf-nightly-gpu-2.0-preview==2.0.0.dev20190327) (0.2.2)
Requirement already satisfied: wheel>=0.26 in c:\users\asus\anaconda3\lib\site-packages (from tf-nightly-gpu-2.0-preview==2.0.0.dev20190327) (0.29.0)
Requirement already satisfied: keras-preprocessing>=1.0.5 in c:\users\asus\appdata\roaming\python\python36\site-packages (from tf-nightly-gpu-2.0-preview==2.0.0.dev20190327) (1.0.5)
Requirement already satisfied: setuptools in c:\users\asus\appdata\roaming\python\python36\site-packages (from protobuf>=3.6.1->tf-nightly-gpu-2.0-preview==2.0.0.dev20190327) (39.1.0)
Requirement already satisfied: markdown>=2.6.8 in c:\users\asus\anaconda3\lib\site-packages (from tb-nightly<1.15.0a0,>=1.14.0a0->tf-nightly-gpu-2.0-preview==2.0.0.dev20190327) (3.0.1)
Requirement already satisfied: werkzeug>=0.11.15 in c:\users\asus\anaconda3\lib\site-packages (from tb-nightly<1.15.0a0,>=1.14.0a0->tf-nightly-gpu-2.0-preview==2.0.0.dev20190327) (0.12.2)
Requirement already satisfied: h5py in c:\users\asus\anaconda3\lib\site-packages (from keras-applications>=1.0.6->tf-nightly-gpu-2.0-preview==2.0.0.dev20190327) (2.8.0)
Installing collected packages: tensorflow-estimator-2.0-preview, tf-nightly-gpu-2.0-preview
Exception:
Traceback (most recent call last):
  File "c:\users\asus\anaconda3\lib\site-packages\pip\_internal\cli\base_command.py", line 179, in main
    status = self.run(options, args)
  File "c:\users\asus\anaconda3\lib\site-packages\pip\_internal\commands\install.py", line 393, in run
    use_user_site=options.use_user_site,
  File "c:\users\asus\anaconda3\lib\site-packages\pip\_internal\req\__init__.py", line 57, in install_given_reqs
    **kwargs
  File "c:\users\asus\anaconda3\lib\site-packages\pip\_internal\req\req_install.py", line 913, in install
    use_user_site=use_user_site, pycompile=pycompile,
  File "c:\users\asus\anaconda3\lib\site-packages\pip\_internal\req\req_install.py", line 445, in move_wheel_files
    warn_script_location=warn_script_location,
  File "c:\users\asus\anaconda3\lib\site-packages\pip\_internal\wheel.py", line 428, in move_wheel_files
    assert info_dir, "%s .dist-info directory not found" % req
AssertionError: tf-nightly-gpu-2.0-preview==2.0.0.dev20190327 .dist-info directory not found

C:\Users\asus>pip install "C:\Users\asus\Downloads\tf_nightly_gpu_2.0-cp36-cp36m-win_amd64.whl"
tf_nightly_gpu_2.0-cp36-cp36m-win_amd64.whl is not a valid wheel filename.

C:\Users\asus>pip install "C:\Users\asus\Downloads\tf_nightly_gpu_2.0-cp36-cp36m-win_amd64.whl"
tf_nightly_gpu_2.0-cp36-cp36m-win_amd64.whl is not a valid wheel filename.

C:\Users\asus>cd Downloads

C:\Users\asus\Downloads>pip install tf_nightly_gpu_2.0-cp36-cp36m-win_amd64.whl
tf_nightly_gpu_2.0-cp36-cp36m-win_amd64.whl is not a valid wheel filename.

C:\Users\asus\Downloads>pip install tf_nightly_gpu_2.0.whl6
Collecting tf_nightly_gpu_2.0.whl6
  Could not find a version that satisfies the requirement tf_nightly_gpu_2.0.whl6 (from versions: )
No matching distribution found for tf_nightly_gpu_2.0.whl6

C:\Users\asus\Downloads>pip install tf_nightly_gpu_2.0.whl
tf_nightly_gpu_2.0.whl is not a valid wheel filename.

Hi @dorian821 , to fix that issue is that you may not rename the .dist-info inside the wheel and also you may not rename the wheel filename. So the places to remove "_preview" and "dev2019XXXX" is the folder (not .dist-info) and the RECORD file.

Hi @aaron-maslim, :+1: thanks a lot! it install just fine!

Great to hear that!

Hi @dorian821 , to fix that issue is that you may not rename the .dist-info inside the wheel and also you may not rename the wheel filename. So the places to remove "_preview" and "dev2019XXXX" is the folder (not .dist-info) and the RECORD file. I am in Windows 7 as well btw.

Hi, I did as instructed,
i only renamed the folders
and then edited the RECORD file without changing the file name..
i'm still getting the error that the file name is still too long
Could not install packages due to an EnvironmentError: [WinError 206] The filename or extension is too long: 'C:\\Users\\-----------------\\AppData\\Local\\Temp\\pip-install-s_wy_wb_\\tf-nightly-2.0-preview\\tf_nightly_2.0_preview-2.0.0.dev20190411-cp36-cp36m-win_amd64/tf_nightly_2.0-2.0.0.data/purelib/tensorflow/include/external/absl_py/absl/third_party/unittest3_backport/

any other suggestions?
thanks

@RadEdje Hi, I resolved this problem by only renaming the data folder and the file paths in the RECORD file. That is, I left the dist-info folder as it is, unchanged.

I would suggest also renaming the wheel file to make it shorter (only remove "_preview" and "dev2019XXXX."), without removing other essential information (https://stackoverflow.com/questions/33213430/whl-is-not-a-valid-wheel-filename-storing-debug-log-for-failure-in-c).

@RadEdje Hi, I resolved this problem by only renaming the data folder and the file paths in the RECORD file. That is, I left the dist-info folder as it is, unchanged.
In your response, you mention that you renamed the folders, but you should only rename one folder, the data folder.

Hi. thanks for the suggestion.
i tried just renaming the data folder.
same thing.
the file path name is still too long.
Could not install packages due to an EnvironmentError: [WinError 206] The filename or extension is too long: 'C:\\Users\\------------------------\\AppData\\Local\\Temp\\pip-install-v3dk57w_\\tf-nightly-2.0-preview\\tf_nightly_2.0_preview-2.0.0.dev20190411-cp36-cp36m-win_amd64/tf_nightly_2.0-2.0.0.data/purelib/tensorflow/include/external/absl_py/absl/third_party/unittest3_backport/'

Thanks for the help.
is there anything else i can remove to make the filepath shorter?
I'm on windows 8.1.
python 3.6
using 7zip
and Microsoft VS code to replace/remove all the instances of _preview and the dev numbers in the RECORD file.

for clarity, what exactly have you renamed?
Also, since you're on windows 8, you should be able to extend the file path limit to 30k+ by changing the relevant key in the registry as discussed in the thread here:
https://github.com/tensorflow/tensorflow/issues/24835

for clarity, what exactly have you renamed?
Also, since you're on windows 8, you should be able to extend the file path limit to 30k+ by changing the relevant key in the registry as discussed in the thread here:
#24835

Oh... I thought that was only for windows 10.
I'll back read.
anyway here are the renamed folder (singular now as you suggested i only renamed the data folder):
tf_nightly_2.0-2.0.0.data

Using VS code I replaced/removed all of the instances of
"_preview" and "dev2019XXXX." strings in the RECORD file but I did not change the file name.
when the
file path name is too long error is no longer seen. A new error shows up.
That's the error that says the
that a file does not exist.
I think it's in the formatting...
note how the slashes change from
slash / to back slash \\
just my two cents.
Anyway i'll try to change the registry in windows 8.

I've just tested renaming (removed -preview and .devXXXX) the wheel file, entries in RECORD, the data folder and the dist-info folder, installation works for me.

ok, great. thanks for clarifying. I had the same thought regarding the slash, but it won't be an issue once the wheel is set up properly.
it looks like you might have grabbed a trailing period in the RECORD file,
the paths should go from:
tf_nightly_2.0_preview-2.0.0.dev20190401.data/purelib/
to
tf_nightly_gpu_2.0.data/purelib/
with a period separating 2.0 and data
check to make sure that this is the case.

ok, great. thanks for clarifying. I had the same thought regarding the slash, but it won't be an issue once the wheel is set up properly.
it looks like you might have grabbed a trailing period in the RECORD file,
the paths should go from:
tf_nightly_2.0_preview-2.0.0.dev20190401.data/purelib/
to
tf_nightly_gpu_2.0.data/purelib/
with a period separating 2.0 and data
it looks like you took "dev2019XXXX." and so may have "tf_nightly_gpu_2.0data/purelib/", which is incorrect
check to make sure that this is the case.

Hi did as you asked,
unfortuantely same error. the file does not exist:
Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: 'C:\\Users\\----------------------\\AppData\\Local\\Temp\\pip-install-eluview2\\tf-nightly-2.0-preview\\tf_nightly_2.0_preview-2.0.0.dev20190411-cp36-cp36m-win_amd64/tf_nightly_2.0.data/purelib/tensorflow/include/external/com_google_absl/absl/base/internal/low_level_scheduling.h'

this is the untouched name of the whl file
tf_nightly_2.0_preview-2.0.0.dev20190411-cp36-cp36m-win_amd64.whl

here is the name of the DATA folder
tf_nightly_2.0.data

here is a sample of a string line in the RECORD file (RECORD FILE name also untouched)
this is also the file that's allegedly missing in the error log
tf_nightly_2.0.data/purelib/tensorflow/include/external/com_google_absl/absl/base/internal/low_level_scheduling.h,sha256=5x8MD_0rHwmOE3L-IeK-B4bbMU0yJtCNHHWzpOB3Zt0,3985

Any suggestions?
thanks again for your patience and help.

@RadEdje  It looks like you removed too many things:

DATA folder, I have : tf_nightly_2.0-2.0.0.data
RECORD sample, I have: tf_nightly_2.0-2.0.0.data/purelib/tensorflow/init.py,sha256=4cVbFxbZBU5CbfDK9gA9BVyoEcDwwrs93aZrX0pMASE,18489

@RadEdje It looks like you removed too many things:
DATA folder, I have : tf_nightly_2.0-2.0.0.data
RECORD sample, I have: tf_nightly_2.0-2.0.0.data/purelib/tensorflow/init.py,sha256=4cVbFxbZBU5CbfDK9gA9BVyoEcDwwrs93aZrX0pMASE,18489
Hi, that was my original format.
That did not work either.
:-(

just wanted to update.
i gave up on windows 8.1
dual booted with ubuntu.
Everything installed and worked flawlessly without a hitch on the linux machine just with the pip install commands. Never found out what I was doing wrong on windows though.


Set successful in lambda release event

Description
Noticed it wasn't set when writing an integration test in dockstore-deploy. Then, in looking at it, I'd essentially duplicated an existing method, so I got rid of it.
Review Instructions

Do a GitHub release for a repo that is configured in Dockstore QA
Look at the app logs in a couple of minutes for the repo's org, and ensure that there is a release event, and that it is successful.

Issue
SEAB-6466
Security and Privacy
If there are any concerns that require extra attention from the security team, highlight them here and check the box when complete.

[x] Security and Privacy assessed

e.g. Does this change...

Any user data we collect, or data location?
Access control, authentication or authorization?
Encryption features?

Please make sure that you've checked the following before submitting your pull request. Thanks!

[x] Check that you pass the basic style checks and unit tests by running mvn clean install
[x] Ensure that the PR targets the correct branch. Check the milestone or fix version of the ticket.
[x] Follow the existing JPA patterns for queries, using named parameters, to avoid SQL injection
[x] If you are changing dependencies, check the Snyk status check or the dashboard to ensure you are not introducing new high/critical vulnerabilities
[x] Assume that inputs to the API can be malicious, and sanitize and/or check for Denial of Service type values, e.g., massive sizes
[x] Do not serve user-uploaded binary images through the Dockstore API
[x] Ensure that endpoints that only allow privileged access enforce that with the @RolesAllowed annotation
[x] Do not create cookies, although this may change in the future
[x] If this PR is for a user-facing feature, create and link a documentation ticket for this feature (usually in the same milestone as the linked issue). Style points if you create a documentation PR directly and link that instead.

failing test may be relevant


Document installation process on windows

psql2csv is listed as a requirement and is available on Homebrew .
I am currently on a windows machine.
Homebrew 2.0.0 documentation indicates that homebrew officially supports Linux and Windows 10 with Windows Subsystem for Linux (WSL).
Do you think this is the way to go and if so do you think that the other requirements - bcfishobs, bcdata and pgdata should be loaded into a conda env that is loaded inside of a WSL as per https://gist.github.com/kauffmanes/5e74916617f9993bc3479f401dfec7da ?

A few things:

this repo is a work in progress, requirements list isn't complete or checked
psql2csv is not essential, I find it handy for quick reporting
I think WSL would be a good way to go for these scripts on windows, then you wouldn't have to modify them then (although modifications should mostly just be the line ending characters)
I haven't tried homebrew on WSL, but that is a good way to grab any other requirements not on the system (like wget)
conda probably isn't necessary if using WSL, I think rasterio/gdal/geopandas should all just work with pip - but using a conda env is a good idea

Looks like I can't easily test WSL. My windows install is on a VM, running the linux VM inside the windows VM might be possible but takes some setup - https://github.com/microsoft/WSL/issues/4120

This seems to work fine for getting started on a fresh WSL (Ubuntu 20.04.1 LTS). Not sure I would bother installing homebrew, conda should do the job - just ignore the psql2csv requirement for now.
sudo apt update
sudo apt upgrade
sudo apt install python3 python3-pip
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
# follow instructions, accept terms, restart terminal

conda create -n geo_env
conda activate geo_env
conda config --env --add channels conda-forge
conda config --env --set channel_priority strict
conda install gdal
conda install geopandas
conda install rasterio
pip3 install bcdata


Handle "help"

Goal
The only not user defined Litr call should be litr --help to show the help text. The help text should be as user friendly as possible and give some meaningful information even on a minimal example.
Open questions

What if the user defines a help parameter?
Should it be possible for the user to define its own help parameter?

One possible solution could be to restrict usage of the help parameter in general, giving an error message to the user that "help is not allowed to specify" if it is used. This error message should give context where it is, why it is not allowed and how users can influence the generate help text (e.g. using the example property on commands).

Went for the "restrict certain parameter names" and documented here: https://github.com/krieselreihe/litr/wiki/Setup#restricted-parameters


master版的XA mode demo几乎不能运行，请修正

我运行至少遇到这几个问题：
1 ：bussiness默认AT mode而不是XA mode
2：order项目缺少controller文件，以致于bussniss找不到rpc create方法
3：storage项目显示xid为null：
2024-04-15 21:41:57.315  INFO 24345 --- [nio-8081-exec-1] org.apache.seata.service.StorageService  : Stock Service Begin ... xid: null
2024-04-15 21:41:57.315  INFO 24345 --- [nio-8081-exec-1] org.apache.seata.service.StorageService  : Deducting inventory SQL: update stock_tbl set count = count - 2 where commodity_code = C00321
2024-04-15 21:41:57.322  INFO 24345 --- [nio-8081-exec-1] org.apache.seata.service.StorageService  : Stock Service End ...
^@^@^@^@^@2024-04-15 21:46:23.923  INFO 24345 --- [nio-8081-exec-3] org.apache.seata.service.StorageService  : Stock Service Begin ... xid: null
2024-04-15 21:46:23.924  INFO 24345 --- [nio-8081-exec-3] org.apache.seata.service.StorageService  : Deducting inventory SQL: update stock_tbl set count = count - 2 where commodity_code = C00321
2024-04-15 21:46:23.925  INFO 24345 --- [nio-8081-exec-3] org.apache.seata.service.StorageService  : Stock Service End ...
最终结果是：purchase运行失败，但是库存却减少了：
mysql> select * from stock_tbl;
+----+----------------+-------+
| id | commodity_code | count |
+----+----------------+-------+
| 24 | C00321         |    98 |
+----+----------------+-------+

BussinessService is just an action trigger. What it does is to wrap RPC calls with @GlobalTransactional. In that case, there is no need for special configuration in XA mode.
You are right. We need to fix it.
You are right. AccountService, OrderSevice and StorageService are not binding the global transaction

Thank you for raising it. I would fix those issues.

为这响应速度点赞！

还有个小问题，配置中的：spring.datasource.url=jdbc:mysql://<IP_ADDRESS>:3306/seata?userSSL=false&useUnicode=true&characterEncoding=UTF8


add easily enabled/configured PrecachedDocumentProvider implementation

Pull Request type

[ ] Bugfix
[x] Feature
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] Other (please describe):

Changes in this PR
Adds a PrecachedDocumentProvider implementation that is backed by a Caffeine cache. This often shaves a significant time off of parsing common query documents.
Can be enabled by setting the dgs.graphql.preparsedDocumentProvider.enabled property to true.

see also: https://github.com/Netflix/dgs/pull/160


llama 3 8b model with A10 GPU, OOM with VLLM, but holds good on HF transformer pipline

Hi,
GPU : A10 24 GB
Model size with safe tensors : 26 GB all together
With HF pipeline, it was possible to load llama3 8b and then convert it too fp16 and run inference
but with VLLM, when I try to load the model itself, it goes OOM,
can this be understood that vllm requires a more memory at loading pace and not possible to handle this,
only after applying optimizations w.r.t model weights (quantization) will I be able to infer llama 3 8b?
blogs like this are kind of misleading with it
https://neuralmagic.com/blog/deploy-llama-3-8b-with-vllm/
          With int4 quantized models you can use parameter `--gpu-memory-utilization` that take up less GPU memory, for Llama3-8B, this parameter can be set to a minimum of 0.6(~12GB) with 4-bit quantization, but not with the original fp16 model(~19GB)!

Originally posted by @yangxianpku in https://github.com/vllm-project/vllm/issues/2948#issuecomment-2122090059

L3-8B is ~16GB not 26GB ... so it should comfortably fit in an A10G gpu
What command are you using to launch vllm?

yes, L2-8B in fp16 is around 16 GB, but with FP32 it is around 26GB in size.
it is stored in safe tensors.
vllm load code
engine_args = EngineArgs(
    model=PATH,
    trust_remote_code=True,
    tensor_parallel_size=1,
    trust_remote_code=True,
    load_format="auto",
    dtype = "float16",
)
self.engine = LLMEngine.from_engine_args(engine_args)

HF transformer lib

self.pipeline = transformers.pipeline(
    "text-generation",  # task
    model=model_uri,
    tokenizer=self.tokenizer,
    trust_remote_code=True,
    device_map="auto",
    torch_dtype=torch.bfloat16, # or fp16 anything at this stage
    max_length=1024,
    max_new_token=512,
    do_sample=True,
    top_k=10,
    num_return_sequences=1,
    eos_token_id=self.tokenizer.eos_token_id,
)

when the model is exported to fp16, then VLLM is able to load and handle it
but with fp32 loading directly it is failing,
hf with accelerate lib is able to handle it I think somehow managing the memory
or is it that I am missing something with VLLM params?

looks fine. can you post the error that you see?

the log

log :

INFO 07-26 15:21:31 config.py:1151] Downcasting torch.float32 to torch.float16.
INFO 07-26 15:21:31 llm_engine.py:161] Initializing an LLM engine (v0.4.3) with config: model='/app/model/', speculative_config=None, tokenizer='/app/model/',
skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_s
eq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False,
kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_mode
l_name=/app/model/)
[2024-07-26 15:21:36 +0000] [109] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/gunicorn/arbiter.py", line 589, in spawn_worker
worker.init_process()
File "/usr/local/lib/python3.8/dist-packages/gunicorn/workers/base.py", line 134, in init_process
self.load_wsgi()
File "/usr/local/lib/python3.8/dist-packages/gunicorn/workers/base.py", line 146, in load_wsgi
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python3.8/dist-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/usr/local/lib/python3.8/dist-packages/seldon_core/gunicorn_utils.py", line 107, in load
self.user_object.load()
File "/app/ZeroShotSlotsV0vllm.py", line 38, in load
self.engine = LLMEngine.from_engine_args(engine_args)
File "/usr/local/lib/python3.8/dist-packages/vllm/engine/llm_engine.py", line 359, in from_engine_args
engine = cls(
File "/usr/local/lib/python3.8/dist-packages/vllm/engine/llm_engine.py", line 222, in init
self.model_executor = executor_class(
File "/usr/local/lib/python3.8/dist-packages/vllm/executor/executor_base.py", line 41, in init
self._init_executor()
File "/usr/local/lib/python3.8/dist-packages/vllm/executor/gpu_executor.py", line 24, in _init_executor
self.driver_worker.load_model()
File "/usr/local/lib/python3.8/dist-packages/vllm/worker/worker.py", line 121, in load_model
self.model_runner.load_model()
File "/usr/local/lib/python3.8/dist-packages/vllm/worker/model_runner.py", line 134, in load_model
self.model = get_model(
File "/usr/local/lib/python3.8/dist-packages/vllm/model_executor/model_loader/init.py", line 21, in get_model
return loader.load_model(model_config=model_config,
File "/usr/local/lib/python3.8/dist-packages/vllm/model_executor/model_loader/loader.py", line 240, in load_model
model = _initialize_model(model_config, self.load_config,
File "/usr/local/lib/python3.8/dist-packages/vllm/model_executor/model_loader/loader.py", line 91, in _initialize_model
return model_class(config=model_config.hf_config,
File "/usr/local/lib/python3.8/dist-packages/vllm/model_executor/models/llama.py", line 332, in init
self.model = LlamaModel(config,
File "/usr/local/lib/python3.8/dist-packages/vllm/model_executor/models/llama.py", line 262, in init
self.layers = nn.ModuleList([
File "/usr/local/lib/python3.8/dist-packages/vllm/model_executor/models/llama.py", line 263, in
LlamaDecoderLayer(config=config,
File "/usr/local/lib/python3.8/dist-packages/vllm/model_executor/models/llama.py", line 200, in init
self.mlp = LlamaMLP(
File "/usr/local/lib/python3.8/dist-packages/vllm/model_executor/models/llama.py", line 69, in init
self.down_proj = RowParallelLinear(input_size=intermediate_size,
File "/usr/local/lib/python3.8/dist-packages/vllm/model_executor/layers/linear.py", line 707, in init
self.quant_method.create_weights(
File "/usr/local/lib/python3.8/dist-packages/vllm/model_executor/layers/linear.py", line 79, in create_weights
weight = Parameter(torch.empty(sum(output_partition_sizes),
File "/usr/local/lib/python3.8/dist-packages/torch/utils/_device.py", line 78, in torch_function
return func(*args, **kwargs)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 112.00 MiB. GPU
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/multiprocess/util.py", line 357, in _exit_function
p.join()
File "/usr/local/lib/python3.8/dist-packages/multiprocess/process.py", line 147, in join
assert self._parent_pid == os.getpid(), 'can only join a child process'
AssertionError: can only join a child process

Observation : I am running this in docker, but when I try to run it locally directly
Sometimes, very rarely, it is able to start the load process again and able to load it

Do you have anything running on your GPU already?

when the model is exported to fp16, then VLLM is able to load and handle it
but with fp32 loading directly it is failing

I think this is expected on vLLM? You're trying to load an FP32 model that itself will take more GPU memory than the capacity by default. You could try enable offloading to CPU and see if that helps

Do you have anything running on your GPU already?

yes, it is the only process running

when the model is exported to fp16, then VLLM is able to load and handle it
but with fp32 loading directly it is failing

I think this is expected on vLLM? You're trying to load an FP32 model that itself will take more GPU memory than the device's capacity by default. You could try enable offloading to CPU and see if that helps

thanks, I will try CPU Offloading , maybe that is what I am looking for
you are right about the sizes, but it is more towards the efficient loading capability,
there are 2 factors here

Model I have is in fp32
Loading them as fp16, so after loading memory will not be 26 GB but 16 GB is my expectation,

point 2 is observed with HF pipeline.
it just points out that VLLM can become much more efficient in handling this case
Anyways I converted the model to fp16 and was able to load it safely
Thanks for the support

Apologies, even with fp16 model I am facing this issue with VLLM on A10 with 24 GB
I am running it with docker
I experimented with few variables to handle this but they are of no use as of now
but of no use
max_parallel_loading_workers
gpu_memory_utilization


Create checkpoint before it is too late

Pre-trained model should be created as soon as possible.

Done and works nicely. Closing.


Bug: searchcount error on refresh after #958

Self Checks

[X] I'm using the latest lualine.
[X] I didn't find the issue in existing issues or PRs.

How to reproduce the problem
nvim starts with error after this PR #958.
It was working fine without errors before that.
It appears that refresh() method is now throwing an error.
The reason refresh() is called it is to avoid unwanted flashing in the tabline.
Expected behaviour
Start neovim without error
Actual behaviour
Receiving the following error when starting neovim
packer.nvim: Error running config for lualine.nvim: ...e/nvim/site/pack/packer/opt/lualine.nvim/lua/lualine.lua:432: Error executing lua: .../opt/lualine.nvim/lua/lualine/components/searchcount.lua:23: bad argumen
t #1 to 'min' (number expected, got nil)
stack traceback:
^I[C]: in function 'min'
^I.../opt/lualine.nvim/lua/lualine/components/searchcount.lua:23: in function 'update_status'
^I...e/pack/packer/opt/lualine.nvim/lua/lualine/component.lua:273: in function 'draw'
^I...ck/packer/opt/lualine.nvim/lua/lualine/utils/section.lua:26: in function 'draw_section'
^I...e/nvim/site/pack/packer/opt/lualine.nvim/lua/lualine.lua:173: in function 'statusline'
^I...e/nvim/site/pack/packer/opt/lualine.nvim/lua/lualine.lua:301: in function <...e/nvim/site/pack/packer/opt/lualine.nvim/lua/lualine.lua:282>
^I[C]: in function 'nvim_win_call'
^I...e/nvim/site/pack/packer/opt/lualine.nvim/lua/lualine.lua:432: in function 'refresh'
^I[string "..."]: in function <[string "..."]:0>
^I[C]: in function 'pcall'
^I/home/my-user/.config/nvim/plugin/packer_compiled.lua:65: in function 'try_loadstring'
^I/home/my-user/.config/nvim/plugin/packer_compiled.lua:323: in function </home/my-user/.config/nvim/plugin/packer_compiled.lua:10>
^I[C]: in function 'pcall'

Minimal config to reproduce the issue
-- statusline
use {
  "nvim-lualine/lualine.nvim", -- statusline plugin
  after = "zephyr-nvim", -- init after color scheme in order to prevent any overrides
  requires = {
    { "tiagovla/scope.nvim" }, -- scope buffers to tabs
    { "kyazdani42/nvim-web-devicons", opt = true }, -- use dev icons
    { "whoissethdaniel/lualine-lsp-progress.nvim" }, -- display lsp progress
  },
  config = function()
    require("scope").setup()

    require("lualine").setup({
      options = {
        theme = "onedark",
        icons_enabled = true,
        section_separators = "", -- disable separators
        component_separators = "", -- disable separators
      },
      sections = {
        lualine_a = { "mode" },
        lualine_b = { "branch", "diff", "diagnostics" },
        lualine_c = { "filename", "filesize", "lsp_progress" },
        lualine_x = { "searchcount", "encoding", "fileformat", "filetype" },
        lualine_y = { "progress" },
        lualine_z = { "location" }
      },
      tabline = {
        lualine_a = {{
          "buffers",
          mode = 0, -- shows only buffer name
          icons_enabled = false, -- disable icons for tabline
          show_filename_only = true,
          show_modified_status = true,
          hide_filename_extension = false,
          max_length = vim.o.columns, -- maximum width of buffers component
          symbols = {
            modified = " ^", -- buffer is modified
            directory = "", -- buffer is a directory
            alternate_file = "#", -- text for alternate file
          },
          filetype_names = {
            fzf = "FZF",
            alpha = "Alpha",
            packer = "Packer",
            NvimTree = "NvimTree",
            dashboard = "Dashboard",
            TelescopePrompt = "Telescope",
          },
        }},
        lualine_b = {},
        lualine_c = {},
        lualine_x = {},
        lualine_y = {},
        lualine_z = { "tabs" }
      }
    })

    require("lualine").refresh()
  end
}

This issue also happened to me, and I had to search any string using / to fix it.
Example: /test
After searching, I restarted Neovim and now the error is not thrown.

My lualine config

Nice, Thanks!
I confirm it works now as expected after the update.


How to use local Medusa head?

It is said that Medusa can significantly accelerate inference speed. During my attempts to utilize it, I have observed that it does not support the use of local Medusa config and head. The code fragment I discovered that pertains to this functionality is as follows, which I have modified. However, I do not comprehend the meaning of 'medusa_sf'. The training process of Medusa does not generate new safetensors. What is this?
medusa_config = f"{model_id}/config_medusa.json"
# medusa_config = hf_hub_download(
#     use_medusa, revision=revision, filename="config.json"
# )
with open(medusa_config, "r") as f:
    config = json.load(f)
medusa_head = f"{model_id}/medusa_lm_head.pt"
# medusa_head = hf_hub_download(
#     use_medusa, revision=revision, filename="medusa_lm_head.pt"
# )
medusa_sf = medusa_head[: -len(".pt")] + ".safetensors"
weights = Weights(
    [medusa_sf], device, dtype, process_group=self.process_group
)
lm_head = model.lm_head
model.lm_head = MedusaModel(config, weights, lm_head)

How should I employ TGI to access the local Medusa? A huge thank for your work!

Linked to #1418  and #1420


cannot render custom component by literal.

Version
3.0.0-alpha.8
Reproduction link
Steps to reproduce

vue2 example
https://codepen.io/dewfall/pen/eYNVXEb

vue3

import { createApp, defineComponent, h } from 'vue'

const customComponent = defineComponent({
  props: {
    text: String,
  },
  setup(props) {
    return () =>
{props.text}

  },
})

createApp({
  components: {
    customComponent,
  },
  setup() {
    return () => h('customComponent', { text: 'this is a component!' })
    // return () => h(customComponent, { text: 'this is a component!' })
  },
}).mount('#app')

is this a broken change or a bug?
What is expected?
vue2 Rendering results:

this is a component!

What is actually happening?
<EMAIL_ADDRESS>Rendering results:
<customcomponent text="this is a component!"></customcomponent>

i'm trying to rewrite the jsx-plugin,
but found  h('custom-component') doesn't work.

It doesn't work with this?
setup() {
     return () => h(customComponent, { text: 'this is a component!' })
}

v3 no longer supports rendering component using strings, since when you are using render functions you can always use the direct value. This also enables better compiler optimizations.
If you have to use a global registered component in render functions, use resolveComponent:
import { resolveComponent } from 'vue'

// ...
const comp = resolveComponent('customComponent')
return h(comp)


youtube: -j --flat-playlist with channel different in CLI than return from API call

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.11.07. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

[x] I've verified and I assure that I'm running youtube-dl 2018.11.07

Before submitting an issue make sure you have:

[x ] At least skimmed through the README, most notably the FAQ and BUGS sections
[x ] Searched the bugtracker for similar issues including closed ones
[x ] Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

[x ] Bug report (encountered problems with youtube-dl)
[ ] Site support request (request for adding support for a new site)
[ ] Feature request (request for a new functionality)
[ ] Question
[ ] Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--flat-playlist', '-j', '-v', 'https://www.youtube.com/channel/UC6RNSPFcqY4BblL2Jg9SUtw']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.11.07
[debug] Python version 3.6.5 (CPython) - Linux-3.10.0-862.14.4.el7.x86_64-x86_64-with-centos-7.5.1804-Core
[debug] exe versions: ffmpeg 2.8.15, ffprobe 2.8.15
[debug] Proxy map: {}
{"_type": "url", "url": "IaNEfhPmhPM", "ie_key": "Youtube", "id": "IaNEfhPmhPM", "title": "Rona in Aurora - the death of a big box"}
{"_type": "url", "url": "JN-2vTf8-WM", "ie_key": "Youtube", "id": "JN-2vTf8-WM", "title": "RealWorldNumbers - How to Test a Raspberry Pi Powered by Alkaline AA Batteries - After Poweroff"}
{"_type": "url", "url": "NmQmM36ja3o", "ie_key": "Youtube", "id": "NmQmM36ja3o", "title": "RealWorldNumbers - How to Test a Raspberry Pi Powered by Alkaline AA Batteries - During the Test"}
{"_type": "url", "url": "MsrgW1Rdlso", "ie_key": "Youtube", "id": "MsrgW1Rdlso", "title": "Raspberry Pi powered by AA batteries direct"}
(bestpa) [bestpa@oraclevm vidbackup]$
---

Playlist: https://www.youtube.com/channel/UC6RNSPFcqY4BblL2Jg9SUtw

---

### Description of your *issue*, suggested solution and other information

I've got a problem whereby when I call  youtube-dl  from the CLI with "-j --flat-playlist" I get a list of videos, but when i make the same call using  the same arguments from an imported instance of the python library, it only presents a single object (instead of iterating through the list of videos).

I've opened up a discussion on stackoverflow, but no real explanation for the behaviour.
https://stackoverflow.com/questions/53288922/youtube-dl-dump-json-returning-different-extractor-output-for-playlist-when-ca

Here's the python code:

    import youtube_dl
    ydl_opts = {
        'extract_flat': True, ## --flat-playlist according to options.py
         'dumpjson': True, ## lower -j
         #'dump_single_json': True, ## UPPER -J
    }
    with youtube_dl.YoutubeDL(ydl_opts) as ydl:
        object = ydl.extract_info(providedURL, download=False)
        print (object)
        print len(object)
  print (youtube_dl.version.unicode_literals)

Her'es the output

[youtube:channel] UC6RNSPFcqY4BblL2Jg9SUtw: Downloading channel page
{'_type': 'url', 'url': 'https://www.youtube.com/playlist?list=UU6RNSPFcqY4BblL2Jg9SUtw', 'ie_key': 'YoutubePlaylist', 'extractor': 'youtube:channel', 'webpage_url': 'https://www.youtube.com/channel/UC6RNSPFcqY4BblL2Jg9SUtw', 'webpage_url_basename': 'UC6RNSPFcqY4BblL2Jg9SUtw', 'extractor_key': 'YoutubeChannel'}
7

Long shot, but wonder if this could be resurgence or related to a closed issue from 2015.  https://github.com/rg3/youtube-dl/issues/4971

I'd like to know more about the procedural different between the cli and using the python .extract_info method.  Is this because of the nature of the interactive shell when using the CLI method?

sorry my markup is terrible on this report.  I hope that you guys can read past it.  Late. Tired. Thanks.

Have you actually read the doc? You're getting exactly what requested.

    extract_flat:      Do not resolve URLs, return the immediate result.
                       Pass in 'in_playlist' to only show this behavior for
                       playlist items.

Pass in 'in_playlist' to only show this behavior for playlist items.

Passing True will forbid any recursive resolving including playlists (will forbid resolving channel to playlist url delegate transition in your case). in_playlist will forbid resolving but only at playlist level (allowing aforementioned transition), that produces info dict with the same list of videos under entries.
2. --dump-* options don't return anything, they are verbose options for printing to stdout. dump* API options have no effect on extract_info.


Trend Micro Vision One - TrendMicro_XDR_WORKBENCH_CL table is not showing any results

Describe the bug
The TrendMicro_XDR_WORKBENCH_CL, TrendMicro_XDR_RCA_Task_CL tables is not showing any results, while table TrendMicro_XDR_OAT_CL seem show some results.
To Reproduce
Steps to reproduce the behavior:
Go to Sentinel > Data connectors > Trend Vision One (using Azure Functions) > Open connector page
Check Data types
Only TrendMicro_XDR_OAT_CL shows green connection icon, other shows grey icon.
Go to Sentinel > Logs > New query
TrendMicro_XDR_OAT_CL shows results.
rendMicro_XDR_WORKBENCH_CL do not show any results.
TrendMicro_XDR_RCA_Task_CL - 'table' operator: Failed to resolve table expression named 'TrendMicro_XDR_RCA_Task_CL'
Request id: 326c8e42-3576-4142-9610-be08ab11227b
TrendMicro_XDR_RCA_Result_CL - 'table' operator: Failed to resolve table expression named 'TrendMicro_XDR_RCA_Result_CL'
Request id: 148180a7-32e3-44dc-b268-a889511f38c8
Expected behavior
The TrendMicro_XDR_WORKBENCH_CL, TrendMicro_XDR_RCA_Task_CL, TrendMicro_XDR_RCA_Result_CL tables should display relevant data as per normal operation.

Hi @vi-lgtm,
We also have checked this solution, this solution is supported with Trend Micro Vision One However, we are not authorized to make any modifications to it. We kindly request that you raise an issue ticket with the Trend Micro Vision One support team.
Sharing the Trend Micro Vision One support link -
Link - "https://success.trendmicro.com/dcx/s/?language=en_US"
Closing this issue from GitHub. If you still need support for this issue, feel free to re-open it any time. Thank you for your co-operation.


Syntax of Insert and Update of embedded document types are inconsistent

The following is reported on v2.0.7:
For embedded documents, the insert and update statements are different. While insert copes with a plain quasi-JSON object (single-quotes suffice, not necessarily double-quotes, thus quasi-JSON), does update not work with this plain JSON type. The additional specs @class and @type must be added. The following statements can be used to illustrate the issue:
create class Address
create property Address.street String
create property Address.city String

create class Client
create property Client.name String
create property Client.phones embeddedSet String
create property Client.addresses embeddedList Address

insert into client set name = 'James Bond', phones = ['1234', '34567'], addresses = [{'city':'Shanghai', 'zip':'3999'}, {'city':'New York', 'street':'57th Ave'}]
update client add addresses = [{'city':'London', 'zip':'67373'}] # not working
update client add addresses = [{'@type':'d','@class':'Address','city':'London', 'zip':'67373'}] # works

Hi @mamobyz.
I believe this was already reported here #3621.

Hi @vitorenesduarte, thanks for the hint!
They are probably too busy with other issues. As you reported this issue months ago, it is sometimes really frustrating to wait for at least an opinion from OrientDB. As @lvca stated, it was super easy to fix #3398, I truly believe that! I don't understand why they need so much time to not respond.
So I will leave also this one open, until someone from OrientDB respond.

@mamobyz
are you getting
com.orientechnologies.orient.core.exception.OValidationException: The field 'Client.addresses' has been declared as EMBEDDEDLIST but an incompatible type is used. Value: 67373

@maggiolo00: Yes, this is the error message.

I just tested this issue on 2.0.10. It still does not work. Isn't the fix already merged into 2.0.9?

And it does not work on 2.1-rc3 either. To be frank, it is really frustrating!

Hi @mamobyz
it's fixed in 2.1 RC3 but not in 2.0.10.
Are you sure you tried correctly with latest RC? I just tried it and it seems to work fine
Luigi

I found this issue as fixed, but when has been fixed? I can't find the relative commit.

@luigidellaquila: If the download file for rc3 provided on your Web page is really rc3, I guess I have got it tested on rc3!
@lvca: I hope, you are not waiting for my reply! I have no influence on your commits. If you have added milestone 2.0.9 and 2.1-rc2 (see above) and closed the ticket afterwards, I assume it is fixed on both milestones.
Sorry, it seems a bit chaotic there ...

mmm... ok, I'm trying it again...
@lvca good question, I tried it on the snapshot and it's OK (or so it seems, I'm checking it again)

Guys, I tried it again, just did this:

downloaded 2.1 RC3 from the website (usually I work on develop branch, but you never know...)
unzipped and launched (root password asked, so it's a new instance, no doubt)
created a new db (!!! this could be the point, are you working with an old db?)
in Studio, copied-pasted your statements and executed one by one

After every insert/update I did a select to check the actual content of the record, and it was ok...
Any thoughts?
Thanks

@luigidellaquila: Maybe you should test it on console. The console uses a different (de-)serializer, is it right. It could well be, that the REST interface adds the superfluous tags so that the final writes works, but it still replies on that the input entails these tags, i.e. @type, @class


fix(kubernetes): popover position

Set popover placement to auto so that it auto-adjusts for mobile view and does not get clipped
MBP-540

CDS Report web#4506.0 ✘
*

Build CA ✘
Build EU ✘
Build US ✘

CDS Report web#4507.0 ✘
*

Build CA ✘
Build EU ✘
Build US ✘


Web site no longer online

Whatever was at https://msdev.gointeract.io/interact/index?interaction=1461173234028-3884f8602eccbe259104553afa8415434b4581-05d1&accountId=microsoft&appkey=196de13c-e946-4531-98f6-2719ec8405ce
is no longer there.

You can try to use https://microsoft.gointeract.io/interact/index?interaction=1461173234028-3884f8602eccbe259104553afa8415434b4581-05d1&accountId=microsoft&appkey=196de13c-e946-4531-98f6-2719ec8405ce&Language=French&CountryCode=FR&CountryLanguage=FR&Click%20To%20Call%20Caller%20Id=+31641190689&startedFromSmsToken=QCYw2pA&sessionID=188781095&token=wXVhvi

You can try to use this link.

That link is now broken too. Has anybody managed to actually reverse engineer the XP activation algorithm and write their own keygen.exe for it?

I have no idea

Needed a working one, this one seems alive now.


Errors creating replicated tables

Hello all,
First of all, thanks for the driver, it's being really useful :).
We're running into a problem that I would like to describe here: we make use of the ClickHouse abilities to replicate tables and we're getting errors while we try to create the table in all the nodes of the cluster
Cluster
Our current setup is a ClickHouse cluster with 3 shards; every shard contains two replicas
    	<us-west-2>
        	<shard>
            	<internal_replication>true</internal_replication>
            	<replica>
                	<host>clickhouse-1-us-west-2</host>
                	<port>9000</port>
           		</replica>
            	<replica>
                	<host>clickhouse-2-us-west-2</host>
                	<port>9000</port>
           		</replica>
        	</shard>
        	<shard>
            	<internal_replication>true</internal_replication>
            	<replica>
                	<host>clickhouse-3-us-west-2</host>
                	<port>9000</port>
           		</replica>
            	<replica>
                	<host>clickhouse-4-us-west-2</host>
                	<port>9000</port>
           		</replica>
        	</shard>
        	<shard>
            	<internal_replication>true</internal_replication>
            	<replica>
                	<host>clickhouse-5-us-west-2</host>
                	<port>9000</port>
           		</replica>
            	<replica>
                	<host>clickhouse-6-us-west-2</host>
                	<port>9000</port>
           		</replica>
        	</shard>
    	</us-west-2>

Queries
We're trying to create a replicated table in all the six previous nodes (we cannot make use of the ON CLUSTER directive)
CREATE TABLE IF NOT EXISTS local_calls (
    trace_id String,
    call_id String,
    date Date MATERIALIZED toDate(t/1000),
    t UInt64,
    a UInt64,
    b String,
    c String
) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/local_calls', '{replica}') PARTITION BY toRelativeDayNum(date) ORDER BY t SETTINGS index_granularity=8192;

I am removing most of the columns for the sake of simplicity
Problem
When we try to run the query on every node, we get the following error:
(details: code: 122, message: Table structure in ZooKeeper is too different from local table structure)’

We get in every replica node: so if we run the queries in the following order: clickhouse-1 to clickhouse-6, we get the errors in nodes 2, 4 and 6
Analysis
I am not 100% sure, but, what I think it's happening is that the columns are being sent in a different order when the query is submitted to the second replica.
So, the first create statement is sent to clickhouse-1, the table is created and all the metadata is stored in Zookeeper, then, when the same query is sent to clickhouse-2, the columns are sent in a different order: the node will check the actual query statement against the metadata stored in Zookeeper (because it's a replicated table) an it will fail with an error like the one already described.
Does it make sense?
Please, let me know if I can help some how (in case this is a real issue :) )

Hi, I think it's a ClickHouse problem/bug. Try to create the table without a slash in default value (date Date MATERIALIZED toDate(t/1000) )

I will give it a try.
Just would like to add that creating the tables using regular HTTP requests works fine

It's very strange. I'll troubleshoot it

I've followed the same process using and HTTP Client and the command line (which uses the TCP protocol, doesn't it) and both works for me.
The only way I am able to reproduce the issue is using the driver

OK, I'll be back on Monday and fix it.

Hi @kshvakov,
do you have any news on this ticket? Where you able to reproduce the problem we encountered?
If there is anything we can do to help out, let us know.
Thanks,
Marcel

I had hoped this was an instance of a map being ranged over for columns being presented to the clickhouse server, however this does not seem to be the case :/

You can ignore this.
The issue was that we were creating a table on one replica and then running all the migration alter statements. The next create would then fail because the initial create is the original table minus the alter deltas.


Pantheon: session crash when try to resume from suspend

Describe the bug
session and all unsaved work gone
journal:
Mar 10 13:13:57 ethmoid avahi-daemon[907]: Interface wlp5s0.IPv4 no longer relevant for mDNS.
Mar 10 13:13:57 ethmoid systemd[1]: Started Network Manager Script Dispatcher Service.
Mar 10 13:13:57 ethmoid wpa_supplicant[1837]: nl80211: deinit ifname=p2p-dev-wlp5s0 disabled_11b_rates=0
Mar 10 13:13:57 ethmoid .io.elementary.[30107]: Source ID 9104 was not found when attempting to remove it
Mar 10 13:13:57 ethmoid .wingpanel-wrap[3895]: DisplayWidget.vala:159: Unknown network state, cannot show the good icon: NETWORK_STATE_FAILED_WIRED
Mar 10 13:13:57 ethmoid kernel: iwlwifi 0000:05:00.0: Applying debug destination EXTERNAL_DRAM
Mar 10 13:13:57 ethmoid wpa_supplicant[1837]: nl80211: deinit ifname=wlp5s0 disabled_11b_rates=0
Mar 10 13:13:57 ethmoid kernel: iwlwifi 0000:05:00.0: Applying debug destination EXTERNAL_DRAM
Mar 10 13:13:57 ethmoid .io.elementary.[22968]: Couldn't connect to accessibility bus: Failed to connect to socket /tmp/dbus-hiJWaewKws: Connection refused
Mar 10 13:13:57 ethmoid kernel: iwlwifi 0000:05:00.0: FW already configured (0) - re-configuring
Mar 10 13:13:57 ethmoid kernel: iwlwifi 0000:05:00.0: Applying debug destination EXTERNAL_DRAM
Mar 10 13:13:58 ethmoid kernel: iwlwifi 0000:05:00.0: Applying debug destination EXTERNAL_DRAM
Mar 10 13:13:58 ethmoid syncthing[3473]: [YWRW6] 13:13:58 INFO: TCP listener ([::]:22000) shutting down
Mar 10 13:13:58 ethmoid syncthing[3473]: [YWRW6] 13:13:58 INFO: Exiting
Mar 10 13:13:58 ethmoid systemd[1]: syncthing.service: Succeeded.
Mar 10 13:13:58 ethmoid systemd[1]: syncthing.service: Consumed 1h 4min 6.876s CPU time, no IP traffic.
Mar 10 13:13:58 ethmoid kernel: iwlwifi 0000:05:00.0: FW already configured (0) - re-configuring
Mar 10 13:13:58 ethmoid NetworkManager[912]: <warn>  [1615378438.1962] dns-sd-resolved[0060b31ecf51f570]: send-updates failed to update systemd-resolved: GDBus>
Mar 10 13:13:58 ethmoid io.elementary.wingpanel.desktop[22963]: Failed to register: GDBus.Error:org.freedesktop.DBus.Error.NoReply: Message recipient disconnec>
Mar 10 13:13:58 ethmoid .wingpanel-wrap[3895]: gtk_widget_destroy: assertion 'GTK_IS_WIDGET (widget)' failed
Mar 10 13:13:58 ethmoid NetworkManager[912]: <warn>  [1615378438.3333] sup-iface[6296d1a1f9ac3a73,1,wlp5s0]: call-p2p-cancel: failed with P2P cancel failed
Mar 10 13:13:58 ethmoid systemd[1]: syncthing.service: Scheduled restart job, restart counter is at 6.
Mar 10 13:13:58 ethmoid systemd[1]: Stopped Syncthing service.
Mar 10 13:13:58 ethmoid systemd[1]: syncthing.service: Consumed 1h 4min 6.876s CPU time, no IP traffic.
Mar 10 13:13:58 ethmoid systemd[1]: Started Syncthing service.
Mar 10 13:13:58 ethmoid syncthing[22990]: [start] 13:13:58 INFO: syncthing v1.8.0 "Fermium Flea" (go1.14.12 linux-amd64) nix@nix 1980-01-01 00:00:00 UTC [stnou>
Mar 10 13:13:58 ethmoid syncthing[22990]: [YWRW6] 13:13:58 INFO: My ID: YWRW6UX-77BCS7Z-BW3NCIZ-2P6EB2R-4BELP3L-OENVOQ4-PXH2ZBY-COVXZAM
Mar 10 13:13:58 ethmoid plank.desktop[19848]: [19848:19848:0310/131358.936441:ERROR:connection.cc(65)] X connection error received.
Mar 10 13:13:58 ethmoid .io.elementary.[3934]: io.elementary.notifications: Fatal IO error 11 (Resource temporarily unavailable) on X server :0.
Mar 10 13:13:58 ethmoid plank.desktop[26027]: The X11 connection broke (error 1). Did the X11 server die?
Mar 10 13:13:58 ethmoid .io.elementary.[30107]: .io.elementary.switchboard-wrapped_: Fatal IO error 104 (Connection reset by peer) on X server :0.
Mar 10 13:13:58 ethmoid .light-locker-w[3938]: light-locker: Fatal IO error 104 (Connection reset by peer) on X server :0.
Mar 10 13:13:58 ethmoid .gala-daemon-wr[3925]: gala-daemon: Fatal IO error 11 (Resource temporarily unavailable) on X server :0.
Mar 10 13:13:58 ethmoid .gsd-wacom-wrap[3772]: gsd-wacom: Fatal IO error 11 (Resource temporarily unavailable) on X server :0.
Mar 10 13:13:58 ethmoid kernel: .keepassxc-wrap[26027]: segfault at 8 ip 00007fca9fddaeb7 sp 00007ffcb726d3a8 error 6 in libQt5Core.so.5.15.2[7fca9fd9d000+30d0>
Mar 10 13:13:58 ethmoid kernel: Code: 06 b8 01 00 00 00 c3 48 83 f8 03 77 05 e9 71 fe ff ff 48 89 c7 e9 c9 00 00 00 90 0f 1f 84 00 00 00 00 00 31 c0 ba 01 00 0>
Mar 10 13:13:58 ethmoid .wingpanel-wrap[3895]: .wingpanel-wrapped_: Fatal IO error 11 (Resource temporarily unavailable) on X server :0.
Mar 10 13:13:58 ethmoid plank[3894]: plank: Fatal IO error 104 (Connection reset by peer) on X server :0.
Mar 10 13:13:58 ethmoid systemd[1]: Stopping Session 13 of user davidak.
Mar 10 13:13:58 ethmoid gnome-session[3718]: gnome-session-binary[3718]: WARNING: App 'pantheon-org.gnome.SettingsDaemon.Wacom.desktop' exited with code 1
Mar 10 13:13:58 ethmoid gnome-session-binary[3718]: WARNING: App 'pantheon-org.gnome.SettingsDaemon.Wacom.desktop' exited with code 1
Mar 10 13:13:58 ethmoid .bamfdaemon-wra[3906]: bamfdaemon: Fatal IO error 11 (Resource temporarily unavailable) on X server :0.
Mar 10 13:13:58 ethmoid sudo[5369]: pam_unix(sudo:session): session closed for user root
Mar 10 13:13:58 ethmoid kernel: rfkill: input handler enabled
Mar 10 13:13:58 ethmoid .nm-applet-wrap[3726]: nm-applet: Fatal IO error 11 (Resource temporarily unavailable) on X server :0.
Mar 10 13:13:59 ethmoid systemd[31453]: bamfdaemon.service: Main process exited, code=exited, status=1/FAILURE
Mar 10 13:13:59 ethmoid systemd[31453]: bamfdaemon.service: Failed with result 'exit-code'.
Mar 10 13:13:59 ethmoid systemd[31453]: nm-applet.service: Main process exited, code=exited, status=1/FAILURE
Mar 10 13:13:59 ethmoid systemd[31453]: nm-applet.service: Failed with result 'exit-code'.
...

Here is another log: https://gist.github.com/davidak/e3feae09909844c3b954d5819260fbcc
To Reproduce
Steps to reproduce the behavior:

suspend
hit enter to get login
login
the login might fail, even with the correct password
login again
you get a new session

Expected behavior
just resume from suspend. use previous session
Additional context
I had this issue before on this device, so i didn't use suspend a long time. It's a Lenovo L470 notebook. Right now i have just one external display connected.
https://codeberg.org/davidak/nixos-config/src/branch/master/machines/ethmoid/configuration.nix
The difference is that it now also happens without an external display connected. The workaround to disconnect the displays before suspending don't work anymore. So, rather logout and suspend then.
Related issue: https://github.com/NixOS/nixpkgs/issues/60082
It is not reproducible on my Workstation which uses the latest mainline kernel. It has 2 displays connected.

In elementary OS 6, we’ve replaced Light Locker with our own “screen shield” implementation. This implementation leans more heavily on features provided by GNOME Settings Daemon and brings greater compatibility with third-party apps.

Source: https://blog.elementary.io/platform-changes-in-elementary-os-6/
Might that fix it? How is your plan to upgrade to elementary 6?
Notify maintainers
@worldofpeace
Metadata

system: "x86_64-linux"
host os: Linux 5.4.95, NixOS, 20.09.3009.8e78c2cfbae (Nightingale)
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.3.10
channels(root): "home-manager-20.09, nixos-20.09.3009.8e78c2cfbae, nixos-hardware, nixos-unstable-21.03pre268835.8c8731330b5"
nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

Maintainer information:
# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

I did not see this issue in a while, so closing for now.


Dataset Viewer issue for shibing624/medical

Link
https://huggingface.co/datasets/shibing624/medical
Description
The dataset viewer is not working for dataset shibing624/medical.
Error details:
Error code:   ResponseNotReady

Thanks for reporting, @shibing624.
We are having issues with the datasets-server, which is busier than usual and the response is not ready yet.
We'll keep you informed once the issues are fixed.

waiting for you ...

The viewer is fixed now:


BUG: io.mmread fails to read skew-symmetric real matrix

Describe your issue.
scipy fails to read a 512x512 skew-symmetric real matrix in MM format. The matrix was generated in MATLAB using mmwrite.m. A symmetric version of the real matrix is read without problems.
mminfo for the skew-symmetric matrix:
mminfo("19-512.mtx")
(512, 512, 262144, 'array', 'real', 'skew-symmetric')

for the symmetric matrix:
mminfo("18-512.mtx")
(512, 512, 262144, 'array', 'real', 'symmetric')

Difference between both files:
-%%MatrixMarket matrix array real symmetric
+%%MatrixMarket matrix array real skew-symmetric
 512 512
  4
 -1

The respective matrices can be found here:
19-512.mtx
18-512.mtx
Reproducing Code Example
mmread("19-512.mtx")

Error message
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/fvanmaele/.local/lib/python3.8/site-packages/scipy/io/_mmio.py", line 76, in mmread
    return MMFile().read(source)
  File "/home/fvanmaele/.local/lib/python3.8/site-packages/scipy/io/_mmio.py", line 436, in read
    return self._parse_body(stream)
  File "/home/fvanmaele/.local/lib/python3.8/site-packages/scipy/io/_mmio.py", line 562, in _parse_body
    a[i, j] = 0
IndexError: index 512 is out of bounds for axis 0 with size 512

SciPy/NumPy/Python version information
1.8.0 1.22.2 sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0)

It looks like there is a bug in the Matlab code that generated the file 19-512.mtx.  According to https://math.nist.gov/MatrixMarket/formats.html, https://networkrepository.com/mtx-matrix-market-format.html and https://people.sc.fsu.edu/~jburkardt/data/mm/mm.html, the diagonal elements are not to be included in the skew-symmetric format.  The file 19-512.mtx has 131330 lines.  The first two are the header, so the remaining 131328 are data.  But for a 512x512 matrix, there should be 512*511/2 = 130816 entries below the diagonal.  If the diagonal is included, the number of elements on or below the diagonal is 512*513/2 = 131328, so it appears that the file 19-512.mtx incorrectly includes the diagonal elements in the file.

Since the problem was apparently a bad file, I'm closing the issue.