I cannot predict with f.data.experimental.bucket_by_sequence_length

Please make sure that this is a bug. As per our
GitHub Policy,
we only address code/doc bugs, performance issues, feature requests and
build/installation issues on GitHub. tag:bug_template
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): source
TensorFlow version (use command below): 2.4.1
Python version: 3.8.5
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version: CUDA v11.0, cuDNN 8
GPU model and memory: Nvidia Quadro P2000 64 GB

You can collect some of this information using our environment capture
script
You can also obtain the TensorFlow version with:

TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"

Describe the current behavior
I cannot predict with f.data.experimental.bucket_by_sequence_length.
Describe the expected behavior
I don't know if there is a bug in my code or there is a bug in the implementation. How come training with dataset using tf.data.experimental.bucket_by_sequence_length is ok, but predicting is not ok? My error has to do with incorrect dimension but isn't that what tf.data.experimental.bucket_by_sequence_length  and assigning shaped of (None, X) are used for?
Standalone code to reproduce the issue
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)

def gen_train():

         for x, y in zip(X_train, y_train):

                yield (x, y)

def gen_val():

         for x, y in zip(X_test, y_test):

         yield (x, y)

X_train_to_dataset = tf.data.Dataset.from_generator(lambda: X_train,
 output_types=(tf.float64),
 output_shapes=(tf.TensorShape([None, len(X_train[0][0])])))

train_data = tf.data.Dataset.from_generator(gen_train,
 output_types=(tf.float64,
tf.float64),output_shapes(tf.TensorShape([None, len(X_train[0][0])]),
  tf.TensorShape([None, len(y_train[0][0])])))

train_data = tf.data.Dataset.zip((train_data.map(lambda c, d: c), train_data.map(lambda a, b: tf.expand_dims(b[:, 1], axis=1))))

val_data = tf.data.Dataset.from_generator(gen_val,
 output_types=(tf.float64, tf.float64),
  output_shapes=(tf.TensorShape([None, len(X_train[0][0])]),
 tf.TensorShape([None, len(y_train[0][0])]))
)

val_data = tf.data.Dataset.zip((val_data.map(lambda c, d: c), val_data.map(lambda a, b: tf.expand_dims(b[:, 1], axis=1))))

def element_length_fn(x, y):

         return tf.shape(x)[0]

max_train = len(max(X_train, key=len))

min_train = len(min(X_train, key=len))

max_val = len(max(X_test, key=len))

min_val = len(min(X_test, key=len))

batch_size = 64

bucket_boundaries_train = list(np.arange(min_train, max_train, 10))
bucket_batch_sizes_train = [batch_size] * (len(bucket_boundaries_train) + 1)
bucket_boundaries_val = list(np.arange(min_val, max_val, 10))
bucket_batch_sizes_val = [batch_size] * (len(bucket_boundaries_val) + 1)

train_data = train_data.apply(tf.data.experimental.bucket_by_sequence_length(
              element_length_func=element_length_fn,

              bucket_batch_sizes=bucket_batch_sizes_train,
              bucket_boundaries=bucket_boundaries_train
)
)

val_data = val_data.apply(
tf.data.experimental.bucket_by_sequence_length(
              element_length_func=element_length_fn,

              bucket_batch_sizes=bucket_batch_sizes_val,
              bucket_boundaries=bucket_boundaries_val,)

def create_model():

     i = Input(shape=(None, len(X_train[0][0])))

     x = Bidirectional(LSTM(128, return_sequences=True))(i)

     x = TimeDistributed(Dense(1))(x)

     return tf.keras.Model(inputs=i, outputs=x)

model = create_model()

model.compile(
loss='mse',
optimizer='adam',metrics=[metrics.RootMeanSquaredError()])

model.fit(train_data,
         epochs=5,
         validation_data=val_data
)

NormLayer = preprocessing.Normalization()

NormLayer.adapt(X_train_to_dataset)

model = create_model_bdlstm128(NormLayer)

model.compile(
loss='mse',optimizer='adam',
metrics=[metrics.RootMeanSquaredError()])

model.fit(
train_data, epochs=5000,
 validation_data=val_data)

y_pred = model.predict(val_data.map(lambda a, b: a))

Other info / logs Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached.
The below is the error for y_pred = model.predict(val_data.map(lambda a, b: a))
 :
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Users\ipancorbo\Anaconda3\envs\tf_recsys\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1646, in predict
    all_outputs = nest.map_structure_up_to(batch_outputs, concat, outputs)
  File "C:\Users\ipancorbo\Anaconda3\envs\tf_recsys\lib\site-packages\tensorflow\python\util\nest.py", line 1159, in map_structure_up_to
    return map_structure_with_tuple_paths_up_to(
  File "C:\Users\ipancorbo\Anaconda3\envs\tf_recsys\lib\site-packages\tensorflow\python\util\nest.py", line 1257, in map_structure_with_tuple_paths_up_to
    results = [
  File "C:\Users\ipancorbo\Anaconda3\envs\tf_recsys\lib\site-packages\tensorflow\python\util\nest.py", line 1258, in <listcomp>
    func(*args, **kwargs) for args in zip(flat_path_gen, *flat_value_gen)
  File "C:\Users\ipancorbo\Anaconda3\envs\tf_recsys\lib\site-packages\tensorflow\python\util\nest.py", line 1161, in <lambda>
    lambda _, *values: func(*values),  # Discards the path arg.
  File "C:\Users\ipancorbo\Anaconda3\envs\tf_recsys\lib\site-packages\tensorflow\python\keras\engine\training.py", line 2716, in concat
    return array_ops.concat(tensors, axis=axis)
  File "C:\Users\ipancorbo\Anaconda3\envs\tf_recsys\lib\site-packages\tensorflow\python\util\dispatch.py", line 201, in wrapper
    return target(*args, **kwargs)
  File "C:\Users\ipancorbo\Anaconda3\envs\tf_recsys\lib\site-packages\tensorflow\python\ops\array_ops.py", line 1677, in concat
    return gen_array_ops.concat_v2(values=values, axis=axis, name=name)
  File "C:\Users\ipancorbo\Anaconda3\envs\tf_recsys\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", line 1192, in concat_v2
    _ops.raise_from_not_ok_status(e, name)
  File "C:\Users\ipancorbo\Anaconda3\envs\tf_recsys\lib\site-packages\tensorflow\python\framework\ops.py", line 6862, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: ConcatOp : Dimensions of inputs should match: shape[0] = [64,29,1] vs. shape[3] = [64,30,1] [Op:ConcatV2] name: concat

Ok, I realized that this is probably because model.predict returns a bumpy array and thus concatenates every prediction (which are obviously, in my case, not equal size, and hence the concatenation error). Does tf.keras have no solution for this??

@inespancorbo
The code provided is not reproducible, please share a colab gist for us to replicate the error reported.


Asynchronous methods in readers and writers.

Contributes to #650
Extracted from #1119 but only covering the async methods on JsonReader and JsonWriter and implementation for JsonTextReader and JsonTextWriter.

I see you have ConfigureAwait(false) in the unit tests. Is that appropriate? I have seen people talk about xunit having a synchronization context.

I see you have ConfigureAwait(false) in the unit tests. Is that appropriate? I have seen people talk about xunit having a synchronization context.

It's probably okay, but it should really be (true) to be on the safe side. It's a mistake that NYE celebrations got in the way of fixing. Will rectify tomorrow.

It's probably okay, but it should really be (true) to be on the safe side. It's a mistake that NYE celebrations got in the way of fixing. Will rectify tomorrow.

I see you have ConfigureAwait(false) in the unit tests.

Where. I can't find where I made that slip (but am more than willing to believe that I did and still can't fnd it...).

I see you have ConfigureAwait(false) in the unit tests.

Where. I can't find where I made that slip (but am more than willing to believe that I did and still can't fnd it...).

Where

Whoops, I see ConfigureAwait and I automatically assume it is for false. I thought ConfigureAwait(true) was the default? Why state it explicitly?

Where

Whoops, I see ConfigureAwait and I automatically assume it is for false. I thought ConfigureAwait(true) was the default? Why state it explicitly?

I decided to state it explicitly precisely because it would be a bad idea to have ConfigureAwait(false) and hence adding in ConfigureAwait(true) marks "I know I want any synchronization to flow here, so don't think you're being clever adding a ConfigureAwait(false) at this point". Whether to be explicit or implicit is as such largely a style matter as to whether to express that explicitly or not. I'll happily follow any style guid decision without argument.

I decided to state it explicitly precisely because it would be a bad idea to have ConfigureAwait(false) and hence adding in ConfigureAwait(true) marks "I know I want any synchronization to flow here, so don't think you're being clever adding a ConfigureAwait(false) at this point". Whether to be explicit or implicit is as such largely a style matter as to whether to express that explicitly or not. I'll happily follow any style guid decision without argument.

Hurrah! This makes me very happy 😊

Hurrah! This makes me very happy 😊

Same, good stuff man!!

Same, good stuff man!!

Responded to feedback, and a few further changes. I did so as a bunch of small commits to make it easier to perhaps revert one or two on the basis of further feedback, they can be squashed if accepted.

What does MS do for async in unit tests and ConfigureAwait?

What does MS do for async in unit tests and ConfigureAwait?

Apparently leave them out (in tests) unless it causes issue
https://github.com/dotnet/corefx/pull/14771#discussion_r94254124

I looked around the corefx a bit and couldn't find any with ConfigureAwait, e.g. https://github.com/dotnet/corefx/blob/c9b1b944763aa8aa83c3bad29c12c4845ef38f21/src/System.IO.FileSystem/tests/FileStream/CopyToAsync.cs
I'd like a find/replace to get rid of the ConfigureAwait(true)s

Looking at those tests, something to worry about now will be tests where the cancellation token is true.

I've actually started on precisely that cancellation test matter, but some chores have gotten in the way, so I won't be pushing until tomorrow.

As discussed above, I took out the commit that changes WriteEscapedJavaScriptString and am leaving all the duplicate code in that and the new Async version as it's a tight loop.

FYI I made a small change to the build script to download nuget.exe during build rather than have it under source control. You'll need to remove it from the PR

Yeah, I noticed that. Good stuff, it was a nuisance there.

I can't think of anything else to share or change. Thanks!
I've done some benchmarking and profiling. Both async reading and writing are slower over a StringReader/StringWriter which is expected. Reading is about half as fast but writing is significantly slower. I'm not an async expert but WriteEscapedJavaScriptStringAsync allocations are very high.

Perf tests https://github.com/JamesNK/Newtonsoft.Json/commit/13a31e3a4d0234dc9d9adcc62af7e18c3b7ebec7
Apologies for the awful perf test class. It is about 10 years old and I've never gotten around to cleaning it up.

Being slower and heavier isn't much of a surprise since we sacrifice speed and memory use (though for the benefit of the better thread use in the sort of cases async benefits). Being that much heavier though, is a surprise, and it is also seemingly particularly with WriteEscapedJavaScriptStringAsync which doesn't seem to offer much in the way of ways to reduce allocations.
I wonder if changing the way it interacts with the buffering would help, though on first glance I imagine it would just mean we've changed from having 1,250,000 Task<char[]> to having as many Task.

Isn't the issue that a new task is being created for each call to WriteEscapedJavaScriptStringAsync to return the buffer? Would placing the buffer inside a wrapper class, and passing that wrapper into WriteEscapedJavaScriptStringAsync, and changing the array ref in the wrapper when needed fix that?

That's what I'm wondering in my last sentence above. There's still going to be tasks created though. Too late at night here for me to try right now though.

This quick experiment eliminated the Task<char[]> allocations - https://github.com/JamesNK/Newtonsoft.Json/commit/2d981720b8d78ebab4ba593894b65a252c109361 - although I'm sure there must be a better way of optimizing this sort of thing.

https://github.com/JamesNK/Newtonsoft.Json/commit/2d981720b8d78ebab4ba593894b65a252c109361
This quick experiment eliminated the Task<char[]> allocations and reduce traffic from 190megs to 140megs. I'm sure there must be a better way of optimizing this sort of thing. Need advice from an expert.
@dfowler @benaadams ? 😸

https://github.com/JamesNK/Newtonsoft.Json/commit/2d981720b8d78ebab4ba593894b65a252c109361
This quick experiment eliminated the Task<char[]> allocations and reduce traffic from 190megs to 140megs. I'm sure there must be a better way of optimizing this sort of thing. Need advice from an expert.
@davidfowl @benaadams ? 😸

The return value is to change the instance buffer, in case of expansion/allocation? So would be mostly same array anyway? Cached task would be nasty especially with static method.
If you're ok with exposing the buffer as a property, change to generic constrained extension method?
Other than test is only called in one place? Change from static to instance method? Would also cut down on params passed.
Ensuring writeBuffer != null && writeBuffer.Length >= UnicodeTextLength (const 6) before the call would get rid of one of the possible allocations. Adding an extra write to flush the buffer if > length would get rid of another.
Remaining one is WriteEscapedStringThroughBuffer but could always rent and return as part of that if oversized; rather than changing and holding on to the buffer? Or is it because it doesn't always use ArrayPool and may be allocating the buffer so better to hold?

Could also drop the instance buffer cache and internalize the pool to the method? Or that no go?

Okay. The last four commits have had some impact on allocations from WriteEscapedJavaScriptStringAsync.
fba5260 changes the allocation approach to one that calls back to the writer to ask that the buffer be large enough. This tightens the coupling, but if there's ever another client for the method it could be loosened again with an internal interface. It reduced total bytes allocated by about 11% and objects by about 13%.
7d56e1e splits up the method a bit for different branches. That slightly worsened allocations, but as well as enabling the next bits it would likely improve things for non-delimiter using branches that could hit it.
45d87e7 checks if escaping is necessary at all and returns just the TextWriter.WriteAsync() task when it can. As well as being more thorough than ShouldEscapeJavaScriptString it also returns the point at the first character needing escaping is found so escaped writing can start by writing the first bit of the string and searching from that point on. This lowered allocations a bit further and could be even more impactful in cases with lots of strings that don't have characters that need escaping.
d97451e finally just tightened up a few unnecessary checks (the buffer can no longer be null so don't test for it) and avoids calling WriteAsync with zero chars. It's effect was marginal, but it's still worth doing.

loosened again with an internal interface

Sounds good.
FYI there is a change to an exception inside JsonTextReader that might need to be replicated in the async version https://github.com/JamesNK/Newtonsoft.Json/pull/1127/files#diff-2e13f50a9db15cfb3f0139531b953d32R1238

Yeah, had to update the test to correspond to the new error message.

Memory profiling shows the Task issue is solved which is great. Writing performance is still rather bad. AFAICT it is overhead from async state machines.

Okay, some more changes that have some improvements in performance.
8ecd5fd is one I'm a bit uneasy about, but it definitely has an impact. It leads to some duplication, though the duplicate cases are visually close to each other which helps in terms of maintenance, but since the change to AutoCompleteAsync means it often returns synchronously and since it, or InternalWriteValueAsync which tail-calls it, followed by a single await we can branch on IsCompleted and just call that task instead of awaiting.
The downside in terms of duplication is not to be sniffed at, though.

Definitely faster. It has gone from about 240ms to 180ms 👍

I think after these last few comments this monster can be merged.

Here we go…

Ready to go I think. Squish it and I'll merge 😸

Squashed.
(I gather there's a way to squash while merging in one handy step in github now, but I haven't been on that side of a github project since it was brought in).

So there is. I'd forgotten about that feature.
Merged :shipit:

It's a good feature, as lots of little commits is great in a PR, but can lead to very chatty logs if merged above about 2 or 3.
Right, now the fun begins on tokens and schemata.

Don't worry about async and JsonValidatingReader. It is obsolete and can use the base async methods from JsonReader. Thing I'd most like to see async are JTokenReader/JTokenWriter

I'm not going to touch the obsolete stuff, but the Schema project can gain from it once it picks it up in the nuget.
The token readers and writers don't have much to gain, since they're mostly doing in-memory stuff, but the serialisers will gain from the tokens having async methods, so it's a sensible next step.


Some clean up, touch support

Thanks to @dark-1 for touch support (https://github.com/dark-1/particles.js/commit/e8cce0ba6a6d6f4d242072b1ce8ab13706c12d4e) for this issue

aligned with https://github.com/dark-1/particles.js/commit/b768bbc8bda2e8e7051b2afeb4ed74900abed532 commit for touch support

@outmanos the issue https://github.com/VincentGarreau/particles.js/issues/356 is now fixed here if you still need touch particles


Panic on Xs Max ios 12.1.1 latest unc0ver b50

Describe the bug
It happens from time to time to freeze then restart. Last time happened when i woke up the phone. I’ve attached a panic log.
Device (please complete the following information):

iOS Version: 12.1.1
iOS Device: Xs Max
unc0ver Version: 3.0.0b50

Place an "x" between the brackets if true:

[X] this is a bug others will be able to reproduce
[ ] this issue is present with all tweaks uninstalled(except for default packages) or disabled
[ ] this issue is present after a rootfs restore
[X] this issue is present on the latest version of unc0ver

Logs
If applicable, add logs or error messages here.
https://www.dropbox.com/s/nqmvem0nfy70ao8/panic-full-2019-04-17-133035.900.ips?dl=0

These are the same kind of panics that I get with sks timeout on XS.
"panicString" : "panic(cpu 0 caller 0xfffffff01f77d4e4): \"sks request timeout (20 strikes)\"@\/BuildRoot\/Library\/Caches\/com.apple.xbs\/Sources\/AppleKeyStore\/AppleKeyStore-639.230.1\/AppleKeyStore.cpp:2515\nDebugger message: panic\nMemory ID: 0xff\nOS version: 16C50\nKernel
"panicString" : "panic(cpu 1 caller 0xfffffff01159d8a4): \"sks request timeout (20 strikes)\"@\/BuildRoot\/Library\/Caches\/com.apple.xbs\/Sources\/AppleKeyStore\/AppleKeyStore-639.220.5\/AppleKeyStore.cpp:2515\nDebugger message: panic\nMemory ID: 0xff\nOS version: 16B92\nKernel

@2600box  @Akasuo  Did you solve the problem?


Update doobie-core, doobie-postgres to 1.0.0-RC3

About this PR
📦 Updates

org.tpolecat:doobie-core
org.tpolecat:doobie-postgres

from 1.0.0-RC2 to 1.0.0-RC3
📜 GitHub Release Notes - Version Diff
Usage
✅ Please merge!
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

⚙ Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.tpolecat" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "org.tpolecat" }
}]

labels: library-update, early-semver-pre-release, semver-spec-pre-release, version-scheme:early-semver, commit-count:1

Superseded by #182.


test: enable nested virtualization

Nested virtualization is needed for Kata Containers. This works on
hosts which have kvm_intel loaded with nested=1. Might also need a
recent enough host CPU.

Let's merge via #374.


Need to add market orders

In addition to limit orders, need to support market orders.

Market orders are now supported as edge cases of limit orders.


Upgrade terraform-provider-azurerm to v3.114.0

This PR was generated via $ upgrade-provider pulumi/pulumi-azure --kind=all --target-bridge-version=latest.
After patches applied cleanly, it didn't build due to duplicate imports and did

upstream.sh checkout
edit
commit
upstream.sh rebase -i
upstream.sh check_in

Upgrading terraform-provider-azurerm from 3.113.0  to 3.114.0.
Fixes #2308

Looks like the Java test failure is something to do with downloading the random provider. I think this might be related to a change in ci-mgmt which disables auto-download of plugins.

@thomas11 Looks like a java test failed with:
  Error: error resolving type of resource random: internal error loading package "random": Could not automatically download and install resource plugin 'pulumi-resource-random' at version v4.16.3, install the plugin using `pulumi plugin install resource random v4.16.3`: error downloading provider random to file: failed to download plugin: random-4.16.3: write /tmp/pulumi-plugin-tar2770403083: no space left on device

I retried.

Oh, it's lack of disk space apparently

Looks like the Java test failure is something to do with downloading the random provider. I think this might be related to a change in ci-mgmt which disables auto-download of plugins.

I'm skeptical, since this PR doesn't include that change. It does bump the version of Java used though...

We should be able to set freeDiskSpaceBeforeTest in the ci-mgmt config:
https://github.com/pulumi/ci-mgmt/blob/3c3b89fcd146996cae678b5775f44ca81def5f71/provider-ci/internal/pkg/templates/bridged-provider.config.yaml#L69-L70


Update head.py

Replace scalar_summary for tf.summary.scalar

Can one of the admins verify this patch?

@tensorflow-jenkins test this please

#11159

@tensorflow-jenkins test this please

@alanyee lease address the remaining test failures in:
https://ci.tensorflow.org/job/tensorflow-pull-requests-cpu/6219/consoleFull

@tensorflow-jenkins test this please

@tensorflow-jenkins test this please

@alanyee @ispirmustafa It looks like there are some non-trivial changes in the tags causing a failure in the unit tests:
======================================================================
FAIL: testBinaryClassificationEval (main.BinaryClassificationHeadTest)
Traceback (most recent call last):
File "/var/lib/jenkins/workspace/tensorflow-pull-requests-cpu/bazel-ci_build-cache/.cache/bazel/_bazel_jenkins/eab0d61a99b6696edb3d2aff87b585e8/execroot/workspace/bazel-out/local-opt/bin/tensorflow/contrib/learn/head_test.runfiles/org_tensorflow/tensorflow/contrib/learn/python/learn/estimators/head_test.py", line 814, in testBinaryClassificationEval
_assert_summary_tags(self, ["loss"])
File "/var/lib/jenkins/workspace/tensorflow-pull-requests-cpu/bazel-ci_build-cache/.cache/bazel/_bazel_jenkins/eab0d61a99b6696edb3d2aff87b585e8/execroot/workspace/bazel-out/local-opt/bin/tensorflow/contrib/learn/head_test.runfiles/org_tensorflow/tensorflow/contrib/learn/python/learn/estimators/head_test.py", line 90, in _assert_summary_tags
test_case.assertItemsEqual(expected_tags or [], actual_tags)
AssertionError: Element counts were not equal:
First has 1, Second has 0:  'loss'
First has 0, Second has 1:  u'binary_logistic_head/loss'

Ah, I have been trying to get around that problem which has occurred for another PR:
Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/tensorflow-pull-requests-cpu/bazel-ci_build-cache/.cache/bazel/_bazel_jenkins/eab0d61a99b6696edb3d2aff87b585e8/execroot/workspace/bazel-out/local-opt/bin/tensorflow/contrib/learn/head_test.runfiles/org_tensorflow/tensorflow/contrib/learn/python/learn/estimators/head_test.py", line 1444, in testMultiClassWithLabelKeysEvalAccuracy1
    _assert_summary_tags(self, ["loss"])
  File "/var/lib/jenkins/workspace/tensorflow-pull-requests-cpu/bazel-ci_build-cache/.cache/bazel/_bazel_jenkins/eab0d61a99b6696edb3d2aff87b585e8/execroot/workspace/bazel-out/local-opt/bin/tensorflow/contrib/learn/head_test.runfiles/org_tensorflow/tensorflow/contrib/learn/python/learn/estimators/head_test.py", line 90, in _assert_summary_tags
    test_case.assertItemsEqual(expected_tags or [], actual_tags)
AssertionError: Element counts were not equal:
First has 1, Second has 0:  'loss'
First has 0, Second has 1:  u'multi_class_head/loss'

The problem is essentially that the training and eval losses show up in different graphs which is why the deprecated API was used in the first place.

Yes we intentionally keep deprecated summary API in head.py under contrib.
The heads we moved to core (python/estimator/canned/head.py) we use the new summary API.
Closing this since we'll not change deprecated api usages in contrib version of head.

@ispirmustafa okay but i wanted to test out one more change to see if it would work


Graph build fail in rare situations with sac_scale due to BikeFlagEncoder bug

I try to build a graph with GH 0.4-rc1 for the Germany region: baden-wuerttemberg-latest.osm.pbf downloaded from Geofabrik today.
(This file was last modified 4 hours ago and contains all OSM data up to 2015-03-03T21:21:02Z.)
I use the options CH off and (car, foot, bike).
But the build fails with:
Exception in thread "PBF Reader" Exception in thread "main" java.lang.RuntimeException: Couldn't process file baden-wuerttemberg-latest.osm.pbf, error: car, no speed found for:cycleway
    at com.graphhopper.reader.OSMReader.writeOsm2Graph(OSMReader.java:319)
    at com.graphhopper.reader.OSMReader.readGraph(OSMReader.java:150)
    at com.graphhopper.GraphHopper.importData(GraphHopper.java:663)
    at com.graphhopper.GraphHopper.process(GraphHopper.java:632)
    at com.graphhopper.GraphHopper.importOrLoad(GraphHopper.java:605)
    at com.graphhopper.tools.Import.main(Import.java:15)
Caused by: java.lang.IllegalStateException: car, no speed found for:cycleway
    at com.graphhopper.routing.util.CarFlagEncoder.getSpeed(CarFlagEncoder.java:148)
    at com.graphhopper.routing.util.CarFlagEncoder.handleWayTags(CarFlagEncoder.java:227)
    at com.graphhopper.routing.util.EncodingManager.handleWayTags(EncodingManager.java:263)
    at com.graphhopper.reader.OSMReader.processWay(OSMReader.java:367)
    at com.graphhopper.reader.OSMReader.writeOsm2Graph(OSMReader.java:299)
    ... 5 more
java.lang.RuntimeException: Unable to read PBF file.
    at com.graphhopper.reader.pbf.PbfReader.run(PbfReader.java:53)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.InterruptedException
    at com.graphhopper.reader.OSMInputFile.process(OSMInputFile.java:265)
    at com.graphhopper.reader.pbf.PbfDecoder.sendResultsToSink(PbfDecoder.java:106)
    at com.graphhopper.reader.pbf.PbfDecoder.processBlobs(PbfDecoder.java:171)
    at com.graphhopper.reader.pbf.PbfDecoder.run(PbfDecoder.java:184)
    at com.graphhopper.reader.pbf.PbfReader.run(PbfReader.java:49)
    ... 1 more
Caused by: java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
    at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)
    at com.graphhopper.reader.OSMInputFile.process(OSMInputFile.java:262)
    ... 5 more

bike or bike2 and graph.bytesForFlags=4?

bike and default graph.bytesForFlags=4.

There was a bug in the bike encoder overwriting the wrong accept bit. Fixed now, please try and reopen if still a problem!
BTW: Thanks a lot for early testing and reporting of RC1!

Thanks Peter!
The builds seem to complete fine now.
Does the fix plays role to the routing too (besides the graph building) ?

Could be effected too, yes. But before you rebuild all areas I fear there will be another major edge version change with #352

Not trying to be impatient :) but you mean some time soon?

working on it :)


feat(build): add wasmx to arm64 build matrix

After much trial and error, I could not make cross-compilation work by injecting the arch via kong_bindings.bzl, so this was the next best/most realistic thing.
The code duplication is pretty blah. Happy to hear recs on how I can cut down on this.

Has this been tested with a full matrix CI run?

Yup yup!


Relaton model for W3C documents

Do we have the mapping in PlantUML? Can we put it up in this repo?

@ronaldtse do you asking if we have W3C grammar? I don't see it in the metanorma-model-iso.

Could you make a new relaton-model-w3c repo to contain this bibdata model?

Should the model inherit from BibliographicItem?

Yes, it should. Thanks!


DCD-1531: Added initial version of JSM end-to-end test.

Pull request description
This PR adds end-to-end test for JSM.
Checklist

[ ] I have successful end to end tests run (with & without domain)
[ ] I have added unit tests (if applicable)
[ ] I have user documentation (if applicable)

e2e successful runs:
jsm: https://github.com/atlassian-labs/data-center-terraform/actions/runs/2389140513
domain: https://github.com/atlassian-labs/data-center-terraform/actions/runs/2389291355
no domain: https://github.com/atlassian-labs/data-center-terraform/actions/runs/2390459985


Not a bug :)

Just wanted to say - awesome component, I like it!!!
Very good job!!!

Thank you very mush,
If you have any request, feature do not hesitate


View couldn't be changed or deleted

System information:

Operating system (distribution) and version win10 x64
DBeaver version 7.1.3

Connection specification:

Database name and version Clickhouse

Describe the problem you're observing:
Error appears when trying to change or delete view.
View changing:
Code: 57, e.displayText() = DB::Exception: Table default.NewView already exists., e.what() = DB::Exception

SQL Error [57]: ClickHouse exception, code: 57, host: <IP_ADDRESS>, port: 40127; Code: 57, e.displayText() = DB::Exception: Table default.NewView already exists., e.what() = DB::Exception

  ClickHouse exception, code: 57, host: <IP_ADDRESS>, port: 40127; Code: 57, e.displayText() = DB::Exception: Table default.NewView already exists., e.what() = DB::Exception

  ClickHouse exception, code: 57, host: <IP_ADDRESS>, port: 40127; Code: 57, e.displayText() = DB::Exception: Table default.NewView already exists., e.what() = DB::Exception

    java.lang.Throwable: Code: 57, e.displayText() = DB::Exception: Table default.NewView already exists., e.what() = DB::Exception

    Code: 57, e.displayText() = DB::Exception: Table default.NewView already exists., e.what() = DB::Exception

Delete view
Code: 62, e.displayText() = DB::Exception: Syntax error: failed at position 6: VIEW default.NewView. Expected one of: DATABASE, TEMPORARY, TABLE, e.what() = DB::Exception

SQL Error [62]: ClickHouse exception, code: 62, host: <IP_ADDRESS>, port: 40127; Code: 62, e.displayText() = DB::Exception: Syntax error: failed at position 6: VIEW default.NewView. Expected one of: DATABASE, TEMPORARY, TABLE, e.what() = DB::Exception

  ClickHouse exception, code: 62, host: <IP_ADDRESS>, port: 40127; Code: 62, e.displayText() = DB::Exception: Syntax error: failed at position 6: VIEW default.NewView. Expected one of: DATABASE, TEMPORARY, TABLE, e.what() = DB::Exception

  ClickHouse exception, code: 62, host: <IP_ADDRESS>, port: 40127; Code: 62, e.displayText() = DB::Exception: Syntax error: failed at position 6: VIEW default.NewView. Expected one of: DATABASE, TEMPORARY, TABLE, e.what() = DB::Exception

    java.lang.Throwable: Code: 62, e.displayText() = DB::Exception: Syntax error: failed at position 6: VIEW default.NewView. Expected one of: DATABASE, TEMPORARY, TABLE, e.what() = DB::Exception

    Code: 62, e.displayText() = DB::Exception: Syntax error: failed at position 6: VIEW default.NewView. Expected one of: DATABASE, TEMPORARY, TABLE, e.what() = DB::Exception

View delete fixed. View alter is not supported by current Clickhouse version (https://github.com/ClickHouse/ClickHouse/issues/1226)

verified


ContractExecuteTransaction.executeForRecord() - more return values

From Smart Contract calls and transactions, need array return types like

bool[]
address[]
etc.

for a solidity method like:
function bulkTransfer(address[] memory _tos, uint256[] memory _values)
        public whenNotPaused
        returns (bool[] memory successes) {

            require(_tos.length == _values.length, "length of tos and values aren't same");

            bool[] memory successes = new bool[](_tos.length);
            for(uint i=0; i<_tos.length; i++) {
                successes[i] = _transfer(msg.sender, _tos[i], _values[i]);
            }

        return successes;
    }

@rahul-ki we originally thought not to implement bool[] as it's incredibly wasteful in the Solidity ABI. Each boolean in the array takes up a whole 32 bytes. If you have control over the contract bytecode, you probably want a custom bitvector based off bytes instead.
address[] is reasonable to add though.

Please open a specific method for a specific ask here if you need something. We don't mind adding what is needed. It's not complicated, just tedious.


fix(azure-cosmosdb): Add caches for Azure CosmosDB vCore

Fixes # (issue)

Actually - would love to tag @sinedied for a second look!

@jacoblee93  @sinedied  comments and conflict resolved, also add test for mongo cache. Could you help review and approve this PR? Thanks a lot


Stopping due to Debug session failure: fatal error: NullReferenceException: Object reference not set to an instance of an object

Issue Type: Bug
push F5
VS Code version: Code 1.37.1 (f06011ac164ae4dc8e753a3fe7f9549844d15e35, 2019-08-15T16:17:55.855Z)
OS version: Windows_NT x64 10.0.18362
Extensions (13)

Extension
Author (truncated)
Version

code-gnu-global
aus
0.2.2

githistory
don
0.4.6

gitlens
eam
9.9.3

git-project-manager
fel
1.7.1

cortex-debug
mar
0.3.1

vscode-clang
mit
0.2.3

vscode-language-pack-ja
MS-
1.37.5

remote-wsl
ms-
0.39.2

cpptools
ms-
0.25.0

csharp
ms-
1.21.0

quicktype
qui
12.0.46

debug
web
0.23.1

linkerscript
Zix
1.0.0

This problem is extensions C/C++ IntelliSense, debugging, and code browsing. version 0.25.0.
There is no problem in version 0.24.1.

Hi @gummoni, since this a problem with the C++ extension, can you please reopen this issue there? https://github.com/Microsoft/vscode-cpptools


Update README.md

Better Readme.md instructions

Added a warning because of wayland and basic input by xdg and xorg.


Transformer: unpacking children

Hello,
First, thanks for making Lark, I just discovered it, and I already like it a lot !
My suggestion would be this:
Adding a transform_unpack alternative to Transformer.transform which passes children in unpacked form:
For instance, in my transformer, a rule of the form
def int(s,x):
    a,b, name, init = x
    ... do something with it

would become
def int(s, a,b, name, init):

which matches more closely the intuitive view of each node as a constructor in a pattern-matching.
Of course in variadic cases, you could still write
def int(s, a,b, name, init, *other_args):

I have only taken a cursory look, but this seems trivial to implement by adding an option to _call_userfunc and calling f(children) or f(*children) depending on whether it is passed. (_call_userfunc could also be duplicated if performance from the added "if" is a concern)
Would this be something you're interested in implementing or, if I submit a PR to that effect, that you would be willing to integrate?
Cheers

Well that was fast and painless.
I had searched the doc and the issues, but for packing/unpacking keywords rather than "inline".
Thanks, and have a nice day.

Sphinx is already set up, and iirc most of the docs are in markdown, so you don't need to worry about it too much.


KMD Basilisk Mode Send logout

The same problem seems to happen on both Linux and Windows.

Please test with latest Agama Wallet and reopen this issue again.


Add textarea to form story

Closes
✅ Pull Request Checklist:

[ ] Included link to corresponding Issue.
[ ] Added/updated unit tests and storybook for this change (for new code or code which already has tests).
[ ] Filled out test instructions.
[ ] Updated documentation (if it already exists for this component).
[ ] Looked at the Accessibility Practices for this feature - Aria Practices

📝 Test Instructions:

🧢 Your Project:

Changing the textarea width will change the form width but hopefully we can have https://github.com/adobe-private/react-spectrum-v3/pull/528 to fix it :D


Keep selection after Find & Replace

FD <IP_ADDRESS> / HD <IP_ADDRESS>
If you want to replace multiple times on the same selected text, it's not preserved--the selection turns into a solitary cursor--so you have to keep re-selecting it.

This is intended behaviour, Visual Studio does this too.

Really?  Then no offence, but I wouldn't like VS, either, for this.  TextPad doesn't do it this way.
Consider this: the TextPad way, if you don't need the selection, moving the arrow keys will ditch it for you anyway--so keeping the selection does not at all get in your way.  But if you do need it to do another replacement, then it's there, and you save a tedious re-select.  The VS/FD/HD way, you are not given the option, so if you don't need it, you save nothing, and if you do need it...tedium.

In what use case you want to replace the same text multiple times?

Not the same text, different text over the same selection.
E.g. you get the logic right for x and width, then duplicate a bunch of such lines.  You then want to replace x with y and width with height.  That's two replacements (two texts!) to do over the same selection.

You mean a case where you operate in a selected lines of code? The block selection is required only when you replace all, you can just replace next occurence from you current position. Or am i missing something here?
Could you create a gif showing exactly what you mean?

No, you understood right--for a Replace All on selected lines of code.

Ah ok. Yes on replace all the selection should be restored.


Admit EXPLAIN queries when required worker count not satisfied

Given coordinator with
node-scheduler.include-coordinator=false
query-manager.required-workers=1

query e.g.
EXPLAIN (TYPE IO) ....

will get into WAITING_FOR_RESOURCES state, waiting for a worker.
However, once the worker shows up and query gets admitted, the query executes on coordinator only.
We should not wait for a worker in this case.

I used EXPLAIN (TYPE IO) as an example, because the results do not depend on workers in any way.
EXPLAIN factors-in current worker count, so the results may be different once the required number
of workers shows up.


CRAN submission.

We can use this issue to discuss anything related to CRAN submission including any feedback that we might get.
#79 is also going to contain some commits so that we can submit to CRAN.

@sdhutchins
I've done a lot of work this afternoon to get the regular build check to work (e.g. devtools::check())
After I get a few minor things fixed I'm going to run R CMD check --as-cran, and then try running our package through Rhub or Winbuilder .

@sdhutchins Any hot tips on how to prevent this:
Warning: replacing previous import 'scales::alpha' by 'microbiome::alpha' when loading 'MicrobiomeR'

or this:
Package suggested but not available for checking: 'covr'

@sdhutchins Any hot tips on how to prevent this:
Warning: replacing previous import 'scales::alpha' by 'microbiome::alpha' when loading 'MicrobiomeR'

or this:
Package suggested but not available for checking: 'covr'

One way with scales is to @import scales instead of @importFrom scales alpha

For covr, I'm not sure. I don't think covr is on cran.

This is an increadibly annoying Note that occurs when using NSE:
> checking R code for possible problems ... NOTE
 convert_proportions: no visible binding for global variable 'SampleID'
  convert_proportions: no visible binding for global variable 'Abundance'
  convert_proportions: no visible global function definition for ':='
  convert_proportions: no visible binding for global variable
    'TreatmentGroup'
  correlation_data: no visible binding for global variable 'treatment_1'
  correlation_data: no visible binding for global variable 'treatment_2'
  correlation_data: no visible binding for global variable
    'wilcox_p_value'
  correlation_data: no visible binding for global variable 'Significance'
  correlation_data: no visible binding for global variable
    'log2_mean_ratio'
  correlation_data: no visible binding for global variable 'label'
  correlation_data: no visible binding for global variable '.'
  correlation_plot: no visible binding for global variable 'mean_treat2'
  correlation_plot: no visible binding for global variable 'mean_treat1'
  correlation_plot: no visible binding for global variable 'Abundance'
  correlation_plot: no visible binding for global variable
    'color_wilcox_p_value'
  correlation_plot: no visible binding for global variable 'rank_label'
  heat_tree_parameters: no visible binding for global variable
    'log2_mean_ratio'
  heat_tree_parameters: no visible binding for global variable
    'wilcox_p_value'
  heat_tree_plots: no visible binding for global variable 'n_supertaxa'
  melt_taxmap: no visible binding for global variable 'X.SampleID'
  melt_taxmap: no visible binding for global variable 'otu_id'
  otu_id_filter: no visible binding for global variable 'otu_id'
  otu_prevalence_filter: no visible binding for global variable
    'n_samples'
  permanova: no visible global function definition for 'as.formula'
  permanova: no visible global function definition for 'anova'
  permanova: no visible global function definition for 'coefficients'
  sample_id_filter: no visible binding for global variable 'sample_id'
  stacked_barplot: no visible global function definition for ':='
  stacked_barplot: no visible binding for global variable 'SampleID'
  stacked_barplot: no visible binding for global variable
    'Relative.Abundance'
  taxa_prevalence_filter: no visible binding for global variable
    'n_samples'
  top_coefficients_barplot: no visible global function definition for
    'par'
  top_coefficients_barplot: no visible global function definition for
    'barplot'
  transformer: no visible binding for global variable '.'
  transposer: no visible binding for global variable '.'
  Undefined global functions or variables:
    . := Abundance Relative.Abundance SampleID Significance
    TreatmentGroup X.SampleID anova as.formula barplot coefficients
    color_wilcox_p_value label log2_mean_ratio mean_treat1 mean_treat2
    n_samples n_supertaxa otu_id par rank_label sample_id treatment_1
    treatment_2 wilcox_p_value
  Consider adding
    importFrom("graphics", "barplot", "par")
    importFrom("stats", "anova", "as.formula", "coefficients")
  to your NAMESPACE file.

Solutions include:

https://dplyr.tidyverse.org/articles/programming.html#different-data-sets

If this function is in a package, using .data also prevents R CMD check from giving a NOTE about undefined global variables (provided that you’ve also imported rlang::.data with @importFrom rlang .data).

https://github.com/STAT545-UBC/Discussion/issues/451#issuecomment-264598618

quiets concerns of R CMD check re: the .'s that appear in pipelines
if(getRversion() >= "2.15.1")  utils::globalVariables(c("."))

For covr, I'm not sure. I don't think covr is on cran.

Should it be in the Remotes:  field in the DESCRIPTION file?

Maybe?
If it's only in Suggests, I'm not sure.

I tried, and I don't think it worked.

I'm going to try putting it in the Imports field.


feature: support user-supplied bucket name prefix

Description
This PR implements a --bucket-name-prefix parameter for scripts/generate_terraform.py. The previous implementation of using the GCP project number has been removed.
The benefits of using a user-supplied bucket name prefix are as follows:

Prevents GCS bucket naming collisions
Better readability and memorize-ability for data consumers
Make naming collisions easy to troubleshoot: If a collision is encountered for a specific env (say, prod), one can simply change the bucket name prefix for that stage

Checklist

[x] Tests pass.
[x] Linters pass.
[x] Please merge this PR for me once it is approved.
[x] If this PR adds/edits/deletes a feature, I have updated the README accordingly.
[x] If this PR adds/edits a dataset or pipeline, it was reviewed and approved by the Google Cloud Public Datasets team beforehand.

Minor change about updating the README
then my only question is are the other changes to the datasets themselves autogenerated? Asking because I noticed some inconsistencies in the before and after with hyphens and underscores in the names (vizgen-merfish vs vizgen_merfish)

Yes, the .tf files are all auto-generated.
And yes, you're right in noticing that. I noticed that we're using a mix of underscores and hyphens for the generate bucket names, previously {PREFIX}-vizgen_merfish and I thought we're renaming the bucket anyway, so I might as well set a standard to use hyphens instead of underscores.
On that note, I added that convention here: 03ad1aa

@leahecole I also added tests for the hyphenated bucket naming convention: 16818df and bfebdd3


ts-web: trying out other crypto libraries

cross reference #1371

I second this pull request, the @noble/ ECC related libraries are much cleaner than elliptic.

rebased, bumped versions

gonna see about using @noble/curves for ed25519 as well

opening this up for review, without any tweetnacl removal. I looked and I ended up getting all paranoid about ADR-0009
major change: EllipticSigner from client-rt is replaced with NobleSigner, with a similar interface. it's not tested against anything external though, we only check that the sdk can verify its own signature

ok good, secp256k1 NobleSigner matches up with the runtime-sdk rust code
import * as oasis from '@oasisprotocol/client';
import * as oasisRT from '@oasisprotocol/client-rt';

describe('secp256k1', () => {
    it('is compatible with runtime-sdk', async () => {
        // Based on `test_memory_signer` in
        // runtime-sdk/src/crypto/signature/mod.rs.
        const priv = await oasis.hash.hash(oasis.misc.fromString('memory signer test'));
        expect(oasis.misc.toHex(priv)).toEqual('4844bcd03d683656f9b7b656f5a53883f7aad7a74d8781fd953d77b5e59956de');
        const signer = oasisRT.signatureSecp256k1.NobleSigner.fromPrivate(priv, 'this key is not important');
        const contextSigner = new oasisRT.signatureSecp256k1.BlindContextSigner(signer);
        const pk = contextSigner.public();
        expect(oasis.misc.toHex(pk)).toEqual('0305327494815250eb0850beb36e78c512a1a42d37503f18843d5eb55554742117');
        const ctx = 'oasis-core/test: context';
        const message = oasis.misc.fromString('this is a message');
        const signature = await contextSigner.sign(ctx, message);
        expect(oasis.misc.toHex(signature)).toEqual('304502210080e4f0e7a94baea44dcded16fb543642291aad5d7ab77606f4fce4cfc5b119260220010c0b40a09803494d7339bb52fb044028c9a58b0964610d863a50274264bd18');
        expect(await oasisRT.signatureSecp256k1.verify(ctx, message, signature, pk)).toEqual(true);
    });
});

gotta sort out the global crypto.getRandomValues in jest

adding this to the test :upside_down_face:
import {webcrypto} from 'crypto';

if (typeof crypto === 'undefined') {
    // @ts-expect-error there are some inconsequential type differences
    globalThis.crypto = webcrypto;
}

other projects using Node will have to do similar patches?

ts-web targets web only, it's based on grpc-web, which internally uses XMLHttpRequest
I think our last recommendation was to use --experimental-global-webcrypto. see more hacks here https://github.com/oasisprotocol/oasis-sdk/blob/main/client-sdk/ts-web/core/playground/e2e-test-nodejs.js

web wallet builds successfully with this version and minor change:
diff --git a/src/app/lib/transaction.ts b/src/app/lib/transaction.ts
index ff834281..c46f2cc1 100644
--- a/src/app/lib/transaction.ts
+++ b/src/app/lib/transaction.ts
@@ -19,7 +19,7 @@ export const signerFromPrivateKey = (privateKey: Uint8Array) => {
 }

 export const signerFromEthPrivateKey = (ethPrivateKey: Uint8Array) => {
-  return oasisRT.signatureSecp256k1.EllipticSigner.fromPrivate(ethPrivateKey, 'this key is not important')
+  return oasisRT.signatureSecp256k1.NobleSigner.fromPrivate(ethPrivateKey, 'this key is not important')
 }

 /** Transaction Wrapper */


Add 7zip integration

What's the Problem?
To compress/extract formats other than zip, you need to install external plugins. It is not possible to browse these formats in Files.
Solution/Idea
We could use the free library sevenzip to offer other formats (exploration/compression/extraction). This will then work without installing any plugins.
For information, we find this kind of UI in Linux.

Alternatives
Use another library.
Priorities

Capability
Priority

This proposal will allow developers to accomplish W
Must

This proposal will allow end users to accomplish X
Should

This proposal will allow developers to accomplish Y
Could

This proposal will allow end users to accomplish Z
Won't

Files Version
No response
Windows Version
No response
Comments
No response

@cinqmilleans we have a branch with 7zip integration but there are a couple of issues with it so it hasn't been merged yet.

Done #9633


Add missing resource requests for Azure CAPI provider

Noticed this when working on
https://issues.redhat.com/browse/HOSTEDCP-243

What this PR does / why we need it:
Which issue(s) this PR fixes (optional, use fixes #<issue_number>(, fixes #<issue_number>, ...) format, where issue_number might be a GitHub issue, or a Jira story:
Fixes #
Checklist

[ ] Subject and description added to both, commit and PR.
[ ] Relevant issues have been referenced.
[ ] This change includes docs.
[ ] This change includes unit tests.

/lgtm


New theme: Dracula for Mathematica/Wolfram Notebooks

Hi, team! I implemented the Dracula theme for Wolfram Notebooks 😄 you can find it here:
Here's a preview for how it works:

If you have a Wolfram desktop application, you can try it out interactively; it's example.nb in the repo.
Additionally, I created an automated installation method for Wolfram Language users by using the stylesheet as a ResourceFunction.

Ever since I made this, I made it my default environment for when I'm working with Wolfram Language code, so I use it everyday! Hope y'all like it.

Hey @jldohmann!
I'm glad to see that you created a theme for Wolfram Notebooks!
Are you interested in transferring your repository to the official organization (https://github.com/dracula)? You'll continue maintaining this project, the only difference is that we can make it more visible and include it on the site (https://draculatheme.com).
I just sent you an invite to join the organization. If you accept, you'll be able to transfer the repo and we can continue with adding to the site.
Thanks for building it!

Thanks @zenorocha ! I just transferred it over! https://github.com/dracula/wolfram-notebooks

https://draculatheme.com/wolfram-notebooks 🎉 🎉 🎉


New command: planner task reference list

Usage
planner task reference list
Description
Lists the references of a Planner tasks.
Options

Option
Description

-i, --taskId <taskId>
ID of the task.

Additional Info
Docs: Get plannerTaskDetails

I also love this command 😍 thanks for adding this suggestion 👍. It seems very sweet and would be nice to have it to be consistent with the add and remove commands.
I only have similar command here like in the checklistitem list that internally when developing this we could use the already existing task-details-get.ts command.
Wouldn't you agree?

Good suggestion @Adam-it: it makes perfect sense to reuse the existing command to simplify development and reuse code that we already have.

@Jwaegebaert I suggest here we take similar approach like in the checklistitem command 😉. Do you plan to take this one or we should leave this open for a friendly contributor ?

Well, I was just planning on saying that this may also be opened up to the community (if everything was in order) 😄

sure thing 😉.
Once again thanks for all you effort in creating those issues 👍and all your work. You rock 🚀

@waldekmastykarz I've discussed this with @Jwaegebaert and would like to help on this issue. Please note that this is my first contribution in the CLI, so it might take a while 😄

Awesome @MathijsVerbeeck! Looking forward to the PR, and please, don't hesitate to reach out if you have any questions.


Getting custom datatype from packet in android AAR

I have written a custom calculator and would like to extract the packet data in java on android.  The datatype is <vector<vector>>.
I am looking for an equivalent to the ios "packet.get"
I would like to do something like this in android:
processor.addPacketCallback(
(packet) -> {
packet.Get<std::vector<std::vector>>();
}
)

I have solved this by editing the jni files and java files in the repo.


link-to-changelog-file: Exclude non-textual files

Closes #4577 (following https://github.com/sindresorhus/refined-github/issues/4577#issuecomment-881951763)
Test URLs
Match: (as before)
https://github.com/pyca/cryptography/releases (CHANGELOG.rst)
https://github.com/nodeca/js-yaml/releases/tag/4.0.0  (CHANGELOG.md)
Not match:
https://github.com/vercel/hyper/releases  (release.js)
Screenshot
(none)
Notes to reviewers

The markdown extension regex is from: vertical-front-matter / view-markdown-source
/\.(mdx?|mkdn?|mdwn|mdown|markdown|litcoffee)$/

I added to it: |txt|rst to maintain matches from the previous related issue examples.

I noticed that findChangelogName() was initially returning string or false, that's why the lengthy:
function findChangelogName(files: string[]): string | false {
	const filename = files.find(name => regexChangelogMarkdownTextFiles.test(name))!;
	const result = (typeof filename === 'undefined') ? false : filename;

	return result;
}

Otherwise, if returning false can be omitted, I could replace it with:
function findChangelogName(files: string[]): string{
	return files.find(name => regexChangelogMarkdownTextFiles.test(name))!;
}

Thanks @darkred! I reverted it to false because undefined clears the cache and '' isn't a good falsey value.


⚠ Bofrost.at has degraded performance

In 2ac01b8, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1163 ms

Resolved: Bofrost.at performance has improved in 37c3c46.


Image Preprocessing for stated top5 accuracy

What image preprocessing was performed on the imagenet images to achieve the stated feedforward top5 accuracy?
E.g. resize uniformly to 256 at the smallest dimension, then center crop
I've had a tough time figuring this out, and any help would be much appreciated.
Many thanks!

Hi, that will be great.
I cannot find this in the paper. Can authors please respond. I am trying to implement and train it from scratch in Tensorflow, but I am not sure what strategy did you used.

I found this information specified in caffe training file here:
https://github.com/DeepScale/SqueezeNet/blob/master/SqueezeNet_v1.0/train_val.prototxt
  transform_param {
    crop_size: 227
    mean_value: 104
    mean_value: 117
    mean_value: 123
}


Post record optimisation

From personal experience it's a tall ask to record in one step on low performance computers.
Suggest add option to post-optimise the original 'uncompressed' file recorded
Here is what is used (...before finding ffscreencast)

i record in as efficient a format as I could find for low performance cpu AND _drive. i could be wrong about libx264 and rgb24 usage.
i also record only 640x480/vga/sntsc at 20 fps then reduce to 10
18MB -> 1.1MB
libopus support for mono 16-20 kbps is very efficient for voiceovers
i've also been looking for a way to record 'faked' mouseclick sounds and keypresses into a channel so it's not using the microphone to record the typing. could use same solution to show modifier and function keys pressed on overlay see https://github.com/wavexx/screenkey

#my 60 second test record
ffmpeg -f x11grab -framerate 20 -video_size 640X480 -i :0.0 -c:v libx264rgb -pix_fmt rgb24 -tune animation -crf 0 -preset ultrafast -t 60  raw.mp4
one minute is about 16mb
#my_ re-compress, oddly -crf 35 still looks good.
ffmpeg -i raw.mp4 -c:v libx264rgb -r 10 -preset veryslow -crf 35 -tune animation -movflags faststart compressed.mp4

Hi @cdplayer ,
thanks for the input. I am still trying to find the best balance between compression (size) and performance on old cpu's.
However, when you want to adjust the way it records, you can simply use --sargs and --oargs.
To verify your command without executing it, simply add --dry (which will only show what would be executed).
I could almost mimic your command with the exception of the resolution as this is auto-picked according to your screen resolution.
$ ffscreencast --oargs="-pix_fmt rgb24 -tune animation -crf 0 -preset ultrafast -t 60" \
--sargs="-framerate 20" --dry

/usr/local/bin/ffmpeg -hide_banner -loglevel info -thread_queue_size 512 \
-f avfoundation -video_size 2880x1800 -framerate 20 -i "1" -c:v libx264 \
-pix_fmt rgb24 -tune animation -crf 0 -preset ultrafast -t 60 -threads 0 \
"/Users/cytopia/Desktop/Screencast 2016-08-26 at 21.48.55.mkv"

Also note that you can add your presets to ~/.config/ffscreencast/ffscreencastrc so you can start it without any arguments and it will use the defined defaults.
Post processing is a neat topic as well, however i would not integrate it, but separate it into a different tool, which could output videos based on presets, including nice gif versions.

Things i've found

some record format like huffyyuv take a load off the processor, BUT add a huge load to disk writing time and space. ffvhuff is slighty better or preffered by ffmpeg.

documentation of ffvhug, rawvideo, huffyyuv is quite poor especially in terms of disk space usage and also how each effects cpu usage. the '-crf 0 -preset ultrafast' to create the intermediate file does appears to be good for performance and disk space usage.

further understanding of the -tune options may lead to better efficiencies. I understand you can compound these commands.-tune animation -tune zerolatency

libx264rgb and libx264 are different. Mostly people use libx264 but is focus towards video (yuv colorspace)  as apposed to displays(rgb/rbg). screencasts (without camera overly) at least seem best to use libx264rgb/rgb24 and pixel formats other than YUV.

With x264 codecs being so efficient and widely used i was wondering if there are other codecs (other than x264) that  ffmpeg supports that is as good as or better for highly efficient playback of screen recordings. My work so far has mostly been around finding most efficient and most compatibly settings for playback with x264 based playback facilities.

I was thinking of having a package type recording being made that had video and audio and camera in seperate files. Was also looking for ways you could add subtext/subtitles to the video (maybe by hand). I'm away of youtubes ability to add this (sort of automatically).

So I would suggest we pick a couple of potential defaults for initial recording, compile a nice way for benchmarking them on different machines and ask people to help collect benchmark data on that.
Best would be to have a standalone Benchmark.md page with all results for different hardware.
Once done I could integrate different presets into ffscreencast for easier recording.
Tasks that would still have to be done is

Post processing (in another script)
Separating input videos into different streams (which I find quite nice)

@cdplayer I added some post-processing scripts in contrib/ via #19


oauth2callback endpoint provided by flask extension has an XSS vulnerability.

Due to a lack of sanitisation in the error parameter, it is possible for arbitrary code to be reflected on the oauth2callback endpoint (this vulnerability is referenced as OTG-INPVAL-001 in the OWASP Testing Guide).
Using the default setup of the contrib flask extension:
oauth2 = oauth2client.contrib.flask_util.UserOAuth2(app)
The /oauth2callback endpoint then allows for an XSS attack like so:
/oauth2callback?error=a%3C!%27/*!%22/*!\%27/*\%22/*--!%3E%3C/Title/%3C/script/%3E%3CInput/Autofocus/%0D*/Onfocus=confirm(document.domain)//%3E%3Csvg%3E

We had this reported as a security issue on all our services that make use of oauth2client, requiring an urgent switch to a different authentication method.

@jkrukoff any recommendations on resolving this? Would it be enough for us to pass it through markupsafe?
In the meantime, you're welcome to subclass UserOAuth2 and override the method used to proecess the response.

My understanding is that it'd be sufficient to HTML escape the value there. I haven't investigated thoroughly, just passing along a replicated report upstream.


Cluster Resizing Proposal

Description
Phase one of cluster management provides the ability to adjust the size of the cluster
(by adding or removing one or more nodes) even if it requires us to implement a
cluster-wide lock that is in place while the resizing occurs.
Scenario
We have a 4-node cluster with a replication factor of 2 made up of hosts:
[
    "<IP_ADDRESS>:10101",
    "<IP_ADDRESS>:10101",
    "<IP_ADDRESS>:10101",
    "<IP_ADDRESS>:10101"
]

The cluster contains the following indexes:
idx0  (maxSlice = 10)
idx1  (maxSlice = 17)

Based on the Pilosa hash algorithm, the slice-to-node allocation looks like the
following (with the slices in parenthesis representing replicas):
node0: idx0 [3, 7, (2, 6, 10)]
node1: idx0 [0, 4, 8, (3, 7)]
node2: idx0 [1, 5, 9, (0, 4, 8)]
node3: idx0 [2, 6, 10, (1, 5, 9)]

node0: idx1 [0, 4, 8, 12, 16, (1, 5, 9, 13, 17)]
node1: idx1 [3, 7, 11, 15, (0, 4, 8, 12, 16)]
node2: idx1 [2, 6, 10, 14, (3, 7, 11, 15)]
node3: idx1 [1, 5, 9, 13, 17, (2, 6, 10, 14)]

We want to add two nodes to the cluster. To do so, we perform the following steps:

launch two new Pilosa servers with the correct configuration (i.e. the hosts list should contain the original 4 nodes followed by the 2 new nodes) with a bootstrap flag.

The bootstrap flag is currently not supported, so we need to implement it. Starting a node in bootstrap mode will immediately put the node into state maintenance. The nodes will need to join the gossip pool, but they shouldn't respond to any mutation messages (like add index/frame).

on the existing cluster, call the following endpoint: /manage/cluster/add-nodes -d '{hosts: ["<IP_ADDRESS>:10101", "<IP_ADDRESS>:10101"]}'
the node that receives the add-nodes call is now the coordinator of this request and will ensure that the following steps occur:

broadcast to the cluster that we are in maintenance mode (effectively locking the cluster). put all nodes into a state of maintenance

the operation should fail (and revert back to "available") if we can't get a confirmation that all nodes are locked
anti-entropy process should be paused while in maintenance

force anti-entropy to run once to be sure that replica data are in sync with primary data
trigger a flush method prior to file transfer (possibly call snapshot)
calculate the resize plan
broadcast the plan to all nodes (including the new nodes)

upon receiving the resize plan, each node should start requesting the fragments that it needs
when each node has successfully gotten its data, it needs to respond back to the coordinator that it has all the data.

add nodes 5 and 6 to the cluster (in memory)

broadcast a message to all nodes with the updated cluster node list

we can now safely delete the unused data files
remove the cluster lock (i.e. change node states back to available)

The new slice-to-node allocation will look like this (with the slices in parenthesis representing replicas):
node0: idx0 [1, (0, 2)]
node1: idx0 [10, (1)]
node2: idx0 [7, 9, (10)]
node3: idx0 [4, 6, 8, (7, 9)]
node4: idx0 [3, 5, (4, 6, 8)]
node5: idx0 [0, 2, (3, 5)]

node0: idx1 [4, 8, 10, (9, 15)]
node1: idx1 [5, 7, 11, (4, 8, 10)]
node2: idx1 [0, 6, 16, (5, 7, 11)]
node3: idx1 [1, 3, 13, 17, (0, 6, 16)]
node4: idx1 [2, 12, 14, (1, 3, 13, 17)]
node5: idx1 [9, 15, (2, 12, 14)]

Endpoints
/manage/cluster/add-nodes
Use this endpoint to add nodes to an existing cluster. This will put all nodes in the cluster into
state maintenance until the entire resize is complete.
Add Nodes:
POST /manage/cluster/add-nodes -d "{hosts: ["<IP_ADDRESS>:10101", "<IP_ADDRESS>:10101"]}"

Return: ??

/manage/cluster/remove-nodes
Use this endpoint to remove nodes from an existing cluster. This will put all nodes in the cluster into
state maintenance until the entire resize is complete.
Remove Nodes:
POST /manage/cluster/add-nodes -d "{hosts: ["<IP_ADDRESS>:10101", "<IP_ADDRESS>:10101"]}"

Return: ??

Broadcast Messages
We need to add support for the following broadcast messages:

SetNodeStateMessage: put all nodes into maintenance mode
TriggerAntiEntropyMessage: force anti-entropy to run
TriggerFlushMessage: trigger a flush (or call snapshot)
ResizePlanMessage: contains the resize plan
NodeResizeCompleteMessage: node signals that it has completed all of its resize operations (and has all the data files that it needs)
ClusterNodesMessage: broadcase the updated list of nodes in the cluster (based on this, each node should update their Nodes list that is registered in memory)
SetNodeStateMessage: put all nodes back to available (at the same time, we could delete any data files that are no longer used, or we could make that a completely separate message)

Enforce that the replication factor is >= 2 on all clusters, or display a warning on startup.
I wanted to clarify the joining cluster state.  When adding a node the new node in the a joining state while existing nodes are in a maintenance state.  Thus the existing nodes can still respond to some requests.
We should define functional access groups for our API endpoints.

Maintenance can read but not write
Bootstrap/Joining can neither read or write
The Query will be a special case and will need to be evaluated after handling the request
During the joining process we also need to transfer
Attribute date
Input definitions

I will break this out into an Epic with issues.


Bootnode flag on windows not interpreted correctly

On Windows, passing the --bootnode /ip4/<IP_ADDRESS>/tcp/7070/p2p/16Uiu2HAkuZirDbGvhEVc5TF3QS77rpwx6ryz9pUp7qTEqprD5DLZ was not interpreted correctly.
This issue was discovered at the testing session by @metacertain

@metacertain , could you please verify whether this bug still persists? If it persists, it would be nice if you could also fix it. If it doesn't, you may close the issue

@metacertain , could you please look into this, closing it perhaps if not reproducible. Thanks

Yes, I can confirm this is still an issue. However, it can be easily worked around by providing the option via a config file rather than a command parameter.
Because this is an environment specific issue that can be worked around, I suggest this to be handled with a FAQ item

This does not look like a bug. Enclosing the bootnode address in quotes should resolve the issue. Closing this


Next problem

All work but after use command "cargo run --release" the following happens:
...
Compiling ethabi v17.2.0
Compiling ethers-core v0.17.0 (https://github.com/gakonst/ethers-rs#454449b0)
Compiling ethers-contract-abigen v0.17.0 (https://github.com/gakonst/ethers-rs#454449b0)
Compiling ethers-providers v0.17.0 (https://github.com/gakonst/ethers-rs#454449b0)
Compiling ethers-signers v0.17.0 (https://github.com/gakonst/ethers-rs#454449b0)
Compiling ethers-etherscan v0.17.0 (https://github.com/gakonst/ethers-rs#454449b0)
Compiling ethers-addressbook v0.17.0 (https://github.com/gakonst/ethers-rs#454449b0)
Compiling ethers-solc v0.17.0 (https://github.com/gakonst/ethers-rs#454449b0)
Compiling ethers-contract-derive v0.17.0 (https://github.com/gakonst/ethers-rs#454449b0)
Compiling ethers-contract v0.17.0 (https://github.com/gakonst/ethers-rs#454449b0)
Compiling ethers-middleware v0.17.0 (https://github.com/gakonst/ethers-rs#454449b0)
Compiling ethers v0.17.0 (https://github.com/gakonst/ethers-rs#454449b0)
Compiling mev-template v0.1.0 (/home/test/projects/mev-template-rs)
Finished release [optimized] target(s) in 8m 30s
Running target/release/mev-template
Calling allPairsLength from 0x152e…1be3
~ [PASS] Total pairs: 23461
test@-virtual-machine:~/projects/mev-template-rs$
As far as I understand there should be monitoring - but there is not.
I apologize for the stupid question - thanks to your programs I started to learn Rust and still do not fully understand the logic of the program.
Do I need to specify any other parameters besides those specified in the env file?

you need a websocket to run the mempool streaming

ws in env file -
PRIVATE_KEY=privatekey
NETWORK_RPC=https://greatest-fittest-panorama.discover.quiknode.pro/0000000000000000000000/
NETWORK_WSS=wss://greatest-fittest-panorama.discover.quiknode.pro/0000000000000000000000/
DISCORD_WEBHOOK=https://discordapp.com/api/webhooks/000000000000000000/000000000000000000000000000000000000
No error messages at all. It builds and runs successfully. Then just a message appears:
Calling allPairsLength from 0x152e...1be3
~ [PASS] Total pairs: 23461
and the program finishes. That is, as far as I understand - it collects information about pairs and that's all, further monitoring does not happen

You need to bring in the updates from the repo into your current repo

Thanks


Adding new msg callout modifier, adding blue arrow

Creates a new message callout variation that sits horizontally above the thing it is pointing with no rotation. Also adds a .-blue modifier and arrow.
@DoSomething/front-end

Looks good to me! :ship:


Webpack HMR does not refresh on HTML changes.

I've been seeing an issue with Webpack, both 1.x and the 2.0 beta, wherein the hot module reloader doesn't refresh after to HTML templates. Webpack sees the change and logs the update, but the webpack/hot/dev-server module doesn't trigger a refresh.
I did a little amateur sleuthing, and it looks like using a subcompiler prevents the Webpack hash from being updated:

When the hot reloader compares the new compilation hash against the old one (https://github.com/webpack/webpack/blob/master/hot/dev-server.js#L9), it just sees that the global compilation hasn't been updated and takes no action.
For the moment I'm making do with a slightly patched version of the HMR module that always returns false from check(). Sadly I don't have a sufficient understanding of Webpack's deep magic to dive in and fix this, but it seemed like something you might want to keep in your backlog.
Thanks for all your hard work on this plugin!

open for pull requests

Is there any further information about this issue? It'd really be a convenience if pages update automatically when template changes. 😏

I guess it's not possible - if so feel free to open a pull request

@jantimon Is this something that should be in the webpack-dev-server repo? Or is this a html-webpack-plugin related limitation? Just tracking.

So far, I've approached this by publishing a custom event on webpack-hot-middleware, and consuming that event in client JavaScript, as in https://github.com/HenrikJoreteg/hjs-webpack/pull/247 This approach isn't my idea, I originally saw it in this comment. As far as I can tell, it's not possible to live reload HTML without something of a hack, but it seems like a relatively benign hack that could possibly lead to a proper approach.
It seems like webpack-dev-server or webpack-hot-middleware are more appropriate to be concerned with this functionality than html-webpack-plugin, but I don't have a good sense of what API they would expose to trigger reloads in the browser.


Fixed missing np. in front of sqrt().

A tiny NumPy error showed up in class today, so I need to make this request.

New release will be out shortly.


Fix #33436

All baseline changes are either:

rewording of but 'T' could be instantiated with a different subtype of constraint '{}' to use constraint unknown.
Additional error elaboration of 'void' is assignable to the constraint of type 'T', but 'T' could be instantiated with a different subtype of constraint 'unknown'. I guess because void is not related to {}, but it is related to unknown.

https://github.com/microsoft/TypeScript/pull/30637/files?file-filters[]=.ts
The above PR did not modify noConstraintType.
I looked over it quickly but it looks like noConstraintType is some kind of sentinel type?
So, while the constraint is unknown in reality now, the error messages still use the sentinel type and use {} in its output?
And this PR is just updating the sentinel type to match the reality?
Sorry if I sound incoherent =(

The error message is a consequence of the fact that noConstraintType is defined as an empty object type, but the error message logic doesn't directly reference noConstraintType, it just looks at the constraint type.
In other words, the error message logic does not look for the sentinel specifically, so it wasn't clear to me whether other code would be the same. I don't know what else uses noConstraintType, or how it uses it, so I wasn't sure if there would be any change in type checking semantics prior to this PR.
The default constraint should be unknown, but it seems there may be a few things that still need tidying up. I'm still not sure whether #32814 is influenced by a legacy implicit constraint.

Right, noConstraintType is just supposed to be a sentinel (the getConstraintFromTypeParameter and friends functions all check for it to know when to return undefined), but if it is leaking into error messages (or comparisons!?!), changing it is pertinent. The only reason it wasn't changed was because it didn't seem like it needed to be, since "no constraint" isn't actually supposed to be a type.

OK, so this error changes because reportRelationError accesses target.immediateBaseConstraint directly - it should then treat an undefined target.immediateBaseConstraint the same as a noConstraintType target.immediateBaseConstraint.

So reportRelationError should probably use getBaseConstraint instead?
From a brief look I don't think it is leaking into comparisons, at least judging by the direct accesses to type.immediateBaseConstraint in checker.ts.
Given the churn on baselines, should we leave this until the outcome of #33436 is finalised? I wasn't sure if that was still in discussion.

I think #33436 is pretty directly an artifact of not checking for noConstraintType while using type.immediateBaseConstraint directly. I think we should just fix it by checking for the not base constraint type the same way we check for undefined. We can also make the noConstraintType another unknown instead of an object, just in case, but it shouldn't be required, strictly speaking.

So reportRelationError should probably use getBaseConstraint instead?

Yeah, I think so.

If noConstraintType is just used as a sentinel, does this mean T extends {} will be interpreted as having no constraint?
Then again why do we need a sentinel at all?  It seems like if <T> and <T extends unknown> are meant to be equivalent, you could just plug in the unknown upfront and avoid the need for a sentinel value entirely?

does this mean T extends {} will be interpreted as having no constraint?

No. That's a T with a constraint of {}.

Okay then I’m confused.  The way you described it before sounded like “no constraint” was being represented internally as T extends noConstraintType and if noConstraintType is currently {}...

noConstraintType is always looked for and replaced with undefined by all callers except this error message, along with the circularConstraintType sentinel.

No, I get that - my question is, what makes noConstraintType (as it’s currently defined) actually distinguishable from a legit {}?  That’s the part I’m hung up on.

It's never supposed to be witnessed as a type. So nothing. It could be the number 4 - it's just a sentinel value. It happens to be a type (with no defined structure) right now, hence the mistake.

So... Better to leave it as {} or change it to a type with a silly name like "you hecked up" and fix the bug causing the sentinel value to be "witnessed"?
Was there a reason to not just use undefined as the sentinel?
Just curious =x

Was there a reason to not just use undefined as the sentinel?

Probably to distinguish between types known to have no constraints, and types with lazy constraints not computed yet.
I updated the other sentinels, just because, but I'm happy to revert those changes. Most of the baselines look fine. I marked a couple I'm not sure about.

Was there a reason to not just use undefined as the sentinel?

undefined is used for the "has not yet been calculated and cached yet" value. :V

Why was this closed? o.0

I don't think there was an associated issue and I'm just doing my bit to keep the random open PR count down. Can always reopen if there is need!

Isn't #33436 the associated issue? =x

Yes, you're right.

@DanielRosenwasser Closing this in favor of #35225 as I messed up some merges.


harquelin opens successfully, but unable to interact with the interface

Hello,
I'm experiencing an issue with Harlequin on Windows. The application opens successfully, but I'm unable to interact with the interface to write scripts. Here are the details:
Harlequin launches and the interface appears.
However, I cannot type or input any text in the script writing area.
The application seems to be running, but it's not responding to keyboard input for script writing.
I've already taken the following steps to try to resolve the issue:
Reinstalled Harlequin using conda
Updated all packages in my Conda environment
Verified that SQLite is installed and up-to-date
Checked that I can import sqlite3 directly in Python without errors
My system details:
Operating System: Windows 11
Python version: 3.12.5 (for miniconda3)
Harlequin version: harlequin, version 1.25.0 Installed Adapters:

duckdb, version 1.25.0
sqlite, version 1.25.0
Conda version: 24.9.1
Could you please provide guidance on how to troubleshoot this issue? Are there any known conflicts or configuration settings that might prevent text input in Harlequin?
Thank you for your assistance.

Charles-André Coste

I'm sorry this isn't working for you. Please use the bug template next time.
What terminal and shell are you using in Windows?

Hi! For harlequin I use windows Terminal with Powershell latest version.
Thanks for your help.
@.***
Le 14 oct. 2024 à 18:34 +0200, Ted Conbeer @.***>, a écrit :

I'm sorry this isn't working for you. Please use the bug template next time.
What terminal and shell are you using in Windows?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

Before Proceeding, please acknowledge:

[X] I have consulted the Troubleshooting guide.
[X] I have searched Issues and Discussions in this repo.
[X] Feature requests should be initiated as Discussions. This is a bug report.

Describe the bug
Completely new user. Harlequin worked fine earlier today, now it hangs on startup.
I started the app with harlequin. The app starts, I see the five dots in 'Data Catalog' and 'Query Editor', only four of the dots are filled. 'Query Results' says "Database connected".
I am unable to interact with the app in any way, neither mouse clicks nor keyboard.
To Reproduce
Install Harlequin on Windows Terminal, with pipx, start Harlequin either with unnamed in-memory database or with a named duckdb database file.
Expected behavior
I expected a window where I could interact with the query editor and run SQL
Actual behavior
Copied from Describe the bug:
The app starts, I see the five dots in 'Data Catalog' and 'Query Editor', only four of the dots are filled. 'Query Results' says "Database connected".
I am unable to interact with the app in any way, neither mouse clicks nor keyboard.
Screenshot of my terminal

Contributing
Are you interested in contributing a fix?

[ ] Yes
[ ] Maybe
[X] No

Additional context
N/A
What is the output of harlequin --version?
harlequin, version 1.25.0

Installed Adapters:
  - duckdb, version 1.25.0
  - sqlite, version 1.25.0

What database adapter are you using with Harlequin? (Default is duckdb)
Only duckdb
What other options are you using when invoking Harlequin? (If you are using a profile, please add relevant items from your profile here).
No profile
Problem happens with or without a named database. I.e. both harlequin and harlequin foo ends up with an unresponsive interface.
Can you tell us more about your system?

Shell: Powershell 7.4.5
Terminal: Windows Terminal Version: 1.21.2361.0
OS of the shell: Windows 11
OS of the terminal (if different from the shell):
Default locale / language for your OS/Terminal/Shell: en-US/UTF-8

[!TIP]
For example, for my system, these are:

Bash
Windows Terminal
Ubuntu 22.04 / WSL2
Windows 11
en_US/UTF-8

How did you install Harlequin?

[X] pipx
[ ] conda/mamba
[ ] pip (global install)
[ ] pip + venv/virtualenv or pipenv
[ ] Poetry
[ ] uv/Rye
[ ] other: _____________

Before Proceeding, please acknowledge:

I have consulted the Troubleshooting guide
https://harlequin.sh/docs/troubleshooting/index.
I have searched Issues and Discussions in this repo.
Feature requests should be initiated as Discussions. This is a bug
report.

Describe the bug
Completely new user. Harlequin worked fine earlier today, now it hangs on
startup.
I started the app with harlequin. The app starts, I see the five dots in
'Data Catalog' and 'Query Editor', only TWO of the dots are filled. 'Query
Results' don't says "Database connected".
I am unable to interact with the app in any way, neither mouse clicks nor
keyboard.
To Reproduce
Install Harlequin on Windows Terminal, with pipx, start Harlequin either
with unnamed in-memory database or with a named duckdb database file.
Expected behavior
I expected a window where I could interact with the query editor and run SQL
Actual behavior
Copied from Describe the bug:
The app starts, I see the five dots in 'Data Catalog' and 'Query Editor',
only TWO of the dots are filled. 'Query Results' don't says "Database
connected".
I am unable to interact with the app in any way, neither mouse clicks nor
keyboard.
Screenshot of my terminal ===>>>>> see attachment
image.png (view on web)
https://github.com/user-attachments/assets/f4e6f462-75ca-46fa-9eb4-ce0b967583c2
Contributing
Are you interested in contributing a fix?

Yes
Maybe
No

Additional context
N/A
What is the output of harlequin --version?
harlequin, version 1.25.0
Installed Adapters:

duckdb, version 1.25.0
sqlite, version 1.25.0

What database adapter are you using with Harlequin? (Default is duckdb)
Only duckdb
What other options are you using when invoking Harlequin? (If you are using
a profile, please add relevant items from your profile here).
No profile
Problem happens with or without a named database. I.e. both harlequin
and harlequin
foo ends up with an unresponsive interface.
Can you tell us more about your system?

Shell: Powershell 7.4.5
Terminal: Windows Terminal Version: 1.21.2361.0
OS of the shell: Windows 11
OS of the terminal (if different from the shell):
Default locale / language for your OS/Terminal/Shell: FR-US/UTF-8

Tip
For example, for my system, these are:

Windows Terminal
Windows 11
en_US/UTF-8 (french)

How did you install Harlequin?
Miniconda3

pipx
conda/mamba
pip (global install)
pip + venv/virtualenv or pipenv
Poetry
uv/Rye
other: _____________

—
Le mar. 15 oct. 2024 à 20:57, mogensNielsen @.***> a
écrit :

Before Proceeding, please acknowledge:

I have consulted the Troubleshooting guide
https://harlequin.sh/docs/troubleshooting/index.
I have searched Issues and Discussions in this repo.
Feature requests should be initiated as Discussions. This is a bug
report.

Describe the bug
Completely new user. Harlequin worked fine earlier today, now it hangs on
startup.
I started the app with harlequin. The app starts, I see the five dots in
'Data Catalog' and 'Query Editor', only four of the dots are filled. 'Query
Results' says "Database connected".
I am unable to interact with the app in any way, neither mouse clicks nor
keyboard.
To Reproduce
Install Harlequin on Windows Terminal, with pipx, start Harlequin either
with unnamed in-memory database or with a named duckdb database file.
Expected behavior
I expected a window where I could interact with the query editor and run
SQL
Actual behavior
Copied from Describe the bug:
The app starts, I see the five dots in 'Data Catalog' and 'Query Editor',
only four of the dots are filled. 'Query Results' says "Database connected".
I am unable to interact with the app in any way, neither mouse clicks nor
keyboard.
Screenshot of my terminal
image.png (view on web)
https://github.com/user-attachments/assets/f4e6f462-75ca-46fa-9eb4-ce0b967583c2
Contributing
Are you interested in contributing a fix?

Yes
Maybe
No

Additional context
N/A
What is the output of harlequin --version?
harlequin, version 1.25.0
Installed Adapters:

duckdb, version 1.25.0
sqlite, version 1.25.0

What database adapter are you using with Harlequin? (Default is duckdb)
Only duckdb
What other options are you using when invoking Harlequin? (If you are
using a profile, please add relevant items from your profile here).
No profile
Problem happens with or without a named database. I.e. both harlequin and harlequin
foo ends up with an unresponsive interface.
Can you tell us more about your system?

Shell: Powershell 7.4.5
Terminal: Windows Terminal Version: 1.21.2361.0
OS of the shell: Windows 11
OS of the terminal (if different from the shell):
Default locale / language for your OS/Terminal/Shell: en-US/UTF-8

Tip
For example, for my system, these are:

Bash
Windows Terminal
Ubuntu 22.04 / WSL2
Windows 11
en_US/UTF-8

How did you install Harlequin?

pipx
conda/mamba
pip (global install)
pip + venv/virtualenv or pipenv
Poetry
uv/Rye
other: _____________

—
Reply to this email directly, view it on GitHub
https://github.com/tconbeer/harlequin/issues/666#issuecomment-2414782846,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AQOONW7MGQI4EE47WZFLSGDZ3VQSXAVCNFSM6AAAAABP5AJ5DSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJUG44DEOBUGY
.
You are receiving this because you authored the thread.Message ID:
@.***>

Exactly the same happening here. I've been using harlequin on Windows 10 successfully for months.
Today I just installed in my new Windows 11 laptop. It worked the first time (I was able to run queries and everything) and then suddenly it all stopped working.
I see a python process eating CPU (and I'm not doing anything with Python right now):

While the UI is totally stuck:

And stays like this forever.
harlequin --version
harlequin, version 1.25.0

Installed Adapters:
  - duckdb, version 1.25.0
  - sqlite, version 1.25.0

conda --version
conda 24.9.1

python --version
Python 3.12.4

(Just installed latest miniconda today)

Nice digging @pablosantosluaces!
I can confirm that deleting cache-1.pickle makes Harlequin work again.
I played around some more:

Deleted cache-1.pickle
Running harlequin
Harlequin runs fine (do not write any SQL in the editor)
Quit (CTRL+Q) Harlequin --> cache-1.pickle is recreated
Running harlequin
Writes some SQL (select 1)
Quit (CTRL+Q) Harlequin
Running harlequin --> Frozen like before
Deleted cache-1.pickle
Kill the terminal window running Harlequin (in frozen mode)
No cache-1.pickle has been created
Running harlequin --> No problem
Quit (CTRL+Q) Harlequin
cache-1.pickle is recreated
Running harlequin --> No problem
Run a SQl (select 1 )
Quit (CTRL+Q) Harlequin
cache-1.pickle increased in size from 191 to 199
Running harlequin --> Frozen like before

It seems to me that the problem has to do with what happens in cache-1.pickle when I run a SQL in the editor and then quit Harlequin.
The file cache-1.pickle increases in size (191 to 199) and subsequent running of Harlequin results in a frozen app.

This duplicates #659

Thanks, but the bug is re-created after in a new session of harlequin, no fix for permanently ?


Fix duplicate daemon options

This is a workaround to fix https://github.com/chef/chef/issues/7737 in chef-client.
Signed-off-by: Joerg Herzinger<EMAIL_ADDRESS>Description
Daemon options appear only once in systemd unit files.
Issues Resolved
#594
Check List

[x] All tests pass. See https://github.com/chef-cookbooks/community_cookbook_documentation/blob/master/TESTING.MD
[x] New functionality includes testing.
[x] New functionality has been documented in the README if applicable
[x] All commits have been signed for the Developer Certificate of Origin. See https://github.com/chef-cookbooks/community_cookbook_documentation/blob/master/CONTRIBUTING.MD

is this is a real fix to remove duplicate options or is this dropping functionality here in this cookbook that other people need, because of a bug in chef?

Sorry for this confusion, and no, this does not drop any functionality at all. The daemon_options will still be there, but just not duplicated.
In the systemd unit resource you can see that the enviornments file gets included (https://github.com/chef-cookbooks/chef-client/blob/master/recipes/systemd_service.rb#L49) and in the environments file the daemon options are set anyway (https://github.com/chef-cookbooks/chef-client/blob/master/templates/default/redhat/sysconfig/chef-client.erb#L14). So this just drops the duplicate daemon_options which just happened to cause a "bug", or more of a nuisance, for me.

cool, that's a better explanation of what is going on here... :+1:

Thanks @joerg


Some code is missing in blog entry

In the last paragraph ("Achieving good multitasking") of this blog entry the source code is missing:
https://textual.textualize.io/blog/2022/12/07/letting-your-cook-multitask-while-bringing-water-to-a-boil/

Thanks. Fixed.


pkgs/stdenv/linux/make-bootstrap-tools-cross.nix: allow entries for t…

…o-be uplaoded targets
Newly added targets should be fine to be added as hudra jobs before the first uplaod happens.
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

/cc @OPNA2608 who works on https://github.com/NixOS/nixpkgs/pull/255451
/cc @Artturin who added the initial wording on tarballs constraints


Roll Flutter Engine from 0b0fab821536 to 636b12c4e3e9 (1 revision)

https://github.com/flutter/engine/compare/0b0fab821536...636b12c4e3e9
2023-12-14<EMAIL_ADDRESS>Suppress warning for the new Activity library. (flutter/engine#49046)
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Please CC jsimmons@google.com,rmistry<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose
To report a problem with the AutoRoller itself, please file a bug:
https://issues.skia.org/issues/new?component=1389291&template=1850622
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

Commit queue failed; closing this roll.


Support for cgroups v2

I am looking into supporting cgroups v2.
However, I'm not an expert, so I'm looking for what I need, useful documentation, design suggestions, etc.
Also, if anyone would like to challenge me, I welcome you to challenge me with me.
References

opencontainers/runc#654
https://www.youtube.com/watch?v=P6Xnm0IhiSo&list=WL&index=58
https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html
https://github.com/opencontainers/runc/tree/master/libcontainer/cgroups/fs2

Sure, would be happy to help. How do you want to proceed @utam0k ? I thought about getting through the references and then attempting a spike of the cgroup v2 manager.

@Furisto
It is a very good idea as a first step. I think an eventual implementation like the crun about it would be nice.
https://github.com/containers/crun/blob/master/crun.1.md#cgroup-v2

I thought about getting through the references and then attempting a spike of the cgroup v2 manager.

I'd love to hear other people's opinions as well as mine, so please let me know what you think.
If you are interested, you can try to implement a cgroups v2 manager?

Progress can be seen here: https://github.com/Furisto/youki/tree/cgroupv2. I am currently fleshing out the cgroups v2 manager and 1-2 subsystems.

@Furisto
I took a look at your commit. As a first step it was exactly what I imagined. I think it's a good policy.
Can I assign this problem to you?
If it looks like it needs to be handled by a variety of controllers, I think it would be a good idea to provide an example like in v1 to give others a chance to try.

@utam0k
You can assign it to me. Yes, the idea is to do the first one or two myself and then the rest can be done by other people.

@Furisto cc: @tsturzl
I decided to make cgroups v2 support a key feature for the first release of youki.
I'm looking forward to it.

@utam0k @tsturzl
I am planning to have a PR ready the day after tomorrow

@Furisto Wow! I am amazed at how fast you work.

@tsturzl
Thanks for your hard work in supporting v2. It was very nice work!
BTW, I'd like to talk to you about how to proceed with v2 support for each subsystem.
I think it is better to proceed with the project in the same way as https://github.com/utam0k/youki/issues/9.
What do you think about it?

@utam0k
I think it is a good idea. I am investigating one open issue, once that is solved we can split up the work for the different subsystems. The issue is that the cgroup manager gets the pid of youki and not of the container process. If i do this
sudo docker run -d --rm -m 30m  --runtime youki progrium/stress --cpu 2
then the process tree looks like this:
root       13519       1  0 00:19 ?        00:00:00 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 509e2e3131d50c63d599a
root       13530   13519  0 00:19 pts/24   00:00:00  \_ /home/furisto/Projects/Existing/youki-me/target/x86_64-unknown-linux-gnu/d
root       13531   13530  0 00:19 pts/24   00:00:00      \_ /usr/bin/stress --verbose --cpu 2
root       13538   13531 99 00:19 pts/24   00:00:03          \_ /usr/bin/stress --verbose --cpu 2
root       13539   13531 99 00:19 pts/24   00:00:03          \_ /usr/bin/stress --verbose --cpu 2

The cgroup manager gets pid 13530, not 13531. We also seem to call apply too late during create, because 13538, 13539  and 13531 end up in the containerd.slice not in the cgroup created by youki. That must mean that when these processes are forked their parent is not yet in the correct cgroup.

@Furisto Hi! I think the problem mentioned above was solved by https://github.com/containers/youki/pull/69. So Can I create a tracking issue to implement some subsystems for cgv2?

@utam0k Yes, you can!

I created https://github.com/containers/youki/issues/78.
Let's continue with cgv2 support in this issue!

Special thanks to @Furisto


Refactor: Remove FsStorageProvider from Table Service

Summary
❗❗ The volume of this PR might be intimidating, please read through the description to make reviewing easier.
The primary goal of this PR is to remove FsStorageProvider from :services:tables (and additionally :tables-test-fixture)
To achieve this goal, we do:
1: Add additional methods to StorageClient interface
StorageClient {
   + String getRootPath()
   + String getEndpoint()
}

2: Add additional method to FileIOManager to reverse lookup a Storage for a FileIO
FileIOManager {
   + Storage getStorage(fileIO)
}

3: Map FsStorageProvider methods to StorageManager
# pattern 1
storageProvider.rootPath() ->
    storageManager.getDefaultStorage().getClient().getRootPath()

# pattern 2
storageProvider.endPoint() ->
    storageManager.getDefaultStorage().getClient().getEndPoint()

# pattern 3
storageProvider.storageClient() ->
    if(ensureHDFS) {
          storageManager.getDefaultStorage().getClient().getNativeClient()
    } else {
          throw exception or log
    }

4: Fix log for StorageType
Instead of printing objecthash, print the type properly, ie StorageType.Type(local)

👆  thats it
Is this the end state of refactoring? NO. The goal of the PR is limited to removing fsStorageProvider. In future PRs we'll address CTAS, fileSecurer, stripPrefix, defaultTableLocation etc, currently these have been tagged with TODO
Changes

[X] Refactoring
[X] Tests

For all the boxes checked, please include additional details of the changes made in this pull request.
Testing Done

[❕] Manually Tested on local docker setup. Please include commands ran, and their output.
[ ✅ ] Added new tests for the changes made.
[ ✅  ] Updated existing tests to reflect the changes made.
[ ✅ ] Some other form of testing like staging or soak time in production. Please explain.

Additional Information

[X] Deprecations
- remove fsStorageProvider
[X] Large PR broken into smaller PRs, and PR plan linked in the description.
- https://github.com/linkedin/openhouse/pull/100

cc: @ctrezzo