Android demo app crashes when using quantized model obtained from graph_transform tool?

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
Only edited ClassifierActivity.java to suit a custom model
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Android smartphone / host machine: Ubuntu 16.04 LTS
TensorFlow installed from (source or binary): Source
TensorFlow version (use command below): 1.1
Bazel version (if compiling from source): 0.4.5
CUDA/cuDNN version: 8.0/5.1
GPU model and memory: GTX 860M

Describe the problem
I am currently following the guide in: http://nilhcem.com/android/custom-tensorflow-classifier
to train a custom classifier. However, I am using my own frozen graph that I obtained from my own training. What I noticed was when I used the quantized graph obtained through the graph_transform method, the app simply crashes without even running. In the guide, it is recommended to run this command on the inference graph:
bazel-bin/tensorflow/python/tools/optimize_for_inference \
  --input=/tf_files/retrained_graph.pb \
  --output=/tf_files/retrained_graph_optimized.pb \
  --input_names=Mul \
  --output_names=final_result

While I think it may be in conflict with the quantized graph transformations, I ran this command to test, and the app did crash. Here are the observations for all 4 permutations I tested:

quantization + optimize_for_inference = app crash
quantization only = app crash
optimize_for_inference on frozen graph = app works
frozen_graph only = app works

So my conclusion is the quantization operations within the quantized graph caused the app to fail.
Here is the quantization command I ran:
/home/kwotsin/tensorflow-android/tensorflow/bazel-bin/tensorflow/tools/graph_transforms/transform_graph \
--in_graph=./frozen_model_mobilenet.pb \
--out_graph=./quantized_model_mobilenet.pb \
--inputs='Placeholder_only' \
--outputs='MobileNet/Predictions/Softmax' \
--transforms='
  add_default_attributes
  strip_unused_nodes(type=float, shape="1,299,299,3")
  remove_nodes(op=Identity, op=CheckNumerics)
  fold_constants(ignore_errors=true)
  fold_batch_norms
  fold_old_batch_norms
  quantize_weights
  quantize_nodes
  strip_unused_nodes
  sort_by_execution_order'

And the java file I edited to build my APK:
  private static final int INPUT_SIZE = 224;
  private static final int IMAGE_MEAN = 128;
  private static final float IMAGE_STD = 128;
  private static final String INPUT_NAME = "Placeholder_only";
  private static final String OUTPUT_NAME = "MobileNet/Predictions/Softmax";

  private static final String MODEL_FILE = "file:///android_asset/quantized_model_mobilenet.pb";
  private static final String LABEL_FILE =
      "file:///android_asset/mobilenet_labels.txt";

Other than these edit, every other file is the same. I then imported this project in Android Studio, and ran the build apk option located on the top toolbar of Android Studio.
So far, existing tutorials I've seen (e.g. those from Pete Warden) use the quantize_graph method to run on mobile devices. Is graph_transform compatible for quantizing models for mobile devices yet?

@asimshankar can you comment or redirect to someone who can? Thanks!

@kwotsin : Do you have any logs/stracktraces you can share from the crash? That will help in diagnosing the problem. Also, is it possible to share the graph that you are quantizing? Basically, I'm looking for detailed information on the error and/or the ability to reproduce the problem to help us diagnose.
FYI @suharshs @petewarden

@asimshankar Is there a way to see the error log if an app crashes? So far the only idea I have that it's not working is just that the app crashes. Because I can't find the error log, I can't seem to find the source of the problem as well.
I have attached the PB files for both the frozen and quantized model here. https://www.dropbox.com/sh/grp7xlxu4r59qss/AADV9imhKpY9auiNAkaSsjyMa?dl=0
I have noticed another problem as well: when I tried to predict custom labels with another model trained on the flowers dataset, my app crashes even if I'm not quantizing it. Could this problem be related to why quantization fails?

The errors should appear like any android app, see https://developer.android.com/studio/debug/am-logcat.html and https://developer.android.com/studio/command-line/logcat.html

Unfortunately I am currently unable to get my android device mounted on my computer, and I've tried asking on StackOverflow in this link some time ago: http://stackoverflow.com/questions/43996994/android-unable-to-mount-android-phone-although-lsusb-shows-device-and-after-ins
I am currently building the APK before transferring it onto dropbox to let my android phone download the app as a workaround.
Also, I realized there's one other java file I have to change: tensorflow/java/src/main/java/org/tensorflow/examples/LabelImage.java
As the labels and pb file names needs to be changed in this file as well, before building the APK. To keep things simple, I simply renamed the pb file name and the labels file name to be the same as what the demo app provided, and the frozen graph APK works fine. Once again, when I tried quantizing the graph, my app crashes. Is there a way to get the error log files without using ADB?

It will be hard to debug without access to debugging information from the android logs.
I don't think there is a way to get access to logs without connecting it to your computer, but maybe there is. That would be a question better suited to Android support forums.

@asimshankar I have finally solved the ADB issue by doing a factory reset of the android phone and then enabling PTP instead of MTP as the file transfer method.
From the Android Studio's android monitor, I have gotten these log when my app crashes using the quantized graph.
05-19 16:48:38.403 23564-23564/? I/art: Late-enabling -Xcheck:jni
05-19 16:48:38.509 23564-23564/? W/ActivityThread: Application org.tensorflow.demo can be debugged on port 8100...
05-19 16:48:38.609 23564-23564/? D/ContextHelper: convertTheme. context->name=org.tensorflow.demo themeResourceId=2131034118
05-19 16:48:38.611 23564-23564/? D/tensorflow: CameraActivity: onCreate org.tensorflow.demo.ClassifierActivity@7b0b069
05-19 16:48:38.614 23564-23564/? I/PhoneWindow: [generateLayout] setColorNavigationBar => color=0x ff000001
05-19 16:48:38.617 23564-23564/? D/PhoneWindowEx: [LMJ][PWEx][generateLayout] setNavigationBarColor2 : colors=0xff000000
05-19 16:48:38.617 23564-23564/? I/PhoneWindow: [setNavigationBarColor2] color=0x ff000000
05-19 16:48:38.704 23564-23564/? D/tensorflow: CameraActivity: onStart org.tensorflow.demo.ClassifierActivity@7b0b069
05-19 16:48:38.704 23564-23564/? D/tensorflow: CameraActivity: onResume org.tensorflow.demo.ClassifierActivity@7b0b069
05-19 16:48:38.722 23564-23597/? D/OpenGLRenderer: Render dirty regions requested: true
05-19 16:48:38.725 23564-23597/? I/Adreno-EGL: <qeglDrvAPI_eglInitialize:410>: EGL 1.4 QUALCOMM build:  ()
                                               OpenGL ES Shader Compiler Version: E<IP_ADDRESS>
                                               Build Date: 12/12/14 금
                                               Local Branch: LA.BF.1.1_RB<IP_ADDRESS>.002.031_20141212_01821442
                                               Remote Branch:
                                               Local Patches:
                                               Reconstruct Branch:
05-19 16:48:38.729 23564-23597/? I/OpenGLRenderer: Initialized EGL, version 1.4
05-19 16:48:38.763 23564-23597/? D/OpenGLRenderer: Enabling debug mode 0
05-19 16:48:38.777 23564-23564/? D/Atlas: Validating map...
05-19 16:48:38.890 23564-23564/? W/ArrayUtils: Ignoring invalid value mw_continuous-picture
05-19 16:48:38.891 23564-23564/? W/ArrayUtils: Ignoring invalid value emboss
05-19 16:48:38.891 23564-23564/? W/ArrayUtils: Ignoring invalid value sketch
05-19 16:48:38.891 23564-23564/? W/ArrayUtils: Ignoring invalid value neon
05-19 16:48:38.891 23564-23564/? W/ArrayUtils: Ignoring invalid value asd
05-19 16:48:38.891 23564-23564/? W/ArrayUtils: Ignoring invalid value backlight
05-19 16:48:38.891 23564-23564/? W/ArrayUtils: Ignoring invalid value flowers
05-19 16:48:38.891 23564-23564/? W/ArrayUtils: Ignoring invalid value AR
05-19 16:48:38.907 23564-23564/? I/tensorflow: CameraConnectionFragment: Valid preview sizes: [1920x1440, 1920x1080, 1776x1080, 1440x1080, 1280x960, 1280x720, 960x720, 880x720, 864x480, 848x480, 800x480, 720x480, 768x432, 640x480, 576x432, 480x320]
05-19 16:48:38.908 23564-23564/? I/tensorflow: CameraConnectionFragment: Rejected preview sizes: [384x288, 352x288, 320x240, 240x160, 176x144, 160x120]
05-19 16:48:38.908 23564-23564/? I/tensorflow: CameraConnectionFragment: Chosen size: 480x320
05-19 16:48:38.914 23564-23564/? W/ArrayUtils: Ignoring invalid value emboss
05-19 16:48:38.914 23564-23564/? W/ArrayUtils: Ignoring invalid value sketch
05-19 16:48:38.914 23564-23564/? W/ArrayUtils: Ignoring invalid value neon
05-19 16:48:38.915 23564-23564/? W/ArrayUtils: Ignoring invalid value asd
05-19 16:48:38.915 23564-23564/? W/ArrayUtils: Ignoring invalid value backlight
05-19 16:48:38.915 23564-23564/? W/ArrayUtils: Ignoring invalid value flowers
05-19 16:48:38.915 23564-23564/? W/ArrayUtils: Ignoring invalid value AR
05-19 16:48:38.936 23564-23564/? I/TensorFlowImageClassifier: Reading labels from: imagenet_comp_graph_label_strings.txt
05-19 16:48:38.944 23564-23564/? I/TensorFlowInferenceInterface: Checking to see if TensorFlow native methods are already loaded
05-19 16:48:38.949 23564-23564/? E/art: No implementation found for long org.tensorflow.contrib.android.RunStats.allocate() (tried Java_org_tensorflow_contrib_android_RunStats_allocate and Java_org_tensorflow_contrib_android_RunStats_allocate__)
05-19 16:48:38.950 23564-23564/? I/TensorFlowInferenceInterface: TensorFlow native methods not found, attempting to load via tensorflow_inference
05-19 16:48:39.065 23564-23564/? I/TensorFlowInferenceInterface: Successfully loaded TensorFlow native methods (RunStats error may be ignored)
05-19 16:48:39.879 23564-23564/? A/libc: Fatal signal 11 (SIGSEGV), code 1, fault addr 0x0 in tid 23564 (tensorflow.demo

I've not done any android development before, so I would really appreciate it if you could share what this error log means.
Also, does the image input size (299299) that my model requires does cause any problem with the camera app activity? I'm not sure if the image will be distorted to fit into 299299 to make it work.

@kwotsin your logcat cuts off in the middle of the relevant error, can you paste the rest please?

@andrewharp After several days of testing, I have managed to locate the source of the error, which is actually coming from quantize_nodes. As the ClassifierActivity.java file requires an input and output nodes of:
private static final String INPUT_NAME = "Placeholder_only";
private static final String OUTPUT_NAME = "MobileNet/Predictions/Softmax";

I realized the quantized file should have such an input output node as well. After confirming that the frozen model has these input and output nodes, I checked whether the quantized version has these nodes as well, and I realized it doesn't. Instead, it has these nodes after quantization:
(<tf.Tensor 'import/MobileNet/conv_ds_6/dw_batch_norm/batchnorm/sub/_7__cf__7_quantized_const:0' shape=(256,) dtype=quint8>,)
(<tf.Tensor 'import/MobileNet/conv_1/batch_norm/batchnorm/sub/_25__cf__25_quantized_min:0' shape=() dtype=float32>,)
(<tf.Tensor 'import/MobileNet/conv_ds_6/depthwise_conv/depthwise_weights_quantized_max:0' shape=() dtype=float32>,)
(<tf.Tensor 'import/MobileNet/conv_1/weights_quantized_max:0' shape=() dtype=float32>,)
(<tf.Tensor 'import/MobileNet/conv_ds_10/pw_batch_norm/batchnorm/sub/_29__cf__29_quantized_const:0' shape=(512,) dtype=quint8>,)
(<tf.Tensor 'import/MobileNet/conv_ds_12/pointwise_conv/biases_quantized_min:0' shape=() dtype=float32>,)
(<tf.Tensor 'import/MobileNet/conv_1/convolution_eightbit/Placeholder_only/reduction_dims:0' shape=(1,) dtype=int32>,)
(<tf.Tensor 'import/MobileNet/conv_ds_3/depthwise_conv/depthwise_weights_quantized_min:0' shape=() dtype=float32>,)
(<tf.Tensor 'import/MobileNet/conv_ds_9/pointwise_conv/weights_quantized_max:0' shape=() dtype=float32>,)
(<tf.Tensor 'import/MobileNet/conv_ds_5/pointwise_conv/biases_quantized_min:0' shape=() dtype=float32>,)
(<tf.Tensor 'import/MobileNet/conv_ds_9/dw_batch_norm/batchnorm/Rsqrt/_34__cf__34_quantized_max:0' shape=() dtype=float32>,)
(<tf.Tensor 'import/MobileNet/conv_ds_2/depthwise_conv/depthwise_weights_quantized_const:0' shape=(3, 3, 32, 1) dtype=quint8>,)
(<tf.Tensor 'import/MobileNet/conv_ds_14/pointwise_conv/weights_quantized_max:0' shape=() dtype=float32>,)
(<tf.Tensor 'import/MobileNet/conv_ds_2/depthwise_conv/biases_quantized_max:0' shape=() dtype=float32>,)
(<tf.Tensor 'import/MobileNet/conv_ds_14/pointwise_conv/weights_quantized_const:0' shape=(1, 1, 1024, 1024) dtype=quint8>,)
(<tf.Tensor 'import/MobileNet/conv_ds_14/depthwise_conv/biases_quantized_const:0' shape=(1024,) dtype=quint8>,)
(<tf.Tensor 'import/MobileNet/fc_16/weights_quantized_min:0' shape=() dtype=float32>,)

After removing each graph transform one by one, I hoped to see if there is a specific transformation which after I removed, will give me the correct input node called 'Placeholder_only', which I included in the graph when I first froze the model. After checking all transformations, I found quantize_nodes is the transformation that removed the input_node. After removing quantize_nodes from the transformation, I have these nodes instead:
(<tf.Tensor 'import/Placeholder_only:0' shape=<unknown> dtype=float32>,)
(<tf.Tensor 'import/MobileNet/conv_ds_8/dw_batch_norm/batchnorm/sub/_53__cf__53_quantized_max:0' shape=() dtype=float32>,)
(<tf.Tensor 'import/MobileNet/conv_ds_8/dw_batch_norm/batchnorm/sub/_53__cf__53_quantized_min:0' shape=() dtype=float32>,)
(<tf.Tensor 'import/MobileNet/conv_ds_8/dw_batch_norm/batchnorm/sub/_53__cf__53_quantized_const:0' shape=(512,) dtype=quint8>,)
(<tf.Tensor 'import/MobileNet/conv_ds_8/dw_batch_norm/batchnorm/sub/_53__cf__53:0' shape=(512,) dtype=float32>,)
(<tf.Tensor 'import/MobileNet/conv_ds_8/dw_batch_norm/batchnorm/Rsqrt/_52__cf__52_quantized_max:0' shape=() dtype=float32>,)
(<tf.Tensor 'import/MobileNet/conv_ds_8/dw_batch_norm/batchnorm/Rsqrt/_52__cf__52_quantized_min:0' shape=() dtype=float32>,)
(<tf.Tensor 'import/MobileNet/conv_ds_8/dw_batch_norm/batchnorm/Rsqrt/_52__cf__52_quantized_const:0' shape=(512,) dtype=quint8>,)

And now my quantized model works on the app. Is there a reason why quantize_nodes would cause the model to not work? Also, even after I've read the documentation here: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/tools/graph_transforms#quantize_nodes
I don't quite understand what quantize_nodes does and the benefits that may be gained. What are the purpose and benefits of this tool?
Also, strangely for the logcat, the error log is all I could see when I run the app and after it crashes immediately. There is nothing produced after the 'fatal signal' output. My current logcat status is at 'Verbose' - is this the correct status to use?

Seems to be issue #8698
This is probably a bug in merge_duplicate_nodes (called by quantize_nodes)
See #8897 for reference

I think the underlying issue here is that you need to specify the right input and output names when you call transform_graph too. In the code above, you're using the default of Mul and final_result when you invoke the tool, but they should be input and MobileNet/Predictions/Softmax, like your application code.

@petewarden Mul and final_result are not the input and output names I used, but rather, the example usage given in the documentation. I included them just for the sake of illustration, but the actual command I used had the input and output node names I used in the java code as well. In fact, I think it would not be possible to run the graph transform tool successfully if the node names are invalid or do not exist within the graph. I think, at least for my use case, quantize_nodes somehow gave me the error when running.

@kwotsin @petewarden  @andrewharp Were you able to solve this issue? I've hit the same error..

@anandcu3 Did you try removing quantize_nodes from your transformation?

@kwotsin But isn't quantize_nodes  necessary for improving performance ?

YMMV but the last time I tried it, this gave me worse results instead. I think it shouldn't take too long for you
to verify whether it gives better accuracy or not - perhaps you can try it and see if your results does improve.

@kwotsin  I tried it and you are right the  Fatal signal 11 (SIGSEGV), code 1 error is resolved, but overall there is no change in either accuracy or performance. The output is the same

@anandcu3 To verify with you, 'no change in either accuracy or performance' - does it mean the accuracy is retained as your original model, or is the poor performance retained? I think quantize_nodes is still largely experimental now, but so far I've seen quite good results with just quantize_weights alone.

@kwotsin Accuracy is retained and the performance is also the same as original model..

I fine-tuned a mobilenet model for binary classification. The problem is when I use Quantised model on Android, it produces exactly same prediction values for all images after retraining. The same mobilenet model when non-quantised produces reasonable values.
This problem is only on Android. On iOS it works nicely for both non-quantised and quantised model. What could be reason for this and solution for Android. I tried

pre-compiled .so of tensorflow inference
compiling .a file with selective registration.

I solved this problem by updating libtensorflow_inference.so from nightly build. Leave this message just in case someone would have same problem as we had.
Try this.
https://ci.tensorflow.org/view/Nightly/job/nightly-android/lastSuccessfulBuild/artifact/out/native/

https://github.com/tensorflow/tensorflow/issues/11182
I solved it by compiling the tensorflow .a lib file for android with flag -DTENSORFLOW_DISABLE_META

It has been 14 days with no activity and this issue has an assignee.Please update the label and/or status accordingly.

If I use the frozen graph the app works but if I use optimize_for_inference_lib.optimize_for_inference, I get loading error in android. Is there a solution for this?

The classification is work with frozengraph, is there something I am missing?


Github Review and make public

Now that this repo is public is this issue done and can be closed?

Sure


examples/restaurantbot  Circuit breaker tripped

Rasa version:
Rasa 1.6.1
Example is from git commit: 64f3927cf91f538c6e4cfd5f40debf6bb5c5d295
Python version:
Python 3.7.3
Operating system (windows, osx, ...):
osx
Issue:
simple demo restaurant bot trips the circuit breaker with a simple input:
"hello"
"hello"
There is a relevant closed issue here: https://github.com/RasaHQ/rasa/issues/2157
Please fix the example code and educate us on how to make correct stories that don't hit this issue.
Error (including full traceback):
(rasa) restaurantbot $ rasa shell
2020-01-13 10:47:32 INFO     root  - Starting Rasa server on http://localhost:5005
2020-01-13 10:47:37 INFO     absl  - Entry Point [tensor2tensor.envs.tic_tac_toe_env:TicTacToeEnv] registered with id [T2TEnv-TicTacToeEnv-v0]
2020-01-13 10:47:54 INFO     rasa.nlu.components  - Added 'SpacyNLP' to component cache. Key 'SpacyNLP-en'.
Bot loaded. Type a message and press enter (use '/stop' to exit):
Your input ->  hello
how can I help you?
Your input ->  hello
2020-01-13 10:48:01 WARNING  rasa.core.processor  - Circuit breaker tripped. Stopped predicting more actions for sender 'default'.
I'm on it
I'm on it
what kind of cuisine would you like?
I'm on it
I'm on it
where?
I'm on it
I'm on it
what kind of cuisine would you like?
I'm on it

Thanks for raising this issue, @TyDunn will get back to you about it soon✨
Please also check out the docs and the forum in case your issue was raised there too 🤗


⚠ Cioos National CKAN Sitemap has degraded performance

In 5dfc4ef, Cioos National CKAN Sitemap (https://catalogue.cioos.ca/sitemap/sitemap-1.xml) experienced degraded performance:

HTTP code: 200
Response time: 12563 ms

Resolved: Cioos National CKAN Sitemap performance has improved in 04613cc after 17 minutes.


BCL API review: Get rid of BasePath on IConfigurationBuilder

Currently there is a ConfigurationBuilder constructor that takes a basePath string and a BasePath string property in the ConfigurationBuilder class and the IConfigurationBuilder interface. This is the typical usage:
var configurationBuilder = new ConfigurationBuilder(basePath)
  .AddIniFile("config.ini")
  .AddJsonFile("config.json");

This introduces a weird conceptual layering issue between core configuration concepts and file system concepts. We have discussed two alternative ways to decouple the two:

Have each individual configuration provider take a separate argument for the base path and compute the path combine based on it in their corresponding add method, e.g.

var configurationBuilder = new ConfigurationBuilder()
  .AddIniFile(basePath, "config.ini")
  .AddJsonFile(basePath, "config.json");

Add a property bag to IConfigurationBuilder which allows flowing additional build-time information to providers. Then define extension method that assign the value of a well-known "BasePath" property for file-based configuration providers.

var configurationBuilder = new ConfigurationBuilder()
  .SetBasePath(basePath)
  .AddIniFile("config.ini")
  .AddJsonFile("config.json");

The main disadvantage of the first option is that it becomes redundant to pass the same argument to each of the potentially many file-based configuration providers in the chain.
The second option seems at first glance a bit far-fetched but we believe we could apply the same pattern to flow other objects that configuration providers might depend on. We can also define the SetBasePath() and GetBasePath() extension methods in a package that extends configuration to support file-based providers.

Note that the second option could be used to implement #183.

@divega I like the second one, too, but I'm curious about what happens if you do not specify the base path.
In the current implementation, basePath seems to be mandatory.
Would AddJsonFile("config.json") throw, if you don't call SetBasePath? I think that would be bad because it's "easy to get wrong".
On the other hand, if basePath would be optional, this syntax would totally make sense!
Are there cases where I could have different basePaths for different configuration sources?
Can I call SetBasePath multiple times?
var configurationBuilder = new ConfigurationBuilder()
 .SetBasePath(basePath)
 .AddIniFile("config.ini")
 .SetBasePath(basePath2)
 .AddJsonFile("config.json");

@MichaCo I am not sure exactly what should happen, but your reasoning seems sound: if there is no reasonable default, we should throw. However the user can provide an already full path to the AddX() method so I think we should just throw if we cannot resolve the file name to an actual file.
I also agree if we implement this feature it will be possible to change the values it could become confusing. Maybe what we need to do is to capture a delegate that sets the value instead of the value directly so that we can easily replay them in order?
var configurationBuilder = new ConfigurationBuilder()
 .SetBasePath(() => basePath)
 .AddIniFile("config.ini")
 .SetBasePath(() => basePath2)
 .AddJsonFile("config.json");

Or maybe we just say that the last one wins.

@divega yup, throwing only if the file cannot be resolved would be great!
Regarding the delegate, I think this would make the api a lot harder to understand. Maybe use a delegate internally, but not as an argument?
And yeah, you are right, you could also just provide a fully qualified file name instead of base path and file name.
Last one wins would again cause confusion I guess, unless we throw an exception if SetBasePath gets called more than once. Could be a solution, too!

At the design discussion last week we agreed on creating the property bag in the builder and the SetBasePath() extension method (which in the end should live in a common place for file-based configuration sources, not in Microsoft.Framework.Configuration). Inside each AddXyz() method we will extract the relevant information from the property bag and pass it to the constructors of the corresponding configuration sources, so there shouldn't be any ambiguity for the case below:
var configurationBuilder = new ConfigurationBuilder()
 .SetBasePath(basePath)
 .AddIniFile("config.ini")
 .SetBasePath(basePath2)
 .AddJsonFile("config.json");

The config.ini file will be looked up in basePath and the config.json file will be looked up under basePath2.

merged bf62a69de10a50f01ddc4b9049abe8572c4995d8


[TECHDEBT] AddrBook management optimization and refactoring - Issue #246

Description
This PR addresses the TECHDEBT items defined in the related Issue #246
Issue
Fixes issue/246
Type of change
Please mark the options that are relevant.

[x] New feature, functionality or library
[x] Code health or cleanup

List of changes

Added peersManager and target in order to abstract away and eliminate redundant computations
refactored debug logging in getTarget to print first and second target on the same line
Refactored AddPeerToAddrBook to use an event-driven approach in order to leverage sorted data structures
Added RemovePeerToAddrBook making use of the same pattern
Improved performance of AddPeerToAddrBook and RemovePeerToAddrBook by making the implementations O(n)

Performance improvements
Via #243 we can compute the "baseline" for the previous implementation with BenchmarkAddrBookUpdates.
Using
go test -benchmem -run=^$ -bench ^BenchmarkAddrBookUpdates$ github.com/pokt-network/pocket/p2p/raintree

By adjusting numAddressessToBeAdded in BenchmarkAddrBookUpdates from 1000 address updates (default) to 10000 on the old and new implementation we have:

numAddressessToBeAdded
Old
New (this PR)
Screenshot

1000
35.14 s
1.42 s

10000
410.02 s
7.71 s

Obviously with higher numbers numAddressessToBeAdded the difference is going to be much bigger.
Testing

[x] make test_all
[x] LocalNet w/ all of the steps outlined in the README

Checklist

[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have tested my changes using the available tooling
[x] If applicable, I have made corresponding changes to related local or global README
[x] If applicable, I have added new diagrams using mermaid.js
[x] If applicable, I have added tests that prove my fix is effective or that my feature works

@Olshansk:
This PR initially was tending only for
// OPTIMIZE(olshansky): We are computing this twice for each message, but it's not that expensive. (source)
// IMPROVE(olshansky): Consolidate so the debug print contains all (i.e. both) targets in one log line (source)
but then I realized that things were more connected than I thought so I increased scope and created #246 which now this PR aims at closing.
Notable changes from the previous review: Removed the router but added the peer_manager (suggestions obviously welcome as always). Everything else should be covered by the PR description

Whenever you're chaining PRs that are dependant on each other,make sure to update the base so its easier for the reviewer to identify the relevant changes:

@deblasis This still has some comments from my previous review, so I wanted to check if it's ready for a re-review?

@deblasis This still has some comments from my previous review, so I wanted to check if it's ready for a re-review?

Not yet buddy, few more comments to address and it will be. ETA tomorrow morning (your timezone).

Whenever you're chaining PRs that are dependant on each other,make sure to update the base so its easier for the reviewer to identify the relevant changes:

I am not sure I understand what you mean but we can take it offline.

@deblasis This still has some comments from my previous review, so I wanted to check if it's ready for a re-review?

Not yet buddy, few more comments to address and it will be. ETA tomorrow morning (your timezone).

 It would be a nice to have IMHO.


"checkpoint_required" error

Ok, after #670 , and after the temporary fix I still can't get login() to work on accounts I know are working.
I did a var_dump of $response->body, at line 1550, and this is what I got:
object(stdClass)#15 (4) {   ["message"]=>   string(19) "checkpoint_required"   ["checkpoint_url"]=>   string(33) "/challenge/9840808137/1DcjKgL2rM/"   ["lock"]=>   bool(false)   ["status"]=>   string(4) "fail" }
I assume this is the challenge @ibnux mentioned in his message. I'm assuming as I run this from the command line, not from the browser.
How can I circumvent this without, as suggested there, creating new accounts?

@eldark Did you succeed to trigger the checkpoint_url directly on your web browser?
If yes, how? (changing ip or other way?)

@eldark Did you succeed to trigger the checkpoint_url directly on your web browser?
If yes, how? (changing ip or other way?)

Nope. My app was designed to run from the command line, so I would have to refactor a lot to try that. But now that you mention, it might be possible to go to that checkpoint_url, capture the button and simulate a click.
If you point to https://www.instagram.com/ + checkpoint_url, you get redirected to the "We Detected An Unusual Login Attempt" page, prompting you to click "This Wasn't Me" or "This Was Me". So there might be something useful there.

@eldark Weird. I tried to connect my dummy account with various IPs around the world and I never get any checkpoint_required. 😞
Do you have a dummy account to share?

@eldark Weird. I tried to connect my dummy account with various IPs around the world and I never get any checkpoint_required. 😞
Do you have a dummy account to share?

Not really. I'm using a client's Instagram account.
Do a var_dump of $response->body, at line 1550 (Instagram.php). There you'll see that array.

@pgrimaud you can try my account, is not my primary account, just for scrapping
user: xnuqi
pass: xnu2020qi

to see an error i change this line
https://github.com/postaddictme/instagram-php-scraper/blob/f98597de5bc75185ecd87f2ae8f1f8c320070637/src/InstagramScraper/Instagram.php#L1561
to show error using json_encode($response)
throw new InstagramAuthException('Something went wrong. Please report issue.'.json_encode($response), $response->code);

Thank you so much @ibnux for your credentials. 🙏 🙏 🙏 🙏 🙏
I may have more question for you, can you contact me on Twitter or Discord?
I got the 400 error now.
I will write a "Proof of concept" script to trigger verification.
How it works :

Try to login
Got 400 error "checkpoint_required"
Trigger email verification
Connect to your email inbox using SMTP
Wait for Instagram verification email
Parse verification code from email
Make a request to instagram with this code to complete verification
Verification is done, then save session

What do you think about it?

Connection through SMTP. Very clever. I'm up to trying your approach.

Thank you so much @ibnux for your credentials. 🙏 🙏 🙏 🙏 🙏
I may have more question for you, can you contact me on Twitter or Discord?
I got the 400 error now.
I will write a "Proof of concept" script to trigger verification.
How it works :
1. Try to login

2. Got 400 error "checkpoint_required"

3. Trigger email verification

4. Connect to your email inbox using SMTP

5. Wait for Instagram verification email

6. Parse verification code from email

7. Make a request to instagram with this code to complete verification

8. Verification is done, then save session

What do you think about it?

You can find me on twitter with same username @ibnux or telegram
or Discord ibnux#2048
yes i got verification number on my email, maybe smtp will work

Connection through SMTP. Very clever. I'm up to trying your approach.

 IMAP, not SMTP. 🤦🏻♂

I can now bypass automatically this checkpoint challenge thank to @ibnux & @eldark 🔥
I tried this :
1. Try to login
2. Got 400 error "checkpoint_required"
3. Trigger email verification
4. Connect to your email inbox using SMTP
5. Wait for Instagram verification email
6. Parse verification code from email
7. Make a request to instagram with this code to complete verification
8. Verification is done, then save session

I wrote a PoC on my own Instagram scraper : https://github.com/pgrimaud/instagram-user-feed/blob/feature/checkpoint-challenge/src/Instagram/Auth/Login.php
I will write better code in the next days then submit a PR here.

Excellent, @pgrimaud . Will check your code and see if I can get it sort of work on my end.
I will also wait for your PR before closing this, so @raiym can work in peace.


DevicePool: for managing active devices across stack

Tracking TODOs and possible issues.
https://github.com/tenstorrent/tt-metal/pull/7839
Notes from meeting with Joseph and Almeet:

Possibly remove num_hw_cqs and l1_bank_remap from device constructor
Merge DevicePool and DeviceMesh
Add APIs to get certain topologies (Mesh, Linear, Ring etc.)

moving this to P1 as we keep bumping into bugs that this would resolve

Picking up this work now, I'll start with the following items:

rename DeviceMesh to DevicePool, and move it to tt_metal namespace.
make it static and use it in the multi device command queue initialization
add an API to return a ring for TTNN multi device APIs

fyi @abhullar-tt @cfjchu

make it static and use it in the multi device command queue initialization

what do you mean by this bullet?

We need a global DevicePool for command queue initialization. So I'm thinking of making DevicePool something like this:
const DevicePool& DevicePool::instance(
    std::vector<chip_id_t> device_ids, const uint8_t num_hw_cqs, const std::vector<uint32_t>& l1_bank_remap) {
    static DevicePool device_pool(device_ids, num_hw_cqs, l1_bank_remap);
    device_pool.l1_bank_remap = l1_bank_remap;
    return device_pool;
}

DevicePool::DevicePool(
    std::vector<chip_id_t> device_ids, const uint8_t num_hw_cqs, const std::vector<uint32_t>& l1_bank_remap) {
    ZoneScoped;
    for (const auto& device_id : device_ids) {
        const auto& mmio_device_id = tt::Cluster::instance().get_associated_mmio_device(device_id);
        for (const auto& mmio_controlled_device_id :
             tt::Cluster::instance().get_devices_controlled_by_mmio_device(mmio_device_id)) {
            if (this->active_devices.find(mmio_controlled_device_id) == this->active_devices.end()) {
                auto dev = new Device(mmio_controlled_device_id, num_hw_cqs, l1_bank_remap);
                this->active_devices.insert({mmio_controlled_device_id, dev});

                detail::InitDeviceProfiler(dev);
            }
        }
    }
    tt::Cluster::instance().set_internal_routing_info_for_ethernet_cores(true);
}

Device* DevicePool::get_active_device(chip_id_t device_id) const {
    TT_ASSERT(
        this->active_devices.find(device_id) != this->active_devices.end(),
        "DevicePool does not contain active device {}",
        device_id);
    auto dev = this->active_devices.at(device_id);
    if (not dev->is_initialized()) {
        dev->initialize(this->l1_bank_remap);
    }
    return dev;
}

So that TTNN and runtime backend can call tt::DevicePool::instance({0}).get_active_device(0) to get device 0. I'll tweak the APIs to make it more usable, but this is the gist of it

Giving an update, still working on changes. Majority of feature for merging global device managers is done. Trying to delete some redundant code and smoothing out CI: https://github.com/tenstorrent/tt-metal/actions/runs/9195121154/job/25290455521

Open issues:

#8936


[WIP] Change mobile alert to dialog

For alerts should we always have an "OK" button bottom right?

Here's what it looks like with the OK button:

Nice! I don't think the OK button is necessary as it has the same outcome / interaction as the "X". What do you think @akegan ?

¯_(ツ)_/¯

¯\_(ツ)_/¯

i'll keep it off, adds a bit more code anywho

Update:
Added a confirm window for the large csv download instead

went back to having "OK" because it makes more sense for the case that the file being uploaded is too big


Reformat the ETL job info message in log.

fix a typo there
add CONTEXT_NAME in logback.xml, so that we can configure it in the future

LGTM


Add definition, hover and completion support for class variables

This issue is exactly the same as #2644, but for class variables. Please refer to the description of #2644 for instructions on how to implement it.
The work is exactly the same, but instead of handling globals $something, we handle class variables @@something.
Note: class instance variables are already handled. No work is needed on that front. That is:
class Foo
  @class_instance_variable = "Hello!"
end

Hi Team, I am looking at this issue.


Buggy drag / reorder behaviour on filtered lists

I think this is related to #8, but I'm getting buggy dragging behaviour when list data is updated dynamically.
It seems to affect short lists the most, though I've seen it on larger lists as well.
Here is a list, filtered from the start which works as expected:

And here is a list which I filter down at run-time, and exhibits the buggy behaviour:

I've tried everything from removing all styles and going down to minimal HTML content, and it seems to be that it's because the list is filtered.
If there's a workaround, such as some kind of update() method in the meantime, I'm happy to use that, otherwise, looking forward to a fix!

@davestewart Can you confirm that the issue is fix so that I can close this? Thanks!!

I'll be able to check at some point later today!

Any update on this @kutlugsahin? seems like the same problem im having. Wishing to have some update method as mentioned

any updates on that? seems like I'm having the same problem here


Add a mechanism to define services

I.e. a job that has to run along with all consuming jobs, before being terminated. This can be used to integrate the invocation of databases, or setting up shared memory allocation (e.g. for loading a STAR index before mapping). A service is automatically submitted together with its consuming jobs. Moreover, there can be multiple instances of the same service for multiple groups

cc @cmeesters

Has been added in another PR-.


hi I want connect autoware.universe and g29

I notice autoware.universe
subscribe /vehicle/status/steering_status
stamp:
sec:<PHONE_NUMBER>
nanosec: 645533388
steering_tire_angle: -0.3019617795944214
g29-force-feedback
publish /ff_target ros_g29_force_feedback/msg/ForceFeedback "header: {stamp: {sec:0, nanosec:0}, frame_id: ' '}, position: 0.3, torque:0.5"
so I want let position=steering_tire_angle、but i don't know how to achieve it, Can you teach me how to do it
my Email is<EMAIL_ADDRESS>Thanks you

After check all project i think author must have do this job！
so I ask for help

You need to write the ros2 node which acts like this

subscribing vehicle status message
position=steering_tire_angle
publish the ForceFeedback message.

You may have done by manually, right?
So just write down the process to the code.

how can i conversion with you？ i meet some trouble

cmake_minimum_required(VERSION 3.8)
project(voyager_pub_sub)
Default to C99
if(NOT CMAKE_C_STANDARD)
set(CMAKE_C_STANDARD 99)
endif()
Default to C++14
if(NOT CMAKE_CXX_STANDARD)
set(CMAKE_CXX_STANDARD 14)
endif()
if(CMAKE_COMPILER_IS_GNUCXX OR CMAKE_CXX_COMPILER_ID MATCHES "Clang")
add_compile_options(-Wall -Wextra -Wpedantic)
endif()
find_package(ament_cmake REQUIRED)
find_package(rclcpp REQUIRED)
find_package(std_msgs REQUIRED)
find_package(rosidl_default_generators REQUIRED)
find_package(autoware_auto_vehicle_msgs REQUIRED)
find_package(ros_g29_force_feedback REQUIRED)
set(dependencies
rclcpp
std_msgs
autoware_auto_vehicle_msgs
ros_g29_force_feedback
)
Generate project targer with msg
rosidl_generate_interfaces(${PROJECT_NAME}
"msg/ForceFeedback.msg"
DEPENDENCIES rosidl_default_generators std_msgs)
Add executable
add_executable(voyager_node src/voyager.cpp)
ament_target_dependencies(voyager_node ${dependencies})
ament_export_dependencies(rosidl_default_runtime)
if(BUILD_TESTING)
find_package(ament_lint_auto REQUIRED)
the following line skips the linter which checks for copyrights
uncomment the line when a copyright and license is not present in all source files
#set(ament_cmake_copyright_FOUND TRUE)
the following line skips cpplint (only works in a git repo)
uncomment the line when this package is not in a git repo
#set(ament_cmake_cpplint_FOUND TRUE)
ament_lint_auto_find_test_dependencies()
endif()
ament_package()

voyager_pub_sub
0.0.0
TODO: Package description
voyager
TODO: License declaration
<buildtool_depend>ament_cmake</buildtool_depend>
<build_depend>rosidl_default_generators</build_depend>
<exec_depend>rosidl_default_runtime</exec_depend>
<member_of_group>rosidl_interface_packages</member_of_group>
std_msgs
rclcpp
autoware_auto_vehicle_msgs
ros_g29_force_feedback
<test_depend>ament_lint_auto</test_depend>
<test_depend>ament_lint_common</test_depend>

ament_cmake

``

colcon build pass!  can you help me check I have some problem?

Good job!
Please test it. If there are some problems, I can help you.

moring

``rriding autoware_auto_vehicle_msgs
Starting >>> autoware_auto_vehicle_msgs
Finished <<< autoware_auto_vehicle_msgs [3.52s]
Starting >>> voyager_pub_sub
--- stderr: voyager_pub_sub
In file included from /usr/include/x86_64-linux-gnu/c++/9/bits/c++allocator.h:33,
from /usr/include/c++/9/bits/allocator.h:46,
from /usr/include/c++/9/memory:63,
from /opt/ros/galactic/include/rclcpp/rclcpp.hpp:154,
from /home/voyager/Desktop/voyager_1/ros2_ws/voyager_pub_sub/src/voyager.cpp:1:
/usr/include/c++/9/ext/new_allocator.h: In instantiation of ‘void __gnu_cxx::new_allocator<_Tp>::construct(_Up*, _Args&& ...) [with _Up = GreateBusinessNode; _Args = {const char (&)[8]}; _Tp = GreateBusinessNode]’:
/usr/include/c++/9/bits/alloc_traits.h:483:4:   required from ‘static void std::allocator_traits<std::allocator<_Tp1> >::construct(std::allocator_traits<std::allocator<_Tp1> >::allocator_type&, _Up*, _Args&& ...) [with _Up = GreateBusinessNode; _Args = {const char (&)[8]}; _Tp = GreateBusinessNode; std::allocator_traits<std::allocator<_Tp1> >::allocator_type = std::allocator]’
/usr/include/c++/9/bits/shared_ptr_base.h:548:39:   required from ‘std::_Sp_counted_ptr_inplace<_Tp, _Alloc, _Lp>::_Sp_counted_ptr_inplace(_Alloc, _Args&& ...) [with _Args = {const char (&)[8]}; _Tp = GreateBusinessNode; _Alloc = std::allocator; __gnu_cxx::_Lock_policy _Lp = __gnu_cxx::_S_atomic]’
/usr/include/c++/9/bits/shared_ptr_base.h:679:16:   required from ‘std::__shared_count<_Lp>::__shared_count(_Tp*&, std::_Sp_alloc_shared_tag<_Alloc>, _Args&& ...) [with _Tp = GreateBusinessNode; _Alloc = std::allocator; _Args = {const char (&)[8]}; __gnu_cxx::_Lock_policy _Lp = __gnu_cxx::_S_atomic]’
/usr/include/c++/9/bits/shared_ptr_base.h:1344:71:   required from ‘std::__shared_ptr<_Tp, _Lp>::__shared_ptr(std::_Sp_alloc_shared_tag<_Tp>, _Args&& ...) [with _Alloc = std::allocator; _Args = {const char (&)[8]}; _Tp = GreateBusinessNode; __gnu_cxx::_Lock_policy _Lp = __gnu_cxx::_S_atomic]’
/usr/include/c++/9/bits/shared_ptr.h:359:59:   required from ‘std::shared_ptr<_Tp>::shared_ptr(std::_Sp_alloc_shared_tag<_Tp>, _Args&& ...) [with _Alloc = std::allocator; _Args = {const char (&)[8]}; _Tp = GreateBusinessNode]’
/usr/include/c++/9/bits/shared_ptr.h:701:14:   required from ‘std::shared_ptr<_Tp> std::allocate_shared(const _Alloc&, _Args&& ...) [with _Tp = GreateBusinessNode; _Alloc = std::allocator; _Args = {const char (&)[8]}]’
/usr/include/c++/9/bits/shared_ptr.h:717:39:   required from ‘std::shared_ptr<_Tp> std::make_shared(_Args&& ...) [with _Tp = GreateBusinessNode; _Args = {const char (&)[8]}]’
/home/voyager/Desktop/voyager_1/ros2_ws/voyager_pub_sub/src/voyager.cpp:37:61:   required from here
/usr/include/c++/9/ext/new_allocator.h:146:4: error: no matching function for call to ‘GreateBusinessNode::GreateBusinessNode(const char [8])’
146 |  { ::new((void *)__p) _Up(std::forward<_Args>(__args)...); }
|    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/voyager/Desktop/voyager_1/ros2_ws/voyager_pub_sub/src/voyager.cpp:10:3: note: candidate: ‘GreateBusinessNode::GreateBusinessNode()’
10 |   GreateBusinessNode()
|   ^~~~~~~~~~~~~~~~~~
/home/voyager/Desktop/voyager_1/ros2_ws/voyager_pub_sub/src/voyager.cpp:10:3: note:   candidate expects 0 arguments, 1 provided
make[2]: *** [CMakeFiles/voyager_node.dir/build.make:63：CMakeFiles/voyager_node.dir/src/voyager.cpp.o] 错误 1
make[1]: *** [CMakeFiles/Makefile2:123：CMakeFiles/voyager_node.dir/all] 错误 2
make: *** [Makefile:141：all] 错误 2
Failed   <<< voyager_pub_sub [2.36s, exited with code 2]

Can you help  me deal with this issus

No constructor in your code.

auto node = std::make_shared();
↓
auto node = std::make_shared();

thank you I have finish this job


bind_api: close the session at the end to avoid "Too many open files" (issue#810)

This is an attempt to fix the Too many open files error (errno=EMFILE=24),
reported in issue#810

https://github.com/tweepy/tweepy/issues/810

This patch does fix the issue! Please merge!

Superseded by #1068.


RoadNetwork RoadMap

Hello everyone, we have a road map for RoadSystem too!

[x] Basic Road Network with graph.
[ ] Road Tool
[x] Road Renderer for Segments
[ ] Road Renderer for Intersections
[ ] Road Tool with states for each type of segment.
[x] Multi-edged graph support. (No self-loops)
[x] Rewritten Road Network with new API
[x] Road Intersection Node
[ ] Road Managers for RoadNetworkInfo and RoadSegmentTypes
[ ] Road Markings
[ ] Road Saving and Loading
[ ] Road Network Info editor.

We'll be implementing all these into the game.
We also plan to make a integration for the editor so it's you can use it as a Addon.

ok nice.

👍


Go back to typehint stubbing

AFAIK PHP 7 support has been added to phpspec

:+1:

Waiting for #40 and #41 to avoid double effort.


Rename enum-naming.md to constant-naming.md

Hi @ahass - Thanks for your contribution. This PR has a merge conflict. Can you please resolve and submit your PR again? Thank you!


Add migration verification

At the moment, there is no verification performed to ensure that each tag is present in the v2 registry after migration; it is assumed based off of the exit codes from each push.  While this is relatively safe, it would be better to verify.

Here is a possible approach: https://github.com/docker/distribution/issues/585#issuecomment-108117282.


[FEATURE REQUEST] Honor chocolatey repos self-serve and admin only flags

Before submitting the issue

[x] I have checked that the requested feature is not a duplicate or, if it is, it needs to be reopened
[x] I have checked that I am running the latest version of WingetUI
[x] I have checked the FAQ section
[x] I have read the contributing guidelines and I agree with the Code of Conduct

Is your feature request related to a problem? Please describe.
WingetUI is an excellent self-serve utility for organizations with internal repositories. It would be great if WingetUI could honor repos flagged for admin only and/or flagged for self-serve.
Describe the solution you'd like
Like if user does not have admin privileges and admin only is set to true then don't show/check that repo. Or if self-serve is set to false and user is not admin don't permit installation (or prompt for elevation), etc. Repos default to false for both so probalby would need an override as well. which i suspect would need to be on by default.

Is there a way to detect if the package requires administrator privileges?

Not particularly. Unless you have Chocolatey for Business it more or less insists on you having admin privileges but it does have some the capability to test for elevation and even request elevation before firing off a package installation. I don't know how widely used this is, but I know it's possible.


[MonorepoBuilder] Only remove packages of monorepo from require/require-dev

Currently, during the merge, all packages that are using the same vendor name as the root composer.json file will be removed from require/require-dev sections.
I found myself in the situation where I want to build a monorepo for some packages of vendor acme (for example) but at the same time some of these packages depend on a package not contained within the monorepo, let's say acme/foo. Right now, the package acme/foo gets removed from the require/require-dev sections because of the same vendor name acme.
It would be nice if the removal of requirements would only consider packages provided by the monorepo and leave all others, no matter if the vendor name matches or not.
What do you think? Would this be a nice feature for you?

Hey, I understahnd the feature and it seems legit to me :+1:
Do you have idea how to implement it?

I'm not sure. One approach could be to pass the list of merged composer files to the DependenciesMerger, so it can pass them to the Decorators, the RootRemoveComposerJsonDecorator could then use this list instead of the vendor name.
Another approach could be to not only set require/require-dev for found packages but additionally add the package to the replace section with a self.version. The list of the replace section could then be used to filter out packages of the require/require-dev sections. So, basically this would result in a composer.json similar to symfony's (https://github.com/symfony/symfony/blob/master/composer.json).
I think I would go with the second approach.
I'm not sure when I will have time to look into it, so if you have some spare time ... :wink:

Well, replace section was also my first idea, but it's not actually required for monorepo and would be hidden magic.
Your first idea was my second :) let's go with that
I'm full-time open source on Rector, but take your time.

Ok, I think I can create a PR in the next couple of days.
However, I think filling up the replace section would be a nice addition. The decorator interface has to be changed anyway, so it should be possible to provide this functionality by an additional decorator. But this would be another PR. :smile:

I think filling up the replace section would be a nice addition.

Well, that would be bending user's own data to match our bad package architecture.

The decorator interface has to be changed anyway, so it should be possible to provide this functionality by an additional decorator. But this would be another PR.

I think this can be done without any interface change thanks to DI, but we'll see :)

...aaaaaaaaand done in #1037 :)

Great, thank you very much. I just came back from holiday last week and didn't have time to look at this yet.
Just read your first notification a couple of minutes ago and boom there it is already. :+1:

I see :) thanks for letting me know.
Have you been to some foreing country or just time off?

I had some lazy days in spain, actually a lot of them :)

Nice :), I've never been there


feat: Added construct_image_chooser_queryset, construct_document_chooser_queryset and construct_page_chooser_queryset hooks

Please see #3371

Thanks @Gagaro! Merged in 46c9cdda1511ecbf2d082123dab2494479b7b842.


Chore: Migrate Discord Bot from SerenityOS to Ladybird

This pull request addresses some of the tasks that were needed to migrate the bot, such as changing the branding, links, and getting rid of Node and Yarn in favour of Bun.

Looks OK to me


First try for the code-names document

Related to #734 - this is an example of how we could format / unify releases of Cucumber across multiple platforms.

To be followed here: https://github.com/cucumber/docs.cucumber.io/pull/402


How to writeandflush() send data directly to the remote end instead of through the channeloutboundbuffer

When I send small byte data through netty, data merging occurs. The receiving end needs to wait a long time to receive a large data block that has been merged. I don't want to have merged data, and I need to send it in real time。
How can I do it? Thank you for telling me

writeAndFlush() should put the data on the wire, that's what the Flush part means. Check with tcpdump exactly what is going on, perhaps?

also depending on the pattern you may want to experiment with TCP_NO_DELAY

I've tried TCP? Nodely, but I can't

I try to read the netty source code and find that there is a do while and LinkedList in addflush() in the channeloutboundbuffer, and then the real write of DoWrite. I don't know if it's the reason here. I guess it's because of outboundbuffer that I can't write data in time. Instead, aggregate small data blocks and send them


autocomplete min-length is not honored

i have a min-length of 2 setup but as soon as i click on the text-box it calls this controller method to do lookup for data and it fails becasue query is null, if i add a check to see if the length is >2 in my vm.getCNCodeData it gives an error saying Cannot read property 'data' of undefined  as its expecting promise instead of nothing.
my problem is that even when the min-length is not satisfied it triggers the autocomplete to do remote load for the list
controller code

            vm.getCNCodeData = function ($query) {
                    return diagnosisService
                        .getCNCode($query);
                }

my view code

                            <tags-input ng-model="vm.prescription.primaryCondition"
                                        key-property="icdCodeCNId"
                                        display-property="icdCodeCNName"
                                        placeholder="Add a primary condition"
                                        replace-spaces-with-dashes="false"
                                        add-from-autocomplete-only="true"
                                        template="condition-autocomplete-template"
                                        add-on-paste="true">
                                <auto-complete source="vm.getCNCodeData($query)"
                                               min-length="4"
                                               debounce-delay="500"
                                               highlight-matched-text="true"
                                               display-property="icdCodeCNName"
                                               load-on-focus="true"
                                               load-on-empty="true"
                                               max-results-to-show="32"
                                               template="condition-autocomplete-template">
                                </auto-complete>
                            </tags-input>

my problem was load-on-focus="true"
load-on-empty="true"


Help to make nipplejs works on React Native

Hi,
I would like to get nipplejs works on React Native, or alternative package on react native also helpful.
Do you have any plan to support/porting nipplejs to React Native? Or can you give me a quick guide?
Thank you.

Hi,
unfortunately, I don't have much time to implement new stuff.
I wanted to first make everything ES6.
Then we'll see for the React(-Native) part later.

@nvl1109 why not https://facebook.github.io/react-native/docs/webview ?

Hi.
I just released a react wrapper - it was fairly simple and straight-forward.
While I have nothing to do with react-native, I guess there shouldn't be much trouble doing necessary adjustments.
See https://www.npmjs.com/package/react-nipple
See https://github.com/loopmode/react-nipple/blob/master/packages/react-nipple/src/ReactNipple.js

There is a dataOnly mode with the library, I don't remember completely, but in theory it shouldn't go through the dom.
Maybe you could leverage it somehow.
Let me know how it goes.

This would be awesome!  There's NOTHING that works for me™ in the controller input area.  All I could find were two libs about a year old that depended on es2015 babelrc presets and after 15 minutes of trying the wife and kids required attention.  I just want a joystick for the Arduino robot my  sons and I are building.  I was hoping to cobble together a Bluetooth remote for android in a weekend.  Proving to be difficult.
The results would end up on github as 'lazer robot st1000 remote' which is what my sons want to call it.
I will check out the dataOnly mode and see if I can get er to chooch.

I have written a React Native component that uses this package.
@wrinkledcheese let me know if there are any other features you want I will look to add them.
It can be found here: https://www.npmjs.com/package/react-native-game-pad

@jim-at-jibba proposed a solution.
I might get a shot at this later. Offer a bit more deeply integrated solution for React/React-Native.


Can't connect to freshly installed LS server

I am trying to set up LS on a private server (say my.server.co.uk) and make it publicly accessible using that URL. I installed LS using pip (python 3.7).
I start the server specifying http://my.server.co.uk as the host and 8080 as the port. However, when I go to http://my.server.co.uk:8080 in my browser, I get:
This page isn’t working
my.server.co.uk didn’t send any data.
ERR_EMPTY_RESPONSE

This is the output I get when I start the server:
$ label-studio start --host http://my.server.co.uk --port 8080 --log-level DEBUG
=> Database and media directory: /home/mf501/.local/share/label-studio
=> Static URL is set to: /static/
=> Hostname correctly is set to: http://my.server.co.uk
=> Database and media directory: /home/mf501/.local/share/label-studio
=> Static URL is set to: /static/
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /pypi/label-studio/json HTTP/1.1" 200 55841
[2023-02-01 21:56:14,850] [core.feature_flags.base::<module>::30] [INFO] Read flags from file /home/mf501/.local/lib/python3.7/site-packages/label_studio/feature_flags.json
[2023-02-01 21:56:14,852] [ldclient.util::get::76] [INFO] Initializing LaunchDarkly Client 7.5.0
[2023-02-01 21:56:14,852] [ldclient.util::_make_update_processor::138] [INFO] Using user-specified update processor: <function Files.new_data_source.<locals>.<lambda> at 0x7f2e292d99e0>
[2023-02-01 21:56:15,182] [ldclient.util::init::117] [DEBUG] Initialized 'segments' store with 0 items
[2023-02-01 21:56:15,182] [ldclient.util::init::117] [DEBUG] Initialized 'features' store with 104 items
[2023-02-01 21:56:15,183] [ldclient.util::__init__::115] [INFO] Waiting up to 5 seconds for LaunchDarkly client to initialize...
[2023-02-01 21:56:15,183] [ldclient.util::__init__::119] [INFO] Started LaunchDarkly Client: OK
[2023-02-01 21:56:15,188] [core.redis::<module>::22] [DEBUG] => Redis is not connected.
[2023-02-01 21:56:16,023] [label_studio.server::check_port_in_use::230] [INFO] Checking if host & port is available :: localhost:8080
[2023-02-01 21:56:16,026] [label_studio.core.utils.common::start_browser::313] [INFO] Start browser at URL: http://my.server.co.uk
Performing system checks...

[2023-02-01 21:56:16,167] [django::register_actions_from_dir::97] [INFO] No module named 'data_manager.actions.__pycache_'
[2023-02-01 21:56:16,167] [django::register_actions_from_dir::97] [INFO] No module named 'data_manager.actions.__pycache_'
System check identified no issues (1 silenced).
February 01, 2023 - 21:56:16
Django version 3.2.16, using settings 'label_studio.core.settings.label_studio'
Starting development server at http://<IP_ADDRESS>:8080/
Quit the server with CONTROL-C.

And this is how I check that the port is open:
$ sudo netstat -lpn |grep :8080
tcp        0      0 <IP_ADDRESS>:8080            <IP_ADDRESS>:*               LISTEN      35433/python3.7

What could be happening? I don't know where to look. Your help is appreciated.

You have to open port 8080 in your target host either by OS firewall or Cloud firewall.

I opened the ports with iptables and, as I showed earlier, port 8080 is open and listening. Am I missing anything else? Is there a LS log I can check? I'm using Ubuntu 18.04 and Apache 2.4.

Run curl localhost:8080/version on your private server
If it response - it means there is an issue with your network set up for your host in Cloud Provider

Thanks. Both curl localhost:8080/version and curl my.server.co.uk:8080/version give some output so I guess there is a network issue as you point out. I'm racking my brains now trying to figure out what it might be...

You can use nmap to scan for open ports:
nmap –p 8080 my.server.co.uk

This is what I'm getting... Doesn't it look OK? (BTW my.server.co.uk is not the real domain name)
$ nmap –p 8080 my.server.co.uk

Starting Nmap 7.60 ( https://nmap.org ) at 2023-02-02 16:16 GMT
Failed to resolve "–p".
Strange read error from <IP_ADDRESS> (22 - 'Invalid argument')
Strange read error from <IP_ADDRESS> (22 - 'Invalid argument')
Nmap scan report for my.server.co.uk (<IP_ADDRESS>)
Host is up (0.00022s latency).
Not shown: 996 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
80/tcp   open  http
443/tcp  open  https
8080/tcp open  http-proxy

Nmap done: 2 IP addresses (1 host up) scanned in 0.10 seconds

@mfelice may I know if this issue has been resolved? how you deal with it in the end? I am facing the same error, that with --host paramenter, it cannot be accessed by others... Thanks!

I could never resolve it, so gave up and used another platform.


Support hot-reload for Wechaty events listeners

The API might add the following new features:
1. addListener
wechaty.on('message', './message-listener')

If the second parameter is a string instead of a function, then we can use func = require('./message-listener') as the listener function, monitor the file, and reload it if the file had been updated.
2. WechatyInstanceOptions
bot = Wechaty.instance({
    listenerDirectory: './listenerDirectory/'    // this is the directory which contains all listeners.
})

Wechaty will scan ./listenerDirectory/ for the files that the same name as the event, then require it and add it as the listener.

See Also

https://github.com/fastest963/require-reload
http://blog.chatie.io/developer/2017/03/20/added-hot-reload-for-bots.html
https://blogs.msdn.microsoft.com/typescript/2017/06/12/announcing-typescript-2-4-rc/
https://www.youtube.com/watch?v=cGqfM6huJCs
Getting the stack trace in a structured form
Get v8 stack traces as an array of CallSite objects.
A node module for enabling a module to determine its caller.
[Design Spec] ESNext import()
Hot module replacement for node.js
Hot reload tool for node.js

CC @Gcaufy

Looks good for me.
There is another way to do it, may be more convenient to the developers
bot = Wechaty.instance({
    listeners: './listeners'    // this is the directory which contains all listeners.
});
bot.init();

May be you can support both ways.

@Gcaufy I just published a new NPM module as a Hot Module Reload tool named hot-import at here: https://github.com/zixia/hot-import
It supports TypeScript, please give it a try and let me know what you think. :)

Landed at v0.8.240.

Get to know more about the usage of hot-import listener in Wechaty: #923 #935


Installing on Android via rnpm does not seem to work.

Receiving undefined is not an object whenever calling Orientation.initialOrientation.

It works with react-native versions below 0.29 only, a symptom of a breaking change in the application template. Pull request #85 fixes the issue, so either use the code in that or (if you can afford to) wait until it is merged and released.

Please follow updated README documentation on installing this package. If problems persist, create a new issue. Works with latest RN 0.44


Add recursive literal checking

Also beefs up Style/DuplicatedKey in the process. I'm sure this can be used to simplify Style/LiteralInCondition too, but it wasn't immediately obvious to me how.

You can almost gut LiteralInCondition now, as you noted. Take this:
        def check_for_literal(node)
          cond, = *node

          # if the cond node is literal we obviously have a problem
          if cond.literal?
            add_offense(cond, :expression)
          else
            # alternatively we have to consider a logical node with a
            # literal argument
            check_node(cond)
          end
        end

...and change to something like:
        def check_for_literal(node)
          cond, = *node
          add_offense(cond, :expression) if cond.recursive_literal?
        end

Then start deleting all the methods which are not used any more.

@alexdowad I tried that, but I don't think I understand why a bunch of the current specs for LiteralInCondition are the way they are -- they seem rather non-literal to me?

Which in particular?

The thing I was struggling with was that recursive_literal? checks if all nodes are literal. What we need for LiteralInCondition is to see if any 'toplevel' (i.e. nodes that a user would see, not the children the lexer produces) are literal. Should be possible to do in a short method anyways.

see if any 'toplevel' (i.e. nodes that a user would see, not the children the lexer produces) are literal.

Could you provide an example illustrating the difference?

@alexdowad
["#{hello}"]

is where all "top-level" nodes are literal
["#{'hello'}"]

is where the entire thing is recursively literal

Let's wrap this PR. What remains to be done?

I think it's fine as-is?

Just rename complex to composite and rebase.

And you should add a changelog entry.

@bbatsov done! sorry for the long delay

:+1:

There's still no changelog entry, but I guess we can add it later.


Added Native Soundcloud Support

Hello,
For my own purposes, I have integrated the soundcloud downloader package with discord-music-player to allow for native soundcloud support. Of course, doing this may have introduced a few bugs, but it works with my implementation. I am putting this pull request here to potentially contribute if you wish to add my code, or to edit it so that it fits. My implementation is a bit janky, but I feel that with a little bit of work, it could add a lot.
Thank you so much, if you have any questions please contact me. And if this pull req isn't accepted, I completely understand.
Sincerely,
John Nomikos.

1091 files changed omg, you clearly did something wrong, like deleting .gitignore or something, I will recommend getting rid of those commits and any unnecessary change

1091 files changed omg, you clearly did something wrong, like deleting .gitignore or something, I will recommend getting rid of those commits and any unnecessary change

Yeah I deleted the gitignore since it wouldn't let me download from git. I'll fix

As @jnomikos said, please upload the files only. You uploaded the build + node modules too. Please fix so we can take a look.

As @slienlaffa said, please upload the files only. You uploaded the build + node modules too. Please fix so we can take a look.

Sounds good. I don't have my computer right now so I'll be moving, but I'll do that when I am able to (sometime after the 12th).

I can't merge it @jnomikos as you pushed all files, even the build ones. Please review your pr and make a new one without pushing everything (look gitignore)


Fix Z-ordering issues + Refactoring

There were a few bugs related with Z-ordering.
This PR:

Move TextZ and TrackZ from Layout to Canvas.
Erase some numeric values and changed them to Canvas' constants.
Refactor Canvas a little.
Fix wrong Z-values (see images).

Before:
Iterators with SelectionOverlay:

Iterators with Track Titles

Now:

Quick question: Are the screenshots showing the bugs or are they showing the rendering after the fixes?

Quick question: Are the screenshots showing the bugs or are they showing the rendering after the fixes?

Sorry for the confusion. Showing the bug. I've just edited and now you can see before & after.

Quick question: Are the screenshots showing the bugs or are they showing the rendering after the fixes?

Sorry for the confusion. Showing the bug. I've just edited and now you can see before & after.

Ok, thanks. Looks great!

Just a small nit, o/w looks reasonable to me.
My understanding is that this does not fix the issue that text is rendered at the end and thus on top of everything else, right? So, transparent overlays drawn "over" text (like the text on the TimeGraph time slices) do not change the color tint of the text (as it should). Or am I wrong about this?

Clarification: I think the green overlay is actually fine, as it's drawn at the very end. But the iterator overlay color does not affect the text on the timeslices (but it should).

My understanding is that this does not fix the issue that text is rendered at the end and thus on top of everything else, right? So, transparent overlays drawn "over" text (like the text on the TimeGraph time slices) do not change the color tint of the text (as it should). Or am I wrong about this?

Clarification: I think the green overlay is actually fine, as it's drawn at the very end. But the iterator overlay color does not affect the text on the timeslices (but it should).

Yeah, it doesn't solve the text & color-lint problem. It was one of my goals when I started looking at, but no solution.
BTW, I don't see any difference between the green overlay and the iterators (except than before they were in differents z-level). Also didn't follow the point about text drawing on top of everything. In one of the images, if the iterator was on top of the text, I can't see the text. Could be some difference between Windows and Linux?

BTW, I don't see any difference between the green overlay and the iterators (except than before they were in differents z-level). Also didn't follow the point about text drawing on top of everything. In one of the images, if the iterator was on top of the text, I can't see the text. Could be some difference between Windows and Linux?

My understanding is the following (please someone correct if any of this is wrong): For the purpose of discussing the issues here, two things matter: drawing order and z values and they influence each other to some degree.
First, the issue that you were not seeing text under the iterator overlay was caused by two things: The text was drawn after the actual transparent overlay and it was drawn with a smaller z value than the overlay. Because of the smaller z value, drawing the text was simply discarded at the depth testing stage. This issue is fixed here, because now the text is drawn with a higher z value.
Second, I was wrong about there being a difference between the green overlay and the iterator overlay. They are both not color tinting the text at all. This is not very obvious as the overlays are pretty light, but can be seen when you make all transparent overlays a lot more opaque. This is because the drawing order is roughly (I'm omitting stuff here that is not relevant):

Draw iterator overlay
Draw green overlay
Draw text (on top of everything else)

For the transparent overlays to affect the color of the text, the rendering order needs to be something like this:

Draw capture window
Draw capture window text
Draw iterator overlays
Draw iterator overlay text (on top of everything so far)
Draw green overlay
Draw green overlay text (on top of everything so far)

So, this does add rendering passes that are not there currently, at least for the text rendering. I think though that we need to change this, as the white text on top of everything else makes it very hard to discern which text belongs to which overlay (depending on rendering location).
Not for this PR, but something we should look into.

Yes, I agree 100% with you now. Really nice summary :)
Only thing that I want to add, if we change the kZValueOverlay constant to be greater than the kZValueText one, and modify the order of drawing elements (for example, storing every draw-element in a priority queue and after that drawing from kZValue lower to higher ones), I think that there won't be drawing-bugs anymore.

Yes, I agree 100% with you now. Really nice summary :)
Only thing that I want to add, if we change the kZValueOverlay constant to be greater than the kZValueText one, and modify the order of drawing elements (for example, storing every draw-element in a priority queue and after that drawing from kZValue lower to higher ones), I think that there won't be drawing-bugs anymore.

I like the idea of using something like a priority queue and then just drawing by z value order (in that case we wouldn't even need depth testing on the GPU anymore, so z values are just used for establishing the order). Given the small number of z values we have effectively, a simple data structure would be sufficient, basically a list of pairs of batchers and text renderers that are indexed by z value.
I still think we need a larger number of different z values for overlays and text layers. There is text that is on the iterator overlays, but that should still be below the green overlay (and affected by its transparent color). So, I think three different text z values and two different overlay z values. The priority queue would then enforce exactly the drawing order I stated in my comment above (right?).

I like the idea of using something like a priority queue and then just drawing by z value order (in that case we wouldn't even need depth testing on the GPU anymore, so z values are just used for establishing the order). Given the small number of z values we have effectively, a simple data structure would be sufficient, basically a list of pairs of batchers and text renderers that are indexed by z value.
I still think we need a larger number of different z values for overlays and text layers. There is text that is on the iterator overlays, but that should still be below the green overlay (and affected by its transparent color). So, I think three different text z values and two different overlay z values. The priority queue would then enforce exactly the drawing order I stated in my comment above (right?).

Yes, exactly as you said. The priority queue could be a hash_map, where keys are "kZValue" and values, the list of elements to draw in this level, and after find the order of these hash_keys and draw the lists in that order.
Also, we can make something like start drawing the text elements in some level and after that the box elements, and avoid to duplicate the amount of kZValues.

FYI, I added a new commit which solves b/162485723 (modified a kZValue).


Document the DefaultToolOptions

There are lots of tool options which are not documented on the website http://ceylon-lang.org/documentation/1.2/reference/tool/config/ including defaults.timeout.
See https://github.com/ceylon/ceylon/blob/master/common/src/com/redhat/ceylon/common/config/DefaultToolOptions.java
Can't we generate this documentation somehow?

Hey why you assign this to me? Most of those options I didn't write, there's even one you added! ;)
Btw, these need to be documented in the tools themselves as well.

AFAICS defaults.flatclasspath and defaults.autoexportmavendependencies are not used, so I won't document those.


Add serde support

Add support for serde to make our serialize function easier to use.

bors merge

Now that we fixed the bug on main, we can try again!
bors merge


HAL and CMSIS files updated to revision r4.2b_v4

Notes:

Pull requests will not be accepted until the submitter has agreed to the contributer agreement.
This is just a template, so feel free to use/remove the unnecessary things

Description
A few sentences describing the overall goals of the pull request's commits.
Status
READY/IN DEVELOPMENT/HOLD
Migrations
If this PR changes any APIs or behaviors, give a short description of what API users should do when this PR is merged.
YES | NO
Related PRs
List related PRs against other branches:

branch
PR

other_pr_production
link

other_pr_master
link

Todos

[ ] Tests
[ ] Documentation

Deploy notes
Notes regarding the deployment of this PR. These should note any required changes in the build environment, tools, compilers and so on.
Steps to test or reproduce
Outline the steps to test or reproduce the PR here.

Closing pull request because changes of this pull request has already been merged through other pull requests.


FastAPI Version Seems to Cause Issue

Bug description
I am getting an issue from FastAPI, which is installed as a dependency.

As shown in the photo, I have fastapi 0.88.0. When I upgrade to 0.95 (the latest version), this error goes away. But, pip complains because Lightning is pinned to this version of FastAPI.
Would it be possible to update the dependency?
FWIW there is a similar issue on the FastAPI Github: https://github.com/tiangolo/fastapi/discussions/5977
They fixed it by updating Starlette, but I think I'm already on the newest version.
What version are you seeing the problem on?
2.0+
How to reproduce the bug
No response
Error messages and logs
# Error messages and logs here please

Environment

Current environment
#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version (e.g., 1.5.0):
#- Lightning App Version (e.g., 0.5.2):
#- PyTorch Version (e.g., 2.0):
#- Python version (e.g., 3.9):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
#- Running environment of LightningApp (e.g. local, cloud):

More info
No response

Would it be possible to update the dependency?

I think so yes! Let's try it. The file here needs to be updated by bumping the version there to the latest: https://github.com/Lightning-AI/lightning/blob/6cbc9dfb9174ec38909a3d0e46f584ce109777cf/requirements/app/base.txt#L15
The fastapi has a "strict" label since something was broken there. We can backtrack through git-blame why this was set and check whether the issue was resolved in newer versions. If so, we can do this: fastapi!=0.89.0


[CMSIS-NN] Align CMSIS-NN in TVM to TFLu SHA

Aligning CMSIS-NN to TFLu's CMSIS-NN SHA. Since the
docker update will take some time, these code changes
contain support for old and new SHA both. After the
docker image update, we can remove the support with
older CMSIS-NN. This commit updates CMSIS-NN and
micronpu demos.

cc: @grant-arm @Mousius @manupa-arm for code review.

LGTM

@Mousius PTAL again. Thanks!

Thanks @ashutosh-arm, this is merged 😸 can you raise a Docker image issue for ci_cpu and ci_qemu?


Unresolved function or method then()

Hi all,
I can't get node-fetch to work. I started a new project and just imported node-fetch. When I try to run the sample function I get the following error:

Any help?

Your IDE environment likely have some issues.
Try this if you need a quick test: https://npm.runkit.com/node-fetch


add platform factsheets: new model + template + macro + content

Added factsheets for Reddit, YouTube, and Twitter
Created parent page for all factsheets, and link to this page is nested under Data governance nav bar

@Quiet27 I've added a "Last updated" field to all the factsheets and set them to be today. What are your thoughts?

@boyd-nguyen that's perfect!


Add option and logic to disable additional out-of-zone data.

NSD follows CNAME, DNAME, and possibly other data outside of the queried zone if the target zone is also configured in NSD. This may not always be desirable. This PR adds a configuration option and logic to disable this behavior. The option defaults to the current behavior.
nsd.conf
server:
        ip-address: <IP_ADDRESS>
        database: ""

remote-control:
        control-enable: yes
        control-interface: <IP_ADDRESS>

zone:
        name: foo.test
        zonefile: /etc/nsd/foo.test.zone
zone:
        name: bar.test
        zonefile: /etc/nsd/bar.test.zone

foo.test.zone
@ IN SOA ns0.foo.test. username.foo.test. 0 0 0 0 0
cname-zone IN CNAME zone
cname-local IN CNAME  local.bar.test.
cname-remote IN CNAME remote.biz.test.
zone A <IP_ADDRESS>

bar.test.zone
@ IN SOA ns0.bar.test. username.bar.test. 0 0 0 0 0
local A <IP_ADDRESS>

Output:
[root@nsd-testing nsd]# dig +short @<IP_ADDRESS> cname-zone.foo.test
zone.foo.test.
<IP_ADDRESS>
[root@nsd-testing nsd]# dig +short @<IP_ADDRESS> cname-local.foo.test
local.bar.test.
<IP_ADDRESS>
[root@nsd-testing nsd]# dig +short @<IP_ADDRESS> cname-remote.foo.test
remote.biz.test.

With the PR changes and additional-from-auth set to no:
root@nsd-testing nsd]# dig +short @<IP_ADDRESS> cname-zone.foo.test
zone.foo.test.
<IP_ADDRESS>
[root@nsd-testing nsd]# dig +short @<IP_ADDRESS> cname-local.foo.test
local.bar.test.
[root@nsd-testing nsd]# dig +short @<IP_ADDRESS> cname-remote.foo.test
remote.biz.test.

I'm open to suggestions on the option name (I took this one from bind although it appears to be obsoleted now). I also contemplated allowing zones/patterns to override the global setting but wasn't sure if calling zone_options_find in the query path was a good idea.

Hi @greg-bock! Thanks for the PR. Most of the code looks good, just two minor remarks.

I'd prefer you'd use dname_compare() instead of testing for pointer equality. While that'll work just fine for now, in the future there could be modifications that would invalidate the check.
The name additional-from-auth doesn't hold much value in NSD as it's an authoritative-only nameserver (bind had additional-from-cache too). Apart from that, as you stated, the option has been removed from bind in 9.12, which may cause some confusion. How about confine-to-zone: <yes|no>? That way the option can also just be initialized to 0.

I'll happily accept the changes once the issues have been resolved.

I'll make the changes shortly. Thanks for the feedback!

@k0ekk0ek let me know if you want me to squash this into a single commit and force push the branch to clean up history.

@greg-bock, sorry for the delay. I wanted to give the other committers a chance to react to the proposed changes. I'm assuming they're fine with the changes, so I'll go ahead and merge this. Squashing won't be necessary, GitHub allows me to squash the changes and combine them in a single commit. Thanks again!

Thanks!


Difference between course and screencast

What is the difference between a screencast and a course?
course:
https://github.com/EbookFoundation/free-programming-books/blob/99672525f2493ac22eb70e4587925b6e85c45a44/courses/free-courses-en.md#L256
screencast:
https://github.com/EbookFoundation/free-programming-books/blob/99672525f2493ac22eb70e4587925b6e85c45a44/casts/free-podcasts-screencasts-en.md#L50

some of the "screencasts" might better be termed "vlogs", but changing would break things.

maybe we should add a better definition on the CONTRIBUTING file, or combine the two

ImVector wrote:

maybe we should add a better definition on the CONTRIBUTING file, or combine the two

Back in the day when you got a print copy of the «undergraduate
announcement» for the academic year, «course» referred to a series of
multiple lectures during spring/fall term.  And I'm still used to
refer to use this like «among the courses, did you read chemistry
201» as hypernym (umbrella term) and contrasting to «did you attend
the lecture on Tuesday» to refer to a specific event and date,
regardless if attendance was in person, or remotely, across the
country via a computer.  Based on this tradition, I would prefer new
contributions to the project to follow this pattern.

as we look to the words, the course is a recorded course while the screencast is a live class.

What is the difference between a screencast and a course?
course:
https://github.com/EbookFoundation/free-programming-books/blob/99672525f2493ac22eb70e4587925b6e85c45a44/courses/free-courses-en.md#L256
screencast:
https://github.com/EbookFoundation/free-programming-books/blob/99672525f2493ac22eb70e4587925b6e85c45a44/casts/free-podcasts-screencasts-en.md#L50

@LuigiImVector
Course: A comprehensive, structured learning experience covering a topic in-depth with lessons, exercises, and a clear syllabus.
Example: A full C++ beginner course with multiple tutorials progressing from basics to advanced topics.
Screencast: A video showing how to perform specific tasks or workflows, usually shorter and more focused on demonstrations.
Example: A video showing how to code specific C++ programs step-by-step.


push entry will not be removed if 401 is returned

fail: MinigolfFriday.Host.Services.WebPushService[0]
      Failed to send push notification to https://wns2-am3p.notify.windows.com/w/?token=<...> with payload MinigolfFriday.Domain.Models.Push.PushNotificationPayload due to unexpected error
      WebPush.WebPushException: Received unexpected response code: 401
         at WebPush.WebPushClient.HandleResponse(HttpResponseMessage response, PushSubscription subscription)
         at WebPush.WebPushClient.SendNotificationAsync(PushSubscription subscription, String payload, Dictionary`2 options, CancellationToken cancellationToken)
         at WebPush.WebPushClient.SendNotificationAsync(PushSubscription subscription, String payload, VapidDetails vapidDetails, CancellationToken cancellationToken)
         at MinigolfFriday.Host.Services.WebPushService.<>c__DisplayClass7_0.<<SendAsync>b__0>d.MoveNext() in C:\projects\minigolf-friday\src\server\host\Services\WebPushService.cs:line 61

How does this even happen? The docs from Microsoft says this:

https://learn.microsoft.com/en-us/windows/apps/design/shell/tiles-and-notifications/push-request-response-headers#response-codes

I copied the database from the production server and removed every entry which was not linked to my account. There were around 10 entries for me. I send the test message via the api and looked through the log. I noticed also, that the tokens which are not in use anymore are not being deleted

Yeah then the VAPID keys don't match. Is this a real scenario?

I had no idea of a vapid key. Ok then it is no issue

The VAPID keys are to make sure only our server can send notifications. If someone got the Subscription endpoint, they would also need the VAPID private key to send notifications.
The keys are provided using the environment variables prefixed with VAPID_. I guess you did not notice, because I adjusted the docker-compose file on the prod server 😉


refactor(types): change account id to newtype and deprecate ValidAccountId

Definitely more involved than I would have liked and it's unclear what is the best way to initialize an AccountId but here is what changes:

Changes AccountId from a String alias to a newtype which performs the validation, JSON/borsh serialization, and other necessary interactions
Removes ValidAccountId

I was planning on aliasing AccountId for this, but it caused a lot of confusing errors linking to it instead of AccountId, and should probably be removed sooner than later

This is quite a large breaking change, because using String is quite nested within everything, and interactions from ValidAccountId specifically will cause breakages with existing contracts.
closes #446

Hey-hey! Node Interfaces team is working on a similar thing in nearcore (@miraclx has it in a branch state for now: https://github.com/near/nearcore/compare/master...miraclx:strict-account-id-validity) and even decided to create a separate crate out of it (near-account-id), so we can merge our efforts here.

Hey-hey! Node Interfaces team is working on a similar thing in nearcore (@miraclx has it in a branch state for now: near/nearcore@master...miraclx:strict-account-id-validity) and even decided to create a separate crate out of it (near-account-id), so we can merge our efforts here.

We can't use that implementation in its current state because it pulls in a few dependencies and uses regex for validation which is too bulky for the SDK (especially since an OKR for the last and upcoming quarter is minimizing contract size)

Because this is such a breaking change, it'd be great to get input from others.
Also, the primary remaining design decision is if we change the underlying type to Box<[u8]> which on one hand could allow for smaller code sizes but also does not hold up the compiler guarantee that it will always be utf8 bytes. It would also require calling from_utf8 on the bytes for every reference, and I'm not sure how that gets optimized.

We can definitely onboard this implementation as I also don't feel very good about pulling the whole regex machinery to validate the account ID format.
Have you measured the difference of Box<[u8]> vs Box<str>?

We can definitely onboard this implementation as I also don't feel very good about pulling the whole regex machinery to validate the account ID format.
Have you measured the difference of Box<[u8]> vs Box<str>?

The size of memory is the same, not sure about the compiled code. I am also noticing when switching to the Boxed versions that BorshSchema is broken for them. I can look into fixing this if it's possible.

@frol wrt above https://github.com/near/borsh-rs/pull/36 which is just a benefit to borsh, but would allow these alternatives to be used here (I checked)

deriving BorshDeserialize for AccountId is wrong

Think I've addressed everything else, but curious why you think this is wrong? Any contract using the old AccountId will have the ID borsh serialized as a string, and it would be a state-breaking change when they update the SDK. What would you suggest is a better format? I don't see how string would be a bad borsh format either, or am I missing something about borsh storing more metadata for strings over byte slices?

why you think this is wrong?

It allows creating AccoundId which is not valid
    #[test]
    fn deser_checks_validity() {
        let r1 = "💩".parse::<AccountId>();
        assert!(r1.is_err()); // correct, 💩 is not a valid account name

        let bytes = "💩".try_to_vec().unwrap();
        let r2 = AccountId::try_from_slice(&bytes);
        let acc = r2.unwrap();
        assert_eq!(acc.as_str(), "💩") // BUG, 💩 is treated as a valid account
    }

As a rule of thumb, type invariants should be checked in every constructor, and deserialization provides an extra constructor.

As a rule of thumb, type's invariants should be checked in every constructor, and deserialization provides an extra constructor.
We shouldn't change serialization format, but we should add an extra validation to deserialization.

Ah, of course, oops


Importing plain Postgres backup

Hello!
I was not able to find similiar issue opened, so I am sorry if it is already reported.
When I backup tables and its data to format "plain" it works fine, but when I try to import them back (by Restore function), it says this:
pg_restore: unknown format of archive "p", supported types are "c", "d" or "t"
In short: it tries to specifiy non existang flag in pg_dump. As specified here: https://www.postgresql.org/docs/current/static/app-pgrestore.html

-F format
--format=format
Specify format of the archive. It is not necessary to specify the format, since pg_restore will determine the format automatically. If specified, it can be one of the following:
c
custom
The archive is in the custom format of pg_dump.
d
directory
The archive is a directory archive.
t
tar
The archive is a tar archive.

pg_restore should not be used to restore data in plain format, as it can't restore them. Documentation specifies:

pg_restore is a utility for restoring a PostgreSQL database from an archive created by pg_dump in one of the non-plain-text formats.

psql itself should be used to import (restore) plain text with usage something like this:
psql -U your_user_name your_db_name < your_dump_file

Hi @MesoX ,
Is this issue related to import database dumps through DBeaver ?

Hey @carlosspohr - yes. You can try that yourself - right click schema in Postgres Database and select Backup, use plain format and backup some tables and datas. When you open it, it is plain SQL format. Now, when you go for Restore, you can again choose "Plain" format for restore. But this one doesn't work.

@MesoX yep, it is a bug. pg_restore doesn't support format -p (in opposite to pg_dump). But it is smart enough to get format from file contents.


Pact Release Management / Binary Deployment

[ ] MacOS
[ ] Linux (statically linked)
[ ] Windows (Appveyor)
[ ] Made available on a GitHub release page (https://github.com/kadena-io/pact/releases)

See https://github.com/kadena-io/pact/pull/415 for statically linked Linux binaries.


Consider simplifying the annotations

Currently the annotations seem to support at least two ways of defining options, i.e. either:
@Annotation(() => Param1, param2)
And an options hash:
@Annotation({ entity: () => Param1, foo: param2 })
Granted, there is a trade-off between letting people use the style they want, but the downside is more complicated docs / tutorials / codebases as some docs use style1 / some use style2, and same thing in codebases, someone writes entity1 with style1, someone else writes entity2 with style2 / etc.
(I.e. anytime I navigate into the typedefs to look up an option I'm looking for, I get a little thrown off by, oh right, which part of this type union am I really using...)
Feel free to keep things as is if you're too worried about breaking backwards compatibility, but given in your in the RC stages (which granted might be a little too late), now might be a good time to just pick one and move everyone to just the single way (which I presume would be the hash approach b/c it's more extensible once you have multiple optional attributes).

Originally there was only the option object approach available. Then FR came for the other style, but mainly because of missing type support. Currently both ways are strictly typed, as long as you provide the entity option with a reference callback (@OneToOne({ entity: () => FooBar, mappedBy: bar => bar.baz }) will be strictly type checked for the mappedBy option validity).
I understand that it adds complexity, but I kinda like supporting both ways. The typing always works the same way, you can use either first or last parameter for options. Also, for obvious reasons, dropping the first options parameter approach is a no go (with ts-morph metadata provider, usually the entity parameter is optional). On the other hand, when using reflect metadata provider, the approach with multiple parameters works better I think.
I will keep this open for some time to see what others think, but personally I lean towards keeping it as is.

@B4nan cool, I'll go ahead and close this out. Just wanted to mention it, if you've already had FR for both, that makes sense...


catppuccin warning

Is there an existing issue outlining your problem?

[X] I have searched the existing issues and they do not solve my problem.

Describe your problem.
I keep getting this warning even though I am not using catppuccin_window_status_enable or catppuccin_window_status_icon_enable.
catppuccin warning: "@catppuccin_window_status_enable" and "@catppuccin_window_status_icon_enable" has been replaced by
       "@catppuccin_window_status" with the options "no", "icon" and "text"

Can you help me avoid this warning?
Paste your configuration.
Full configuration at https://github.com/srgsanky/tmux/blob/main/tmux.conf.
Here is the catppuccin specific ones
set -g @catppuccin_flavor 'mocha' # default is mocha
set -g @plugin 'catppuccin/tmux' # Theme

set -g @catppuccin_status_modules_right "directory date_time"
set -g @catppuccin_date_time_text "%Y-%m-%d %H:%M"

# every window
# See below for an explanation of the conditional
set -g @catppuccin_window_default_text "#{?#{==:#T,#{host}}, #W , #T: #W }"
set -g @catppuccin_window_default_fill "number"
# Got the dull color from https://www.color-hex.com/color-palette/80211
set -g @catppuccin_window_default_color "#c9e9e7"
set -g @catppuccin_window_default_background "black"

# only current window
set -g @catppuccin_window_current_text "#[fg=orange,bold,reverse]#{?#{==:#T,#{host}},     #W#F     ,     #T: #W#F     }"
set -g @catppuccin_window_current_fill "number" # number, all , none

Attach screenshots.
No response
What tmux version are you seeing the issue on?
tmux next-3.5
Any additional comments?
No response

Check the migration guide issue

My bad you are not using any of them will fix

Thanks!
For the time being, if I explicitly set the config I don't get the warning
set -g @catppuccin_window_status "icon"
set -g @catppuccin_window_status_icon "icon"

Sorry for the trouble