Add QGC Mission import capability

This patch

[x] Adds new capability to Mission plugin.
Now, missions can be imported from QGroundControl plan file. This enables users to create plans on QGroundControl and simply import them into DroneCore applications. We use json11 library for parsing QGC plan files.
[x] Adds unit test for import QGC mission feature which tests with a sample QGC plan.
[x] Re-factors Fly mission example
[x] Adds Fly QGC mission example to demonstrate popular QGC plan.
[x] Addresses #249
[X] Adds methods to know whether a mission item has position info set.
[x] Appveyor fixes for Windows build.
 Thanks.

[----------] 1 test from QGCMissionImport
[ RUN      ] QGCMissionImport.ValidateQGCMissonItems
[       OK ] QGCMissionImport.ValidateQGCMissonItems (1 ms)
[----------] 1 test from QGCMissionImport (1 ms total)

Travis CI says below error while performing Docker build for Ubuntu & Fedora.
This job ran on our Trusty environment,
which has been updated on Tuesday, December 12th, 2017.
Read all about this on our blog and take note that you can
add group: deprecated-2017Q4 in your .travis.yml file to use the previous image version.

Rest are successful. Can we ignore this ?

Travis fails on Ubuntu and Fedora because of a style error (see last lines of the log). Regarding the update warning, IMO it is not so important for us, so we can ignore it.

@JonasVautherin  Yeah, its strange; when I ran this locally, there are no such style errors reported.

Does the ./fix_style.sh work locally? Maybe you have to run it manually :-).

Yes. I ran make fix_style which in-turn invokes fix_style.sh. But none reported.

@julianoes Yes, it runs as part of make run_unit_tests

@julianoes Need your help in debugging. As discussed, in grouping mission items, I was successful in getting expected number of DroneCore mission items. But However, when I run Fly QGC Mission example with sample QGC plan, I get below error:
Found 6 mission items in the given QGC plan.
Uploading mission...
[09:44:56|Debug] Send mission item 0 (mission_impl.cpp:781)
[09:44:56|Debug] Send mission item 1 (mission_impl.cpp:781)
[09:44:56|Debug] Send mission item 2 (mission_impl.cpp:781)
[09:44:56|Debug] Send mission item 3 (mission_impl.cpp:781)
[09:44:56|Debug] Send mission item 4 (mission_impl.cpp:781)
[09:44:56|Debug] Send mission item 5 (mission_impl.cpp:781)
[09:44:57|Debug] Send mission item 6 (mission_impl.cpp:781)
[09:44:57|Debug] Send mission item 7 (mission_impl.cpp:781)
[09:44:57|Debug] Send mission item 8 (mission_impl.cpp:781)
Mission uploaded.
Arming...
[09:44:57|Info ] Mission accepted (mission_impl.cpp:146)
Armed.
Starting mission.
[09:44:57|Debug] MAVLink: info: ARMED by arm/disarm component command (device.cpp:247)
**[09:44:57|Debug] MAVLink: critical: Distance between waypoints too far: 5335120 meter (device.cpp:247)**
[09:44:57|Debug] MAVLink: info: [logger] file: rootfs/fs/microsd/log/2018-02-06/1 (device.cpp:247)
[09:44:57|Warn ] command temporarily rejected (176). (mavlink_commands.cpp:143)
Started mission.
Mission start failed: Error

Distance between waypoints too far: 5335120 meter (device.cpp:247)
That's far 😄 Is mission item 0 something without position?

@julianoes I think so.
I think its safe to make this assumption while importing QGC Mission.

First mission item is TAKEOFF command  (When you select to add waypoints, TAKEOFF gets selected by default)

AppVeyor Windows build succeeded. But, at the end it did exit with an error code. @julianoes it happened sometime ago too, isn't it ?

I don't understand what happens in AppVeyor :(. Need to try it on my Windows box at some point.

Weird... seems like one of those fails: cd build\core\Debug && unit_tests_runner.exe?

@JonasVautherin Like I mentioned here, is that correct ?

@JonasVautherin  Some of the CurlTest tests are failing;

Re-based with develop.

Why do "submodule" in .gitmodules still contain "libs" while we did replace with "third-party" ?
@jones

Actually I think it's fine for json11, but it's right that, interestingly, we still have some modules identified with the "libs/..." name (but with the correct "thirdparty/..." path). Not sure why and if it matters. Is it the id of the module?

Thanks @JonasVautherin for the json11 submodule fix.

@hamishwillee any more comments? Are you ok with this?

@julianoes

@hamishwillee any more comments? Are you ok with this?

Lot of change there, but it looks good.
@shakthi-prashanth-m It would be cool if you could add a QGroundControl screenshot of the planned mission, the executed mission (ie path vehicle took) and the console log. This will enable me to do documentation even if my VM is playing up.

@hamishwillee Sure, I'll provide screenshot of QGC and console logs of Fly QGC example.

Screenshot of QGroundControl when Fly QGC Mission example:

Fly QGC Mission example console logs:
$ ./fly_qgc_mission
Usage: ./fly_qgc_mission <path of QGC Mission plan>
Importing mission from Default mission plan: ../qgroundcontrol_sample.plan
Waiting to discover device...
[02:25:09|Info ] New device on: <IP_ADDRESS>:14557 (udp_connection.cpp:211)
[02:25:09|Debug] MAVLink: info: DISARMED by auto disarm on land (device.cpp:247)
[02:25:09|Debug] MAVLink: info: data link #1 lost (device.cpp:247)
[02:25:09|Debug] MAVLink: info: data link #0 regained (device.cpp:247)
[02:25:09|Debug] MAVLink: info: data link #1 regained (device.cpp:247)
[02:25:09|Debug] Discovered<PHONE_NUMBER> (dronecore_impl.cpp:219)
Discovered device with UUID:<PHONE_NUMBER>
Waiting for device to be ready
Waiting for device to be ready
Device ready
Found 8 mission items in the given QGC plan.
Uploading mission...
[02:25:11|Debug] Send mission item 0 (mission_impl.cpp:781)
[02:25:11|Debug] Send mission item 1 (mission_impl.cpp:781)
[02:25:11|Debug] Send mission item 2 (mission_impl.cpp:781)
[02:25:11|Debug] Send mission item 3 (mission_impl.cpp:781)
[02:25:11|Debug] Send mission item 4 (mission_impl.cpp:781)
[02:25:11|Debug] Send mission item 5 (mission_impl.cpp:781)
[02:25:11|Debug] Send mission item 6 (mission_impl.cpp:781)
[02:25:11|Debug] Send mission item 7 (mission_impl.cpp:781)
[02:25:11|Debug] Send mission item 8 (mission_impl.cpp:781)
[02:25:11|Debug] Send mission item 9 (mission_impl.cpp:781)
[02:25:11|Debug] Send mission item 10 (mission_impl.cpp:781)
[02:25:11|Debug] Send mission item 11 (mission_impl.cpp:781)
[02:25:12|Debug] Send mission item 12 (mission_impl.cpp:781)
[02:25:12|Debug] Send mission item 13 (mission_impl.cpp:781)
[02:25:12|Debug] Send mission item 14 (mission_impl.cpp:781)
Mission uploaded.
Arming...
[02:25:12|Info ] Mission accepted (mission_impl.cpp:146)
Armed.
Starting mission.
[02:25:12|Debug] MAVLink: info: ARMED by arm/disarm component command (device.cpp:247)
[02:25:12|Debug] MAVLink: info: [logger] file: rootfs/fs/microsd/log/2018-02-15/0 (device.cpp:247)
Started mission.
[02:25:12|Debug] MAVLink: info: Executing mission. (device.cpp:247)
[02:25:12|Debug] MAVLink: info: Takeoff to 15.0 meters above home. (device.cpp:247)
[02:25:12|Debug] MAVLink: info: Takeoff detected (device.cpp:247)
Mission status update: 0 / 8
Mission status update: 1 / 8
[02:25:30|Debug] MAVLink: info: data link #0 lost (device.cpp:247)
Mission status update: 1 / 8
Mission status update: 1 / 8
Mission status update: 1 / 8
Mission status update: 2 / 8
[02:25:32|Debug] MAVLink: info: data link #0 regained (device.cpp:247)
Mission status update: 2 / 8
Mission status update: 2 / 8
Mission status update: 2 / 8
Mission status update: 3 / 8
Mission status update: 3 / 8
Mission status update: 3 / 8
Mission status update: 3 / 8
Mission status update: 4 / 8
Mission status update: 4 / 8
Mission status update: 4 / 8
Mission status update: 4 / 8
Mission status update: 5 / 8
Mission status update: 5 / 8
Mission status update: 5 / 8
Mission status update: 5 / 8
Mission status update: 6 / 8
Mission status update: 6 / 8
Mission status update: 6 / 8
Mission status update: 6 / 8
Mission status update: 7 / 8
Mission status update: 7 / 8
Mission status update: 7 / 8
[02:26:19|Debug] MAVLink: info: Mission finished, loitering. (device.cpp:247)
Mission status update: 8 / 8
Commanding RTL...
Commanded RTL.

Build succeeded.
    0 Warning(s)
    0 Error(s)
Time Elapsed 00:00:04.38
cd ..\..\..
cd C:\dronecore
if "%configuration%"=="Debug" ( copy build\third_party\gtest\googlemock\gtest\Debug\gtestd.dll build\core\Debug\ && copy build\third_party\gtest\googlemock\gtest\Debug\gtest_maind.dll build\core\Debug\ && copy build\third_party\gtest\googlemock\Debug\gmockd.dll build\core\Debug\ && cd build\core\Debug && unit_tests_runner.exe ) else ( copy build\third_party\gtest\googlemock\gtest\Release\gtest.dll build\core\Release\ && copy build\third_party\gtest\googlemock\gtest\Release\gtest_main.dll build\core\Release\ && copy build\third_party\gtest\googlemock\Release\gmock.dll build\core\Release\ && cd build\core\Release && unit_tests_runner.exe )
        1 file(s) copied.
        1 file(s) copied.
        1 file(s) copied.
'unit_tests_runner.exe' is not recognized as an internal or external command,
operable program or batch file.
Command exited with code 9009

This indicates successful build; but it says:
'unit_tests_runner.exe' is not recognized as an internal or external command

This is mostly due to PATH settings in Windows. I didn't get this on My Windows machine. Should setting Windows PATH be done in cmake  ?

I guess that since the unit tests moved back to the root CMakeLists, the output path (here is not true anymore. Can you check on your machine that this is right and correct it in appveyor.yml?

@hamishwillee What's your current opinion on the PR? Are the changes good enough for you, or do you still require changes?

@julianoes now ready to be merged unless @hamishwillee requests for changes.

Thanks @shakthi-prashanth-m for the continuous fixing and improvements 😄.

Congratulations on getting this in @shakthi-prashanth-m ! I'll *try to get associated docs updates in next week: https://github.com/dronecore/docs/issues/99

Congratulations on getting this in @shakthi-prashanth-m ! I'll *try to get associated docs updates in next week: dronecore/docs#99

Thanks @hamishwillee .
I would like to thank @julianoes and @JonasVautherin for the review and valuable suggestions! :+1:


[BUG] Formatting is sometimes messed up

In some cases the Formatting of loaded Data is messed up and not correct

This should now be fixed as of this commit


Adding native 'typings'

The (relatively new) way of handling type declarations in the TypeScript compiler
is documented here.
This pull request simply adds support for 'typings' so that users can seamlessly import
the 'axios' module without the need to add an explicit declarations file to their
tsconfig.json file.

Closing in favor of #419.


TF2.0 distributed training fails after adding an embedding layer

Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): somewhat
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):Linux Ubuntu 18.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): v2.0.0-rc2-26-g64c3d38 2.0.0
Python version: 3.7.4
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:

Describe the current behavior
Try to train with MultiWorkerMirroredStrategy a simple model with and without an Embedding layer. In the former case the distribute training fails (see below). The following code is executed on two workers with TF_CONFIG={"cluster": {"worker": ["localhost:65535","localhost:65532"]}, "task": {"index": 0, "type": "worker"}} and TF_CONFIG={"cluster": {"worker": ["localhost:65535","localhost:65532"]}, "task": {"index": 1, "type": "worker"}}, respectively.
Code to reproduce the issue
import tensorflow as tf
from tensorflow.keras.optimizers import Adam
from tensorflow.keras import losses
import numpy as np

batch_size = 32

strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy()

NUM_WORKERS = strategy.num_replicas_in_sync
GLOBAL_BATCH_SIZE = batch_size * NUM_WORKERS
N_CAT = 47

def some_func(*args: tf.Tensor):
    tensor_dict_x, tensor_dict_y = {}, {}

    for index in range(1):
        tensor_dict_x[
            f"input_{index+1}"
        ] = tf.expand_dims(args[index], axis=-1)
        tensor_dict_y[
            f"dense"
        ] = tf.expand_dims(args[index], axis=-1)

    return tensor_dict_x, tensor_dict_y

def read_data():
    train_data = np.random.randint(1,N_CAT, size=9000)
    val_data = np.random.randint(1,N_CAT, size=999)

    dataset_train = (tf.data.Dataset.from_tensor_slices(train_data)
                     .prefetch(-1)
                     .map(map_func=some_func)
                     .batch(batch_size=GLOBAL_BATCH_SIZE)
                     .shuffle(1000)
                     .repeat())
    dataset_val = (tf.data.Dataset.from_tensor_slices(val_data)
                   .prefetch(-1)
                   .map(map_func=some_func)
                   .batch(batch_size=GLOBAL_BATCH_SIZE)
                   .shuffle(1000)
                   .repeat())

    return dataset_train, dataset_val

with strategy.scope():
    optimizer = Adam(lr=0.1)
    loss = losses.sparse_categorical_crossentropy
    model = build_and_compile_model(optimizer, loss)

dataset_train, dataset_val = read_data()
model.fit(x=dataset_train,
          epochs=5,
          steps_per_epoch=9000//batch_size,
          validation_data=dataset_val,
          validation_steps=999//batch_size,
)

where build_and_compile_model:
def build_and_compile_model(optimizer, loss):
    my_input = tf.keras.layers.Input(shape=(1,))
    my_dense = tf.keras.layers.Dense(N_CAT)(my_input)

    model = tf.keras.Model(my_input, my_dense)

    model.compile(optimizer=optimizer,loss=loss)

    return model

The above case works fine with the output:
2019-11-07 09:23:57.387629: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-11-07 09:23:57.410325: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency:<PHONE_NUMBER> Hz
2019-11-07 09:23:57.411228: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x562e2c78c6a0 executing computations on platform Host. Devices:
2019-11-07 09:23:57.411241: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
2019-11-07 09:23:57.413306: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:258] Initialize GrpcChannelCache for job worker -> {0 -> localhost:65535, 1 -> localhost:65532}
2019-11-07 09:23:57.414734: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:365] Started server with target: grpc://localhost:65535
WARNING:tensorflow:`eval_fn` is not passed in. The `worker_fn` will be used if an "evaluator" task exists in the cluster.
WARNING:tensorflow:`eval_strategy` is not passed in. No distribution strategy will be used for evaluation.
WARNING:tensorflow:ModelCheckpoint callback is not provided. Workers will need to restart training if any fails.
2019-11-07 09:24:09.204390: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:400] Cannot find shardable dataset, adding a shard node at the end of the dataset instead. This may have performance implications.
Train for 281 steps, validate for 31 steps
Epoch 1/5
2019-11-07 09:24:09.208710: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:400] Cannot find shardable dataset, adding a shard node at the end of the dataset instead. This may have performance implications.
281/281 [==============================] - 4s 13ms/step - loss: 15.9838 - val_loss: 16.1050
Epoch 2/5
281/281 [==============================] - 1s 3ms/step - loss: 16.0496 - val_loss: 16.1015
Epoch 3/5
281/281 [==============================] - 1s 3ms/step - loss: 16.0210 - val_loss: 16.1007
Epoch 4/5
281/281 [==============================] - 1s 3ms/step - loss: 16.0517 - val_loss: 16.0995
Epoch 5/5
281/281 [==============================] - 1s 3ms/step - loss: 16.0147 - val_loss: 16.0992
2019-11-07 09:24:16.090800: W tensorflow/core/common_runtime/eager/context.cc:290] Unable to destroy server_ object, so releasing instead. Servers don't support clean shutdown.

Process finished with exit code 0

However, adding just one Embedding layer so that build_and_compile_model now is:
def build_and_compile_model(optimizer, loss):

    my_input = tf.keras.layers.Input(shape=(1,))
    emb_layer = tf.keras.layers.Embedding(N_CAT,5)
    emb_inp = emb_layer(my_input)
    my_dense = tf.keras.layers.Dense(N_CAT)(emb_inp)

    model = tf.keras.Model(my_input, my_dense)

    model.compile(optimizer=optimizer,loss=loss)

    return model

leads to the error:
2019-11-07 09:26:45.812362: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-11-07 09:26:45.834388: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency:<PHONE_NUMBER> Hz
2019-11-07 09:26:45.835018: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55ea66430da0 executing computations on platform Host. Devices:
2019-11-07 09:26:45.835032: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
2019-11-07 09:26:45.836899: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:258] Initialize GrpcChannelCache for job worker -> {0 -> localhost:65535, 1 -> localhost:65532}
2019-11-07 09:26:45.838638: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:365] Started server with target: grpc://localhost:65535
WARNING:tensorflow:`eval_fn` is not passed in. The `worker_fn` will be used if an "evaluator" task exists in the cluster.
WARNING:tensorflow:`eval_strategy` is not passed in. No distribution strategy will be used for evaluation.
WARNING:tensorflow:ModelCheckpoint callback is not provided. Workers will need to restart training if any fails.
2019-11-07 09:26:50.351284: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:400] Cannot find shardable dataset, adding a shard node at the end of the dataset instead. This may have performance implications.
Train for 281 steps, validate for 31 steps
Epoch 1/5
2019-11-07 09:26:50.355581: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:400] Cannot find shardable dataset, adding a shard node at the end of the dataset instead. This may have performance implications.
  1/281 [..............................] - ETA: 9:05 - loss: 9.83752019-11-07 09:26:52.321935: E tensorflow/core/common_runtime/ring_alg.cc:279] Aborting RingGather with Internal: [_Derived_]Inconsistent output shapes, got [16], but expected is [64].
	 [[{{node Adam/allreduce_1/CollectiveGather_1}}]]
Additional GRPC error information:
{"created":"@1573115212.321897522","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"[_Derived_]Inconsistent output shapes, got [16], but expected is [64].\n\t [[{{node Adam/allreduce_1/CollectiveGather_1}}]]","grpc_status":13}
2019-11-07 09:26:52.321956: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Internal: [_Derived_]Inconsistent output shapes, got [16], but expected is [64].
	 [[{{node Adam/allreduce_1/CollectiveGather_1}}]]
Additional GRPC error information:
{"created":"@1573115212.321897522","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"[_Derived_]Inconsistent output shapes, got [16], but expected is [64].\n\t [[{{node Adam/allreduce_1/CollectiveGather_1}}]]","grpc_status":13}
2019-11-07 09:26:52.321966: E tensorflow/core/common_runtime/ring_alg.cc:279] Aborting RingReduce with Internal: [_Derived_]Inconsistent output shapes, got [16], but expected is [64].
	 [[{{node Adam/allreduce_1/CollectiveGather_1}}]]
Additional GRPC error information:
{"created":"@1573115212.321897522","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"[_Derived_]Inconsistent output shapes, got [16], but expected is [64].\n\t [[{{node Adam/allreduce_1/CollectiveGather_1}}]]","grpc_status":13}
2019-11-07 09:26:52.321971: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Internal: [_Derived_]Inconsistent output shapes, got [16], but expected is [64].
	 [[{{node Adam/allreduce_1/CollectiveGather_1}}]]
Additional GRPC error information:
{"created":"@1573115212.321897522","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"[_Derived_]Inconsistent output shapes, got [16], but expected is [64].\n\t [[{{node Adam/allreduce_1/CollectiveGather_1}}]]","grpc_status":13}
2019-11-07 09:26:52.321976: E tensorflow/core/common_runtime/ring_alg.cc:279] Aborting RingGather with Internal: [_Derived_]Inconsistent output shapes, got [16], but expected is [64].
	 [[{{node Adam/allreduce_1/CollectiveGather_1}}]]
Additional GRPC error information:
{"created":"@1573115212.321897522","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"[_Derived_]Inconsistent output shapes, got [16], but expected is [64].\n\t [[{{node Adam/allreduce_1/CollectiveGather_1}}]]","grpc_status":13}
2019-11-07 09:26:52.321996: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Internal: [_Derived_]Inconsistent output shapes, got [16], but expected is [64].
	 [[{{node Adam/allreduce_1/CollectiveGather_1}}]]
Additional GRPC error information:
{"created":"@1573115212.321897522","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"[_Derived_]Inconsistent output shapes, got [16], but expected is [64].\n\t [[{{node Adam/allreduce_1/CollectiveGather_1}}]]","grpc_status":13}
2019-11-07 09:26:52.322010: E tensorflow/core/common_runtime/ring_alg.cc:279] Aborting RingReduce with Internal: [_Derived_]Inconsistent output shapes, got [16], but expected is [64].
	 [[{{node Adam/allreduce_1/CollectiveGather_1}}]]
Additional GRPC error information:
{"created":"@1573115212.321897522","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"[_Derived_]Inconsistent output shapes, got [16], but expected is [64].\n\t [[{{node Adam/allreduce_1/CollectiveGather_1}}]]","grpc_status":13}
2019-11-07 09:26:52.322017: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Internal: [_Derived_]Inconsistent output shapes, got [16], but expected is [64].
	 [[{{node Adam/allreduce_1/CollectiveGather_1}}]]
Additional GRPC error information:
{"created":"@1573115212.321897522","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"[_Derived_]Inconsistent output shapes, got [16], but expected is [64].\n\t [[{{node Adam/allreduce_1/CollectiveGather_1}}]]","grpc_status":13}
2019-11-07 09:26:52.322045: E tensorflow/core/common_runtime/ring_alg.cc:279] Aborting RingReduce with Cancelled: [_Derived_]Cancelled
Additional GRPC error information:
{"created":"@1573115212.321980682","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Cancelled","grpc_status":1}
2019-11-07 09:26:52.322051: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Cancelled: [_Derived_]Cancelled
Additional GRPC error information:
{"created":"@1573115212.321980682","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Cancelled","grpc_status":1}
2019-11-07 09:26:52.322053: W tensorflow/core/framework/op_kernel.cc:1622] OP_REQUIRES failed at collective_ops.cc:125 : Internal: [_Derived_]Inconsistent output shapes, got [16], but expected is [64].
	 [[{{node Adam/allreduce_1/CollectiveGather_1}}]]
Additional GRPC error information:
{"created":"@1573115212.321897522","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"[_Derived_]Inconsistent output shapes, got [16], but expected is [64].\n\t [[{{node Adam/allreduce_1/CollectiveGather_1}}]]","grpc_status":13}
2019-11-07 09:26:52.322054: W tensorflow/core/framework/op_kernel.cc:1622] OP_REQUIRES failed at collective_ops.cc:234 : Internal: [_Derived_]Inconsistent output shapes, got [16], but expected is [64].
	 [[{{node Adam/allreduce_1/CollectiveGather_1}}]]
Additional GRPC error information:
{"created":"@1573115212.321897522","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"[_Derived_]Inconsistent output shapes, got [16], but expected is [64].\n\t [[{{node Adam/allreduce_1/CollectiveGather_1}}]]","grpc_status":13}
2019-11-07 09:26:52.322101: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Internal: [_Derived_]Inconsistent output shapes, got [16], but expected is [64].
	 [[{{node Adam/allreduce_1/CollectiveGather_1}}]]
Additional GRPC error information:
{"created":"@1573115212.321897522","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"[_Derived_]Inconsistent output shapes, got [16], but expected is [64].\n\t [[{{node Adam/allreduce_1/CollectiveGather_1}}]]","grpc_status":13}
	 [[Adam/allreduce_1/CollectiveGather]]
2019-11-07 09:26:52.322097: W tensorflow/core/framework/op_kernel.cc:1622] OP_REQUIRES failed at collective_ops.cc:125 : Internal: [_Derived_]Inconsistent output shapes, got [16], but expected is [64].
	 [[{{node Adam/allreduce_1/CollectiveGather_1}}]]
Additional GRPC error information:
{"created":"@1573115212.321897522","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"[_Derived_]Inconsistent output shapes, got [16], but expected is [64].\n\t [[{{node Adam/allreduce_1/CollectiveGather_1}}]]","grpc_status":13}
2019-11-07 09:26:52.322123: W tensorflow/core/framework/op_kernel.cc:1622] OP_REQUIRES failed at collective_ops.cc:234 : Cancelled: [_Derived_]Cancelled
Additional GRPC error information:
{"created":"@1573115212.321980682","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Cancelled","grpc_status":1}
2019-11-07 09:26:52.322108: W tensorflow/core/framework/op_kernel.cc:1622] OP_REQUIRES failed at collective_ops.cc:234 : Internal: [_Derived_]Inconsistent output shapes, got [16], but expected is [64].
	 [[{{node Adam/allreduce_1/CollectiveGather_1}}]]
Additional GRPC error information:
{"created":"@1573115212.321897522","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"[_Derived_]Inconsistent output shapes, got [16], but expected is [64].\n\t [[{{node Adam/allreduce_1/CollectiveGather_1}}]]","grpc_status":13}
Traceback (most recent call last):
  File ".../.PyCharmCE2019.2/config/scratches/scratch_4.py", line 71, in <module>
    validation_steps=999//batch_size,
  File ".../lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py", line 728, in fit
    use_multiprocessing=use_multiprocessing)
  File .../lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_distributed.py", line 789, in fit
    *args, **kwargs)
  File ".../lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_distributed.py", line 776, in wrapper
    mode=dc.CoordinatorMode.INDEPENDENT_WORKER)
  File ".../lib/python3.7/site-packages/tensorflow_core/python/distribute/distribute_coordinator.py", line 853, in run_distribute_coordinator
    task_id, session_config, rpc_layer)
  File ".../mostly-engine-tf20-p374/lib/python3.7/site-packages/tensorflow_core/python/distribute/distribute_coordinator.py", line 360, in _run_single_worker
    return worker_fn(strategy)
  File ".../mostly-engine-tf20-p374/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_distributed.py", line 771, in _worker_fn
    return method(model, **kwargs)
  File ".../mostly-engine-tf20-p374/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 324, in fit
    total_epochs=epochs)
  File ".../mostly-engine-tf20-p374/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 123, in run_one_epoch
    batch_outs = execution_function(iterator)
  File ".../mostly-engine-tf20-p374/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 86, in execution_function
    distributed_function(input_fn))
  File ".../mostly-engine-tf20-p374/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 457, in __call__
    result = self._call(*args, **kwds)
  File ".../mostly-engine-tf20-p374/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 487, in _call
    return self._stateless_fn(*args, **kwds)  # pylint: disable=not-callable
  File ".../mostly-engine-tf20-p374/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1823, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "/.../mostly-engine-tf20-p374/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1141, in _filtered_call
    self.captured_inputs)
  File ".../lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1224, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager)
  File ".../lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 511, in call
    ctx=ctx)
  File ".../lib/python3.7/site-packages/tensorflow_core/python/eager/execute.py", line 67, in quick_execute
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InternalError:  [_Derived_]Inconsistent output shapes, got [16], but expected is [64].
	 [[node Adam/allreduce_1/CollectiveGather_1 (defined at /miniconda3/envs/mostly-engine-tf20-p374/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1751) ]]
Additional GRPC error information:
{"created":"@1573115212.321897522","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"[_Derived_]Inconsistent output shapes, got [16], but expected is [64].\n\t [[node Adam/allreduce_1/CollectiveGather_1 (defined at /miniconda3/envs/mostly-engine-tf20-p374/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1751) ]]","grpc_status":13}
	 [[Adam/allreduce_1/CollectiveGather]] [Op:__inference_distributed_function_888]

Function call stack:
distributed_function

2019-11-07 09:26:52.540347: W tensorflow/core/common_runtime/eager/context.cc:290] Unable to destroy server_ object, so releasing instead. Servers don't support clean shutdown.

Process finished with exit code 1

Sounds like a similar issue to #33339, which has been fixed. Closing this issue now, but feel free to reopen if this is still a problem.


Launch Pad Lamp v1.0 from Github

My Launch Pad Lamp mod.   My KSP forum username is Trolllception.

Not sure why this is failing.  I am able to run netkan against the file and it generates a .CKAN file and installs.

CKAN identifiers must consist only of letters, numbers, and dashes, and must start with a letter or number.

Remove the spaces from the identifier

Hmm again I am at a loss here.

Looks great now. Merging with thanks!


chore(main): release 0.2.1

:robot: I have created a release beep boop
0.2.1 (2024-02-19)
Bug Fixes

allow creation of multiple rate limiters of same HTTP path but different methods (#38) (60a0e97)
emit default values http server (#48) (795bb40)
surface fetch challenge cache get error (#49) (44ad3eb)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/kyamalabs/auth/releases/tag/v0.2.1 :sunflower:


Merge bitcoin#14822,  bitcoin#14903, bitcoin#14631, bitcoin#12246,bitcoin#14008

@UdjinM6 @PastaPastaPasta

14468
This is marked as DNM in the Sheet ,  also in the Notes column mentioned "we have equivalent " but I am not aware what is that equivalent

DNM (Did Not Merge)

3668bb3
Merge bitcoin#14468: [wallet] Deprecate generate RPC method

14631 requires 14468  (Since 14468 is DNM and has an equivalent, trying to find its equivalent)
12248 requires 14180 (#4426) (waiting for this PR to merge)
Applied 14008 to Dash specific rpcs too

I haven't looked at this super closely, but it looks like we should partially backport 14468 (ie making (most of) the changes that are done in the test suite, but not actually removing the generate RPC), that should then allow you to fully backport 14631

@UdjinM6
I am not merging
12248 requires 14180 (#4426)

Few notes:

14631 requires 14468
> * 12248 requires 14180 (#4426)
should probably apply 14008 to Dash specific rpcs too
see below

Hello @UdjinM6  I am not merging 12248 , Did you mean 14822 or 12246?
This PR contains merge for (Merge bitcoin#14822, bitcoin#14903, bitcoin#14631, bitcoin#12246,bitcoin#14008 )

Hello @PastaPastaPasta @UdjinM6  As suggested I have partially backported 14468 (ie making (most of) the changes that are done in the test suite, but not actually removing the generate RPC), and then fully backported 14631 (basically test_node.py file)

This PR has become too messy , will close this , Opened new PR https://github.com/dashpay/dash/pull/4452 with all review comments implemented

No, don't do that, we shouldn't be opening new PRs like this. Just force push your new branch vijaydasmp:backport_v18_vijay_batch4_2_new to this branch vijaydasmp:backport_v18_vijay_batch4_2

Got it
On Sat, 25 Sep 2021, 00:11 PastaPastaPasta, @.***>
wrote:

No, don't do that, we shouldn't be opening new PRs like this. Just force
push your new branch vijaydasmp:backport_v18_vijay_batch4_2_new to this
branch vijaydasmp:backport_v18_vijay_batch4_2
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dashpay/dash/pull/4448#issuecomment-926845040, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAR5IIQXQZWWCD57DSYESB3UDTA45ANCNFSM5ENEZYLQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

No, don't do that, we shouldn't be opening new PRs like this. Just force push your new branch vijaydasmp:backport_v18_vijay_batch4_2_new to this branch vijaydasmp:backport_v18_vijay_batch4_2

Hello @PastaPastaPasta as suggested , force pushed new branch vijaydasmp:backport_v18_vijay_batch4_2

Merge #14282: [wallet] Remove -usehd (why is this DNM) , The upstream seems to have removed it

Hello, @UdjinM6 @PastaPastaPasta, on this PR currently trying to understand wallet_hd.py test failure, cant seem to figure it out yet though

This PR has become too messy , will close this , Opened new PR https://github.com/dashpay/dash/pull/4452 with all review comments implemented ,closed #4452

:ɪ •ᴡɪʟʟ •ɪɴᴛʀᴏᴅᴜᴄᴇ •ʏᴏᴜ •on• how •to •make• profit•through•ɪɴᴠᴇꜱᴛᴍᴇɴᴛ Text' me' on' •ᴡʜᴀᴛꜱᴀᴘᴘ••
+••1••3••2••3••4••5••2••5••4••7••6••√

No, don't do that, we shouldn't be opening new PRs like this. Just force push your new branch vijaydasmp:backport_v18_vijay_batch4_2_new to this branch vijaydasmp:backport_v18_vijay_batch4_2

:ɪ •ᴡɪʟʟ •ɪɴᴛʀᴏᴅᴜᴄᴇ •ʏᴏᴜ •on• how •to •make• profit•through•ɪɴᴠᴇꜱᴛᴍᴇɴᴛ Text' me' on' •ᴡʜᴀᴛꜱᴀᴘᴘ••
+••1••3••2••3••4••5••2••5••4••7••6••√

HELLO EVERYONE, :ɪ •ᴡɪʟʟ •ɪɴᴛʀᴏᴅᴜᴄᴇ •ʏᴏᴜ •on• how •to •make• profit•through•ɪɴᴠᴇꜱᴛᴍᴇɴᴛ Text' me' on' •ᴡʜᴀᴛꜱᴀᴘᴘ••
+13234525476.


Wait until first render is done before starting up cron job.

It was possible that cron job was ran while first render was still running, causing problems when two threads simultaneously wrote same output image.
This PR makes the first render synchronous operation and starts cron job after the first image has finished.

Very good spot, thanks a lot for the PR!


Improve LUCATS Classifications

User should have some notion of what applications are still in progress (and they can get involved in) vs those that are "complete".  This will require some additional work to scrape LUCATS a bit more deeply, but should be possible.

Closed by #300


KIP-392: Allow consumers to fetch from closest replica

KIP-392: Allow consumers to fetch from closest replica

@dneralla thanks for contributing!
I've taken a first pass over the changes and they look good. Please can you sign the CLA?
Have you tried them out against a real 2.4/2.5 cluster with rack aware support enabled?

@dnwe Signed the CLA.  Not sure how this CI is integrated it seems to be not running again.
I haven't tried out yet on 2.4 or 2.5 cluster. Will do some time later this week.

Thanks for contributing 🙏 I've re-triggered CI.

Do you think you could rebase your branch off of latest master please?

@dneralla did you get around to testing your changes against a 2.4/2.5 cluster yet?

The positive test case, to verify the feature, though, the replica.selector.class broker configuration must be set to the fully qualified class name that implements ReplicaSelector.  This configuration item should be set to org.apache.kafka.common.replica.RackAwareReplicaSelector

What's the ETA towards merge?

@marc-ostrow still waiting for some confirmation that this has been tested against a properly configured cluster and shown to work before we can merge the feature. I haven’t had time to do this testing myself sadly

I haven't found time to test with new kafka cluster as well sadly.

I have a small 3-az MSK cluster running Kafka 2.4.1.  On May 20th, I realized I needed to set the cluster configuration to include replica.selector.class set to org.apache.kafka.common.replica.RackAwareReplicaSelector so I did that.
Today, I cloned https://github.com/dneralla/sarama.git and added a little additional log output to the kip-392 branch.  I then wrote a small test driver app in ./examples/kip392verify.  I ran it on a utility instance for that MSK cluster.

The utility instance is in us-east-1a (rack: use1-az1),
and the particular topic's 0 partition's leader is on a broker in us-east-1b (rack: use1-az2).

My consumer started out on the leader broker and stayed there.  FetchResponseBlock.decode was receiving PreferredReadReplica set to -1.
I searched Cloudwatch broker logs to see what the cluster configuration was set to.  The following snippet confirmed to me that I had successfully updated the cluster configuration; this from the broker log just after the restart.  Notice the values for replica.selector.class and broker.rack.
CloudWatch Logs Insights
region: us-east-1
log-group-names: /aws/msk/system-events-dev-blue-ref20
start-time: 2020-05-21T01:46:00.000Z
end-time: 2020-05-21T01:47:00.000Z
query-string:
filter @logStream = 'system-events-dev-blue-ref20-e2a3ab84-b5da-4f6b-b434-f0b0a91dfcca-5-Broker-1' and @timestamp ><PHONE_NUMBER>000 and @timestamp <<PHONE_NUMBER>000
 | fields @timestamp, @message

Click to expand

@timestamp

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,333] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,805] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,805] INFO starting (kafka.server.KafkaServer)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,806] INFO Connecting to zookeeper on INTERNAL_ZK_DNS,INTERNAL_ZK_DNS,INTERNAL_ZK_DNS, (kafka.server.KafkaServer)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,821] INFO [ZooKeeperClient Kafka server] Initializing a new session to INTERNAL_ZK_DNS,INTERNAL_ZK_DNS,INTERNAL_ZK_DNS,. (kafka.zookeeper.ZooKeeperClient)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,826] INFO Client environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT (org.apache.zookeeper.ZooKeeper)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,826] INFO Client environment:host.name=HOST_NAME (org.apache.zookeeper.ZooKeeper)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,826] INFO Client environment:java.version=1.8.0_242 (org.apache.zookeeper.ZooKeeper)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,826] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,826] INFO Client environment:java.home=JAVA_HOME (org.apache.zookeeper.ZooKeeper)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,827] INFO Client environment:java.class.path=CLASSPATH (org.apache.zookeeper.ZooKeeper)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,827] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,827] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,827] INFO Client environment:java.compiler= (org.apache.zookeeper.ZooKeeper)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,827] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,827] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,827] INFO Client environment:os.version=4.14.171-136.231.amzn2.x86_64 (org.apache.zookeeper.ZooKeeper)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,827] INFO Client environment:user.name=kafka-user (org.apache.zookeeper.ZooKeeper)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,827] INFO Client environment:user.home=/home/kafka-user (org.apache.zookeeper.ZooKeeper)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,827] INFO Client environment:user.dir=/ (org.apache.zookeeper.ZooKeeper)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,827] INFO Client environment:os.memory.free=1996MB (org.apache.zookeeper.ZooKeeper)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,827] INFO Client environment:os.memory.max=6144MB (org.apache.zookeeper.ZooKeeper)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,827] INFO Client environment:os.memory.total=2048MB (org.apache.zookeeper.ZooKeeper)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,829] INFO Initiating client connection, connectString=INTERNAL_ZK_DNS,INTERNAL_ZK_DNS,INTERNAL_ZK_DNS, sessionTimeout=18000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@59fd97a8 (org.apache.zookeeper.ZooKeeper)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,833] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,837] INFO jute.maxbuffer value is 10485759 Bytes (org.apache.zookeeper.ClientCnxnSocket)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,842] INFO zookeeper.request.timeout value is 0. feature enabled= (org.apache.zookeeper.ClientCnxn)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,843] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,904] INFO Opening socket connection to server INTERNAL_ZK_DNS/INTERNAL_IP. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,910] INFO Socket connection established, initiating session, client: /INTERNAL_IP, server: INTERNAL_ZK_DNS/INTERNAL_IP (org.apache.zookeeper.ClientCnxn)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,916] INFO Session establishment complete on server INTERNAL_ZK_DNS/INTERNAL_IP, sessionid = 0x100000953f00019, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn)

2020-05-21 01:46:01.000
[2020-05-21 01:46:00,919] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)

2020-05-21 01:46:01.000
[2020-05-21 01:46:01,130] INFO Cluster ID = _HDXTbbgS4eGlQYqtScwBg (kafka.server.KafkaServer)

2020-05-21 01:46:01.000
[2020-05-21 01:46:01,195] INFO KafkaConfig values:

2020-05-21 01:46:01.000
advertised.host.name = null

2020-05-21 01:46:01.000
advertised.listeners = null

2020-05-21 01:46:01.000
advertised.port = null

2020-05-21 01:46:01.000
alter.config.policy.class.name = null

2020-05-21 01:46:01.000
alter.log.dirs.replication.quota.window.num = 11

2020-05-21 01:46:01.000
alter.log.dirs.replication.quota.window.size.seconds = 1

2020-05-21 01:46:01.000
authorizer.class.name = kafka.security.auth.SimpleAclAuthorizer

2020-05-21 01:46:01.000
auto.create.topics.enable = false

2020-05-21 01:46:01.000
auto.leader.rebalance.enable = true

2020-05-21 01:46:01.000
background.threads = 10

2020-05-21 01:46:01.000
broker.id = 1

2020-05-21 01:46:01.000
broker.id.generation.enable = false

2020-05-21 01:46:01.000
broker.rack = use1-az2

2020-05-21 01:46:01.000
client.quota.callback.class = null

2020-05-21 01:46:01.000
compression.type = producer

2020-05-21 01:46:01.000
connection.failed.authentication.delay.ms = 100

2020-05-21 01:46:01.000
connections.max.idle.ms = 600000

2020-05-21 01:46:01.000
connections.max.reauth.ms = 0

2020-05-21 01:46:01.000
control.plane.listener.name = null

2020-05-21 01:46:01.000
controlled.shutdown.enable = true

2020-05-21 01:46:01.000
controlled.shutdown.max.retries = 3

2020-05-21 01:46:01.000
controlled.shutdown.retry.backoff.ms = 5000

2020-05-21 01:46:01.000
controller.socket.timeout.ms = 30000

2020-05-21 01:46:01.000
create.topic.policy.class.name = null

2020-05-21 01:46:01.000
default.replication.factor = 3

2020-05-21 01:46:01.000
delegation.token.expiry.check.interval.ms = 3600000

2020-05-21 01:46:01.000
delegation.token.expiry.time.ms = 86400000

2020-05-21 01:46:01.000
delegation.token.master.key = null

2020-05-21 01:46:01.000
delegation.token.max.lifetime.ms = 604800000

2020-05-21 01:46:01.000
delete.records.purgatory.purge.interval.requests = 1

2020-05-21 01:46:01.000
delete.topic.enable = true

2020-05-21 01:46:01.000
fetch.purgatory.purge.interval.requests = 1000

2020-05-21 01:46:01.000
group.initial.rebalance.delay.ms = 3000

2020-05-21 01:46:01.000
group.max.session.timeout.ms = 1800000

2020-05-21 01:46:01.000
group.max.size =<PHONE_NUMBER>

2020-05-21 01:46:01.000
group.min.session.timeout.ms = 6000

2020-05-21 01:46:01.000
host.name =

2020-05-21 01:46:01.000
inter.broker.listener.name = REPLICATION

2020-05-21 01:46:01.000
inter.broker.protocol.version = 2.4.1

2020-05-21 01:46:01.000
kafka.metrics.polling.interval.secs = 10

2020-05-21 01:46:01.000
kafka.metrics.reporters = []

2020-05-21 01:46:01.000
leader.imbalance.check.interval.seconds = 300

2020-05-21 01:46:01.000
leader.imbalance.per.broker.percentage = 10

2020-05-21 01:46:01.000
listener.security.protocol.map = CLIENT:PLAINTEXT,CLIENT_SECURE:SSL,REPLICATION:PLAINTEXT,REPLICATION_SECURE:SSL

2020-05-21 01:46:01.000
listeners = CLIENT://b-1.system-events-dev-blue.uwjleq.c5.kafka.us-east-1.amazonaws.com:9092,REPLICATION://INTERNAL_BROKER_DNS

2020-05-21 01:46:01.000
log.cleaner.backoff.ms = 15000

2020-05-21 01:46:01.000
log.cleaner.dedupe.buffer.size = 134217728

2020-05-21 01:46:01.000
log.cleaner.delete.retention.ms = 86400000

2020-05-21 01:46:01.000
log.cleaner.enable = true

2020-05-21 01:46:01.000
log.cleaner.io.buffer.load.factor = 0.9

2020-05-21 01:46:01.000
log.cleaner.io.buffer.size = 524288

2020-05-21 01:46:01.000
log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308

2020-05-21 01:46:01.000
log.cleaner.max.compaction.lag.ms =<PHONE_NUMBER>854775807

2020-05-21 01:46:01.000
log.cleaner.min.cleanable.ratio = 0.5

2020-05-21 01:46:01.000
log.cleaner.min.compaction.lag.ms = 0

2020-05-21 01:46:01.000
log.cleaner.threads = 1

2020-05-21 01:46:01.000
log.cleanup.policy = [delete]

2020-05-21 01:46:01.000
log.flush.interval.messages =<PHONE_NUMBER>854775807

2020-05-21 01:46:01.000
log.flush.interval.ms = null

2020-05-21 01:46:01.000
log.flush.offset.checkpoint.interval.ms = 60000

2020-05-21 01:46:01.000
log.flush.scheduler.interval.ms =<PHONE_NUMBER>854775807

2020-05-21 01:46:01.000
log.flush.start.offset.checkpoint.interval.ms = 60000

2020-05-21 01:46:01.000
log.index.interval.bytes = 4096

2020-05-21 01:46:01.000
log.index.size.max.bytes = 10485760

2020-05-21 01:46:01.000
log.message.downconversion.enable = true

2020-05-21 01:46:01.000
log.message.format.version = 2.4.1

2020-05-21 01:46:01.000
log.message.timestamp.difference.max.ms =<PHONE_NUMBER>854775807

2020-05-21 01:46:01.000
log.message.timestamp.type = CreateTime

2020-05-21 01:46:01.000
log.preallocate = false

2020-05-21 01:46:01.000
log.retention.bytes = -1

2020-05-21 01:46:01.000
log.retention.check.interval.ms = 300000

2020-05-21 01:46:01.000
log.retention.hours = 168

2020-05-21 01:46:01.000
log.retention.minutes = null

2020-05-21 01:46:01.000
log.retention.ms = null

2020-05-21 01:46:01.000
log.roll.hours = 168

2020-05-21 01:46:01.000
log.roll.jitter.hours = 0

2020-05-21 01:46:01.000
log.roll.jitter.ms = null

2020-05-21 01:46:01.000
log.roll.ms = null

2020-05-21 01:46:01.000
log.segment.bytes =<PHONE_NUMBER>

2020-05-21 01:46:01.000
log.segment.delete.delay.ms = 60000

2020-05-21 01:46:01.000
max.connections =<PHONE_NUMBER>

2020-05-21 01:46:01.000
max.connections.per.ip =<PHONE_NUMBER>

2020-05-21 01:46:01.000
max.connections.per.ip.overrides =

2020-05-21 01:46:01.000
max.incremental.fetch.session.cache.slots = 1000

2020-05-21 01:46:01.000
message.max.bytes = 1000012

2020-05-21 01:46:01.000
metric.reporters = []

2020-05-21 01:46:01.000
metrics.num.samples = 2

2020-05-21 01:46:01.000
metrics.recording.level = INFO

2020-05-21 01:46:01.000
metrics.sample.window.ms = 30000

2020-05-21 01:46:01.000
min.insync.replicas = 2

2020-05-21 01:46:01.000
num.io.threads = 8

2020-05-21 01:46:01.000
num.network.threads = 5

2020-05-21 01:46:01.000
num.partitions = 1

2020-05-21 01:46:01.000
num.recovery.threads.per.data.dir = 2

2020-05-21 01:46:01.000
num.replica.alter.log.dirs.threads = null

2020-05-21 01:46:01.000
num.replica.fetchers = 2

2020-05-21 01:46:01.000
offset.metadata.max.bytes = 4096

2020-05-21 01:46:01.000
offsets.commit.required.acks = -1

2020-05-21 01:46:01.000
offsets.commit.timeout.ms = 5000

2020-05-21 01:46:01.000
offsets.load.buffer.size = 5242880

2020-05-21 01:46:01.000
offsets.retention.check.interval.ms = 600000

2020-05-21 01:46:01.000
offsets.retention.minutes = 10080

2020-05-21 01:46:01.000
offsets.topic.compression.codec = 0

2020-05-21 01:46:01.000
offsets.topic.num.partitions = 50

2020-05-21 01:46:01.000
offsets.topic.replication.factor = 3

2020-05-21 01:46:01.000
offsets.topic.segment.bytes = 104857600

2020-05-21 01:46:01.000
password.encoder.cipher.algorithm = AES/CBC/PKCS5Padding

2020-05-21 01:46:01.000
password.encoder.iterations = 4096

2020-05-21 01:46:01.000
password.encoder.key.length = 128

2020-05-21 01:46:01.000
password.encoder.keyfactory.algorithm = null

2020-05-21 01:46:01.000
password.encoder.old.secret = null

2020-05-21 01:46:01.000
password.encoder.secret = null

2020-05-21 01:46:01.000
port = 9092

2020-05-21 01:46:01.000
principal.builder.class = null

2020-05-21 01:46:01.000
producer.purgatory.purge.interval.requests = 1000

2020-05-21 01:46:01.000
queued.max.request.bytes = -1

2020-05-21 01:46:01.000
queued.max.requests = 500

2020-05-21 01:46:01.000
quota.consumer.default =<PHONE_NUMBER>854775807

2020-05-21 01:46:01.000
quota.producer.default =<PHONE_NUMBER>854775807

2020-05-21 01:46:01.000
quota.window.num = 11

2020-05-21 01:46:01.000
quota.window.size.seconds = 1

2020-05-21 01:46:01.000
replica.fetch.backoff.ms = 1000

2020-05-21 01:46:01.000
replica.fetch.max.bytes = 1048576

2020-05-21 01:46:01.000
replica.fetch.min.bytes = 1

2020-05-21 01:46:01.000
replica.fetch.response.max.bytes = 10485760

2020-05-21 01:46:01.000
replica.fetch.wait.max.ms = 500

2020-05-21 01:46:01.000
replica.high.watermark.checkpoint.interval.ms = 5000

2020-05-21 01:46:01.000
replica.lag.time.max.ms = 30000

2020-05-21 01:46:01.000
replica.selector.class = org.apache.kafka.common.replica.RackAwareReplicaSelector

2020-05-21 01:46:01.000
replica.socket.receive.buffer.bytes = 65536

2020-05-21 01:46:01.000
replica.socket.timeout.ms = 30000

2020-05-21 01:46:01.000
replication.quota.window.num = 11

2020-05-21 01:46:01.000
replication.quota.window.size.seconds = 1

2020-05-21 01:46:01.000
request.timeout.ms = 30000

2020-05-21 01:46:01.000
reserved.broker.max.id = 2000

2020-05-21 01:46:01.000
sasl.client.callback.handler.class = null

2020-05-21 01:46:01.000
sasl.enabled.mechanisms = [GSSAPI]

2020-05-21 01:46:01.000
sasl.jaas.config = null

2020-05-21 01:46:01.000
sasl.kerberos.kinit.cmd = /usr/bin/kinit

2020-05-21 01:46:01.000
sasl.kerberos.min.time.before.relogin = 60000

2020-05-21 01:46:01.000
sasl.kerberos.principal.to.local.rules = [DEFAULT]

2020-05-21 01:46:01.000
sasl.kerberos.service.name = null

2020-05-21 01:46:01.000
sasl.kerberos.ticket.renew.jitter = 0.05

2020-05-21 01:46:01.000
sasl.kerberos.ticket.renew.window.factor = 0.8

2020-05-21 01:46:01.000
sasl.login.callback.handler.class = null

2020-05-21 01:46:01.000
sasl.login.class = null

2020-05-21 01:46:01.000
sasl.login.refresh.buffer.seconds = 300

2020-05-21 01:46:01.000
sasl.login.refresh.min.period.seconds = 60

2020-05-21 01:46:01.000
sasl.login.refresh.window.factor = 0.8

2020-05-21 01:46:01.000
sasl.login.refresh.window.jitter = 0.05

2020-05-21 01:46:01.000
sasl.mechanism.inter.broker.protocol = GSSAPI

2020-05-21 01:46:01.000
sasl.server.callback.handler.class = null

2020-05-21 01:46:01.000
security.inter.broker.protocol = PLAINTEXT

2020-05-21 01:46:01.000
security.providers = null

2020-05-21 01:46:01.000
socket.receive.buffer.bytes = 102400

2020-05-21 01:46:01.000
socket.request.max.bytes = 104857600

2020-05-21 01:46:01.000
socket.send.buffer.bytes = 102400

2020-05-21 01:46:01.000
ssl.cipher.suites = []

2020-05-21 01:46:01.000
ssl.client.auth = none

2020-05-21 01:46:01.000
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]

2020-05-21 01:46:01.000
ssl.endpoint.identification.algorithm = https

2020-05-21 01:46:01.000
ssl.key.password = null

2020-05-21 01:46:01.000
ssl.keymanager.algorithm = SunX509

2020-05-21 01:46:01.000
ssl.keystore.password = null

2020-05-21 01:46:01.000
ssl.keystore.type = JKS

2020-05-21 01:46:01.000
ssl.principal.mapping.rules = DEFAULT

2020-05-21 01:46:01.000
ssl.protocol = TLS

2020-05-21 01:46:01.000
ssl.provider = null

2020-05-21 01:46:01.000
ssl.secure.random.implementation = null

2020-05-21 01:46:01.000
ssl.trustmanager.algorithm = PKIX

2020-05-21 01:46:01.000
ssl.truststore.password = null

2020-05-21 01:46:01.000
ssl.truststore.type = JKS

2020-05-21 01:46:01.000
transaction.abort.timed.out.transaction.cleanup.interval.ms = 60000

2020-05-21 01:46:01.000
transaction.max.timeout.ms = 900000

2020-05-21 01:46:01.000
transaction.remove.expired.transaction.cleanup.interval.ms = 3600000

2020-05-21 01:46:01.000
transaction.state.log.load.buffer.size = 5242880

2020-05-21 01:46:01.000
transaction.state.log.min.isr = 2

2020-05-21 01:46:01.000
transaction.state.log.num.partitions = 50

2020-05-21 01:46:01.000
transaction.state.log.replication.factor = 3

2020-05-21 01:46:01.000
transaction.state.log.segment.bytes = 104857600

2020-05-21 01:46:01.000
transactional.id.expiration.ms = 604800000

2020-05-21 01:46:01.000
unclean.leader.election.enable = true

2020-05-21 01:46:01.000
zookeeper.connect = INTERNAL_ZK_DNS,INTERNAL_ZK_DNS,INTERNAL_ZK_DNS,

2020-05-21 01:46:01.000
zookeeper.connection.timeout.ms = null

2020-05-21 01:46:01.000
zookeeper.max.in.flight.requests = 10

2020-05-21 01:46:01.000
zookeeper.session.timeout.ms = 18000

2020-05-21 01:46:01.000
zookeeper.set.acl = false

2020-05-21 01:46:01.000
zookeeper.sync.time.ms = 2000

2020-05-21 01:46:01.000
(kafka.server.KafkaConfig)

2020-05-21 01:46:01.000
[2020-05-21 01:46:01,205] INFO KafkaConfig values:

2020-05-21 01:46:01.000
advertised.host.name = null

2020-05-21 01:46:01.000
advertised.listeners = null

2020-05-21 01:46:01.000
advertised.port = null

2020-05-21 01:46:01.000
alter.config.policy.class.name = null

2020-05-21 01:46:01.000
alter.log.dirs.replication.quota.window.num = 11

2020-05-21 01:46:01.000
alter.log.dirs.replication.quota.window.size.seconds = 1

2020-05-21 01:46:01.000
authorizer.class.name = kafka.security.auth.SimpleAclAuthorizer

2020-05-21 01:46:01.000
auto.create.topics.enable = false

2020-05-21 01:46:01.000
auto.leader.rebalance.enable = true

2020-05-21 01:46:01.000
background.threads = 10

2020-05-21 01:46:01.000
broker.id = 1

2020-05-21 01:46:01.000
broker.id.generation.enable = false

2020-05-21 01:46:01.000
broker.rack = use1-az2

2020-05-21 01:46:01.000
client.quota.callback.class = null

2020-05-21 01:46:01.000
compression.type = producer

2020-05-21 01:46:01.000
connection.failed.authentication.delay.ms = 100

2020-05-21 01:46:01.000
connections.max.idle.ms = 600000

2020-05-21 01:46:01.000
connections.max.reauth.ms = 0

2020-05-21 01:46:01.000
control.plane.listener.name = null

2020-05-21 01:46:01.000
controlled.shutdown.enable = true

2020-05-21 01:46:01.000
controlled.shutdown.max.retries = 3

2020-05-21 01:46:01.000
controlled.shutdown.retry.backoff.ms = 5000

2020-05-21 01:46:01.000
controller.socket.timeout.ms = 30000

2020-05-21 01:46:01.000
create.topic.policy.class.name = null

2020-05-21 01:46:01.000
default.replication.factor = 3

2020-05-21 01:46:01.000
delegation.token.expiry.check.interval.ms = 3600000

2020-05-21 01:46:01.000
delegation.token.expiry.time.ms = 86400000

2020-05-21 01:46:01.000
delegation.token.master.key = null

2020-05-21 01:46:01.000
delegation.token.max.lifetime.ms = 604800000

2020-05-21 01:46:01.000
delete.records.purgatory.purge.interval.requests = 1

2020-05-21 01:46:01.000
delete.topic.enable = true

2020-05-21 01:46:01.000
fetch.purgatory.purge.interval.requests = 1000

2020-05-21 01:46:01.000
group.initial.rebalance.delay.ms = 3000

2020-05-21 01:46:01.000
group.max.session.timeout.ms = 1800000

2020-05-21 01:46:01.000
group.max.size =<PHONE_NUMBER>

2020-05-21 01:46:01.000
group.min.session.timeout.ms = 6000

2020-05-21 01:46:01.000
host.name =

2020-05-21 01:46:01.000
inter.broker.listener.name = REPLICATION

2020-05-21 01:46:01.000
inter.broker.protocol.version = 2.4.1

2020-05-21 01:46:01.000
kafka.metrics.polling.interval.secs = 10

2020-05-21 01:46:01.000
kafka.metrics.reporters = []

2020-05-21 01:46:01.000
leader.imbalance.check.interval.seconds = 300

2020-05-21 01:46:01.000
leader.imbalance.per.broker.percentage = 10

2020-05-21 01:46:01.000
listener.security.protocol.map = CLIENT:PLAINTEXT,CLIENT_SECURE:SSL,REPLICATION:PLAINTEXT,REPLICATION_SECURE:SSL

2020-05-21 01:46:01.000
listeners = CLIENT://b-1.system-events-dev-blue.uwjleq.c5.kafka.us-east-1.amazonaws.com:9092,REPLICATION://INTERNAL_BROKER_DNS

2020-05-21 01:46:01.000
log.cleaner.backoff.ms = 15000

2020-05-21 01:46:01.000
log.cleaner.dedupe.buffer.size = 134217728

2020-05-21 01:46:01.000
log.cleaner.delete.retention.ms = 86400000

2020-05-21 01:46:01.000
log.cleaner.enable = true

2020-05-21 01:46:01.000
log.cleaner.io.buffer.load.factor = 0.9

2020-05-21 01:46:01.000
log.cleaner.io.buffer.size = 524288

2020-05-21 01:46:01.000
log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308

2020-05-21 01:46:01.000
log.cleaner.max.compaction.lag.ms =<PHONE_NUMBER>854775807

2020-05-21 01:46:01.000
log.cleaner.min.cleanable.ratio = 0.5

2020-05-21 01:46:01.000
log.cleaner.min.compaction.lag.ms = 0

2020-05-21 01:46:01.000
log.cleaner.threads = 1

2020-05-21 01:46:01.000
log.cleanup.policy = [delete]

2020-05-21 01:46:01.000
log.flush.interval.messages =<PHONE_NUMBER>854775807

2020-05-21 01:46:01.000
log.flush.interval.ms = null

2020-05-21 01:46:01.000
log.flush.offset.checkpoint.interval.ms = 60000

2020-05-21 01:46:01.000
log.flush.scheduler.interval.ms =<PHONE_NUMBER>854775807

2020-05-21 01:46:01.000
log.flush.start.offset.checkpoint.interval.ms = 60000

2020-05-21 01:46:01.000
log.index.interval.bytes = 4096

2020-05-21 01:46:01.000
log.index.size.max.bytes = 10485760

2020-05-21 01:46:01.000
log.message.downconversion.enable = true

2020-05-21 01:46:01.000
log.message.format.version = 2.4.1

2020-05-21 01:46:01.000
log.message.timestamp.difference.max.ms =<PHONE_NUMBER>854775807

2020-05-21 01:46:01.000
log.message.timestamp.type = CreateTime

2020-05-21 01:46:01.000
log.preallocate = false

2020-05-21 01:46:01.000
log.retention.bytes = -1

2020-05-21 01:46:01.000
log.retention.check.interval.ms = 300000

2020-05-21 01:46:01.000
log.retention.hours = 168

2020-05-21 01:46:01.000
log.retention.minutes = null

2020-05-21 01:46:01.000
log.retention.ms = null

2020-05-21 01:46:01.000
log.roll.hours = 168

2020-05-21 01:46:01.000
log.roll.jitter.hours = 0

2020-05-21 01:46:01.000
log.roll.jitter.ms = null

2020-05-21 01:46:01.000
log.roll.ms = null

2020-05-21 01:46:01.000
log.segment.bytes =<PHONE_NUMBER>

2020-05-21 01:46:01.000
log.segment.delete.delay.ms = 60000

2020-05-21 01:46:01.000
max.connections =<PHONE_NUMBER>

2020-05-21 01:46:01.000
max.connections.per.ip =<PHONE_NUMBER>

2020-05-21 01:46:01.000
max.connections.per.ip.overrides =

2020-05-21 01:46:01.000
max.incremental.fetch.session.cache.slots = 1000

2020-05-21 01:46:01.000
message.max.bytes = 1000012

2020-05-21 01:46:01.000
metric.reporters = []

2020-05-21 01:46:01.000
metrics.num.samples = 2

2020-05-21 01:46:01.000
metrics.recording.level = INFO

2020-05-21 01:46:01.000
metrics.sample.window.ms = 30000

2020-05-21 01:46:01.000
min.insync.replicas = 2

2020-05-21 01:46:01.000
num.io.threads = 8

2020-05-21 01:46:01.000
num.network.threads = 5

2020-05-21 01:46:01.000
num.partitions = 1

2020-05-21 01:46:01.000
num.recovery.threads.per.data.dir = 2

2020-05-21 01:46:01.000
num.replica.alter.log.dirs.threads = null

2020-05-21 01:46:01.000
num.replica.fetchers = 2

2020-05-21 01:46:01.000
offset.metadata.max.bytes = 4096

2020-05-21 01:46:01.000
offsets.commit.required.acks = -1

2020-05-21 01:46:01.000
offsets.commit.timeout.ms = 5000

2020-05-21 01:46:01.000
offsets.load.buffer.size = 5242880

2020-05-21 01:46:01.000
offsets.retention.check.interval.ms = 600000

2020-05-21 01:46:01.000
offsets.retention.minutes = 10080

2020-05-21 01:46:01.000
offsets.topic.compression.codec = 0

2020-05-21 01:46:01.000
offsets.topic.num.partitions = 50

2020-05-21 01:46:01.000
offsets.topic.replication.factor = 3

2020-05-21 01:46:01.000
offsets.topic.segment.bytes = 104857600

2020-05-21 01:46:01.000
password.encoder.cipher.algorithm = AES/CBC/PKCS5Padding

2020-05-21 01:46:01.000
password.encoder.iterations = 4096

2020-05-21 01:46:01.000
password.encoder.key.length = 128

2020-05-21 01:46:01.000
password.encoder.keyfactory.algorithm = null

2020-05-21 01:46:01.000
password.encoder.old.secret = null

2020-05-21 01:46:01.000
password.encoder.secret = null

2020-05-21 01:46:01.000
port = 9092

2020-05-21 01:46:01.000
principal.builder.class = null

2020-05-21 01:46:01.000
producer.purgatory.purge.interval.requests = 1000

2020-05-21 01:46:01.000
queued.max.request.bytes = -1

2020-05-21 01:46:01.000
queued.max.requests = 500

2020-05-21 01:46:01.000
quota.consumer.default =<PHONE_NUMBER>854775807

2020-05-21 01:46:01.000
quota.producer.default =<PHONE_NUMBER>854775807

2020-05-21 01:46:01.000
quota.window.num = 11

2020-05-21 01:46:01.000
quota.window.size.seconds = 1

2020-05-21 01:46:01.000
replica.fetch.backoff.ms = 1000

2020-05-21 01:46:01.000
replica.fetch.max.bytes = 1048576

2020-05-21 01:46:01.000
replica.fetch.min.bytes = 1

2020-05-21 01:46:01.000
replica.fetch.response.max.bytes = 10485760

2020-05-21 01:46:01.000
replica.fetch.wait.max.ms = 500

2020-05-21 01:46:01.000
replica.high.watermark.checkpoint.interval.ms = 5000

2020-05-21 01:46:01.000
replica.lag.time.max.ms = 30000

2020-05-21 01:46:01.000
replica.selector.class = org.apache.kafka.common.replica.RackAwareReplicaSelector

2020-05-21 01:46:01.000
replica.socket.receive.buffer.bytes = 65536

2020-05-21 01:46:01.000
replica.socket.timeout.ms = 30000

2020-05-21 01:46:01.000
replication.quota.window.num = 11

2020-05-21 01:46:01.000
replication.quota.window.size.seconds = 1

2020-05-21 01:46:01.000
request.timeout.ms = 30000

2020-05-21 01:46:01.000
reserved.broker.max.id = 2000

2020-05-21 01:46:01.000
sasl.client.callback.handler.class = null

2020-05-21 01:46:01.000
sasl.enabled.mechanisms = [GSSAPI]

2020-05-21 01:46:01.000
sasl.jaas.config = null

2020-05-21 01:46:01.000
sasl.kerberos.kinit.cmd = /usr/bin/kinit

2020-05-21 01:46:01.000
sasl.kerberos.min.time.before.relogin = 60000

2020-05-21 01:46:01.000
sasl.kerberos.principal.to.local.rules = [DEFAULT]

2020-05-21 01:46:01.000
sasl.kerberos.service.name = null

2020-05-21 01:46:01.000
sasl.kerberos.ticket.renew.jitter = 0.05

2020-05-21 01:46:01.000
sasl.kerberos.ticket.renew.window.factor = 0.8

2020-05-21 01:46:01.000
sasl.login.callback.handler.class = null

2020-05-21 01:46:01.000
sasl.login.class = null

2020-05-21 01:46:01.000
sasl.login.refresh.buffer.seconds = 300

2020-05-21 01:46:01.000
sasl.login.refresh.min.period.seconds = 60

2020-05-21 01:46:01.000
sasl.login.refresh.window.factor = 0.8

2020-05-21 01:46:01.000
sasl.login.refresh.window.jitter = 0.05

2020-05-21 01:46:01.000
sasl.mechanism.inter.broker.protocol = GSSAPI

2020-05-21 01:46:01.000
sasl.server.callback.handler.class = null

2020-05-21 01:46:01.000
security.inter.broker.protocol = PLAINTEXT

2020-05-21 01:46:01.000
security.providers = null

2020-05-21 01:46:01.000
socket.receive.buffer.bytes = 102400

2020-05-21 01:46:01.000
socket.request.max.bytes = 104857600

2020-05-21 01:46:01.000
socket.send.buffer.bytes = 102400

2020-05-21 01:46:01.000
ssl.cipher.suites = []

2020-05-21 01:46:01.000
ssl.client.auth = none

2020-05-21 01:46:01.000
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]

2020-05-21 01:46:01.000
ssl.endpoint.identification.algorithm = https

2020-05-21 01:46:01.000
ssl.key.password = null

2020-05-21 01:46:01.000
ssl.keymanager.algorithm = SunX509

2020-05-21 01:46:01.000
ssl.keystore.password = null

2020-05-21 01:46:01.000
ssl.keystore.type = JKS

2020-05-21 01:46:01.000
ssl.principal.mapping.rules = DEFAULT

2020-05-21 01:46:01.000
ssl.protocol = TLS

2020-05-21 01:46:01.000
ssl.provider = null

2020-05-21 01:46:01.000
ssl.secure.random.implementation = null

2020-05-21 01:46:01.000
ssl.trustmanager.algorithm = PKIX

2020-05-21 01:46:01.000
ssl.truststore.password = null

2020-05-21 01:46:01.000
ssl.truststore.type = JKS

2020-05-21 01:46:01.000
transaction.abort.timed.out.transaction.cleanup.interval.ms = 60000

2020-05-21 01:46:01.000
transaction.max.timeout.ms = 900000

2020-05-21 01:46:01.000
transaction.remove.expired.transaction.cleanup.interval.ms = 3600000

2020-05-21 01:46:01.000
transaction.state.log.load.buffer.size = 5242880

2020-05-21 01:46:01.000
transaction.state.log.min.isr = 2

2020-05-21 01:46:01.000
transaction.state.log.num.partitions = 50

2020-05-21 01:46:01.000
transaction.state.log.replication.factor = 3

2020-05-21 01:46:01.000
transaction.state.log.segment.bytes = 104857600

2020-05-21 01:46:01.000
transactional.id.expiration.ms = 604800000

2020-05-21 01:46:01.000
unclean.leader.election.enable = true

2020-05-21 01:46:01.000
zookeeper.connect = INTERNAL_ZK_DNS,INTERNAL_ZK_DNS,INTERNAL_ZK_DNS,

2020-05-21 01:46:01.000
zookeeper.connection.timeout.ms = null

2020-05-21 01:46:01.000
zookeeper.max.in.flight.requests = 10

2020-05-21 01:46:01.000
zookeeper.session.timeout.ms = 18000

2020-05-21 01:46:01.000
zookeeper.set.acl = false

2020-05-21 01:46:01.000
zookeeper.sync.time.ms = 2000

2020-05-21 01:46:01.000
(kafka.server.KafkaConfig)

@marc-ostrow Did u set the rack id in the consumer config ?

Yes, my rack, the availability zone ID, of my instance was set in the config.  I also had logs where the rack if was sent in the request!  So either we’re not sending the request correctly, some other issues are present.  Not sure if it’s client or server that’s having issues.  But the -1 value obviously is caught as an invalid value and we stay on the leader.

@dneralla Yes, I did.  Here's some initial log output showing the brokers, their rack id's, the Leader broker's rack ID, and the rack id in the encode and the preferred replica in the decode of fetch request/response.
[ec2-user@ip-10-232-166-105 ~]$ ./kip392verify -brokers $BROKERS -topic CCR-Signal-Quality -verbose true
[kip392_verify] 2020/06/15 14:35:12 My Rack: use1-az1 (us-east-1a)
[kip392_verify] 2020/06/15 14:35:12 MSK Broker Rack: use1-az2 (us-east-1b), Address: b-1.system-events-dev-blue.uwjleq.c5.kafka.us-east-1.amazonaws.com
[kip392_verify] 2020/06/15 14:35:12 MSK Broker Rack: use1-az4 (us-east-1c), Address: b-2.system-events-dev-blue.uwjleq.c5.kafka.us-east-1.amazonaws.com
[kip392_verify] 2020/06/15 14:35:12 MSK Broker Rack: use1-az1 (us-east-1a), Address: b-3.system-events-dev-blue.uwjleq.c5.kafka.us-east-1.amazonaws.com
[sarama] 2020/06/15 14:35:12 Initializing new client
[sarama] 2020/06/15 14:35:12 client/metadata fetching metadata for all topics from broker b-2.system-events-dev-blue.uwjleq.c5.kafka.us-east-1.amazonaws.com:9092
[sarama] 2020/06/15 14:35:12 Connected to broker at b-2.system-events-dev-blue.uwjleq.c5.kafka.us-east-1.amazonaws.com:9092 (unregistered)
[sarama] 2020/06/15 14:35:12 client/brokers registered new broker #2 at b-2.system-events-dev-blue.uwjleq.c5.kafka.us-east-1.amazonaws.com:9092
[sarama] 2020/06/15 14:35:12 client/brokers registered new broker #3 at b-3.system-events-dev-blue.uwjleq.c5.kafka.us-east-1.amazonaws.com:9092
[sarama] 2020/06/15 14:35:12 client/brokers registered new broker #1 at b-1.system-events-dev-blue.uwjleq.c5.kafka.us-east-1.amazonaws.com:9092
[sarama] 2020/06/15 14:35:12 Successfully initialized new client
[kip392_verify] 2020/06/15 14:35:12 Sarama Broker: ID: 2, Rack: use1-az4 Address: b-2.system-events-dev-blue.uwjleq.c5.kafka.us-east-1.amazonaws.com:9092
[kip392_verify] 2020/06/15 14:35:12 Sarama Broker: ID: 3, Rack: use1-az1 Address: b-3.system-events-dev-blue.uwjleq.c5.kafka.us-east-1.amazonaws.com:9092
[kip392_verify] 2020/06/15 14:35:12 Sarama Broker: ID: 1, Rack: use1-az2 Address: b-1.system-events-dev-blue.uwjleq.c5.kafka.us-east-1.amazonaws.com:9092
[kip392_verify] 2020/06/15 14:35:12 Leader Broker for topic "CCR-Signal-Quality": ID: 1, Rack: use1-az2, Address: b-1.system-events-dev-blue.uwjleq.c5.kafka.us-east-1.amazonaws.com:9092
[sarama] 2020/06/15 14:35:12 Connected to broker at b-1.system-events-dev-blue.uwjleq.c5.kafka.us-east-1.amazonaws.com:9092 (registered as #1)
[sarama] 2020/06/15 14:35:12 consumer/broker/1 added subscription to CCR-Signal-Quality/0
[sarama] 2020/06/15 14:35:12 encode version 11,  RackID use1-az1
[sarama] 2020/06/15 14:35:12 encode version 11,  RackID use1-az1
[sarama] 2020/06/15 14:35:12 decoded version 11, PreferredReadReplica -1

@dneralla Hi, so I also tried running a docker network cluster of three ZK, and three Kafka nodes using confluentinc/cp-kafka:5.4.1 which I believe incorporates Kafka 2.4.1.
I actually modified the fetch handler to log the client metadata so I could verify we were indeed sending the rackId; and we are.  I cloned the confluentinc/kafka repo and checked out the 5.4.1 tag, made my mods, and built an unsigned tarbar.  I extracted kafka_2.12-5.4.1-ccs/libs/kafka_2.12-5.4.1-ccs.jar from the kafka_2.12-5.4.1-ccs.tgz tarball. I then built a new image from the confluentinc/cp-kafka:5.4.1 base image rewriting the VOLUME and CMD dockerfile commands to match the cp-docker-images dockerfile.
FROM confluentinc/cp-kafka:5.4.1
COPY kafka_2.12-5.4.1-ccs.jar /usr/share/java/kafka/
VOLUME ["/var/lib/kafka/data", "/etc/kafka/secrets"]
CMD ["/etc/confluent/docker/run"]

My diffs were as follows
diff --git a/core/src/main/scala/kafka/server/KafkaApis.scala b/core/src/main/scala/kafka/server/KafkaApis.scala
index 124b2e356..6018303a3 100644
--- a/core/src/main/scala/kafka/server/KafkaApis.scala
+++ b/core/src/main/scala/kafka/server/KafkaApis.scala
@@ -609,6 +609,10 @@ class KafkaApis(val requestChannel: RequestChannel,
       None
     }

+    if (versionId >= 11) {
+      info("handleFetchRequest: clientMetadata is " + clientMetadata.getOrElse("not set"))
+    }
+
     def errorResponse[T >: MemoryRecords <: BaseRecords](error: Errors): FetchResponse.PartitionData[T] = {
       new FetchResponse.PartitionData[T](error, FetchResponse.INVALID_HIGHWATERMARK, FetchResponse.INVALID_LAST_STABLE_OFFSET,
         FetchResponse.INVALID_LOG_START_OFFSET, null, MemoryRecords.EMPTY)

Btw, I highly suggest we modify the pull request and change any protocol version checks against version 11 api, to be true for any version greater than or equal to 11.

Btw, I further modified my instrumentation as follows to reduce log output:
diff --git a/fetch_request.go b/fetch_request.go
index cffc339..9802ae2 100644
--- a/fetch_request.go
+++ b/fetch_request.go
@@ -65,6 +65,8 @@ const (
        ReadCommitted
 )

+var encodeCount = uint64(0)
+
 func (r *FetchRequest) encode(pe packetEncoder) (err error) {
        pe.putInt32(-1) // replica ID is always -1 for clients
        pe.putInt32(r.MaxWaitTime)
@@ -124,7 +126,10 @@ func (r *FetchRequest) encode(pe packetEncoder) (err error) {
                if err != nil {
                        return err
                }
-               Logger.Printf("encode version %d,  RackID %s\n", r.Version, r.RackID)
+               if encodeCount%100 == 0 {
+                       Logger.Printf("encode version %d,  RackID %s\n", r.Version, r.RackID)
+               }
+               encodeCount++
        }

        return nil
diff --git a/fetch_response.go b/fetch_response.go
index a60e948..291ed9f 100644
--- a/fetch_response.go
+++ b/fetch_response.go
@@ -41,6 +41,8 @@ type FetchResponseBlock struct {
        Partial              bool
 }

+var decodeCount = uint64(0)
+
 func (b *FetchResponseBlock) decode(pd packetDecoder, version int16) (err error) {
        tmp, err := pd.getInt16()
        if err != nil {
@@ -89,7 +91,10 @@ func (b *FetchResponseBlock) decode(pd packetDecoder, version int16) (err error)
                if err != nil {
                        return err
                }
-               Logger.Printf("decoded version %d, PreferredReadReplica %d\n", version, b.PreferredReadReplica)
+               if decodeCount%100 == 0 {
+                       Logger.Printf("decoded version %d, PreferredReadReplica %d\n", version, b.PreferredReadReplica)
+               }
+               decodeCount++
        }

        recordsSize, err := pd.getInt32()
@@ -230,7 +235,6 @@ type FetchResponse struct {
        Version       int16
        LogAppendTime bool
        Timestamp     time.Time
-       Broker        *Broker
 }

 func (r *FetchResponse) decode(pd packetDecoder, version int16) (err error) {

I experienced the following:

With no message traffic, I did see PreferredReadReplica set to the broker id of the broker matching my client's RackID.
With consistent volume traffic, I only saw PreferredReadReplica set to -1; But I also wasn't seeing my messages, like I was in my MSK setup.  I forgot to check the under replicated partitions (I'm afraid I was getting tired. ;-( )
When testing in MSK, I saw messages, and received PreferredReadReplica set to -1, but I did check UnderReplicatedPartitions in Cloudwatch which was 0 for each broker.

Sorry... will have to try again next weekend if I have time.

Our major concern with cross-az traffic is with bringing in upwards to 7000 mpeg transport streams.  Not really an idiomatic use of Kafka, but it works using single partition topics, one for each stream.   If we had this feature, assuming the cluster was performing appropriately, we wouldn't have to worry so much about the cost of cross-az traffic with our consumers of these streams.  However, we would still have to watch the leader brokers per partition/topic, and move the producers into the same AZ to eliminate cross-az traffic.
We've basically opted to go the route of gRPC for these streams which means all consumers will have to operate within the same AZ as their gRPC server to reduce cross-az traffic.
BUT, has anyone gotten this pull request to work, and verify the bulk of traffic is between a client and it's nearest replica, that being in its rack?

Anyone get this working?

Hello,
Is there any ETA for this to be merged?
Is anything that i could do to help with this?
Angelos

@angeloskaltsikis i believe it’s still waiting for someone to test the changes and confirm they function as expected by analysing the traffic. The earlier comments seemed to suggest it wasn’t working as it should do we wouldn’t want to merge and advertise the functionality until it was

Hi there,
I spent some time validating this and, hopefully, improving it. Hope you don't mind me taking it a bit further, @dneralla!
My branch is here. These are the main changes from this PR:

Addressed feedback
Tightened up diff a bit (removed unrelated whitespace changes, etc)
Changed to only using a preferredReadReplica field, -1 indicates no preference
Moved up setting preferredReadReplica in parseResponse, which sets if if there are no records in the response
Set FetchResponse.PreferredReadReplica to -1 when decoding versions prior to 11
Add more tests to cover error cases and capture the behavior I observed that -1 is returned when no change should me made
Change consumeMsgs in functional tests to use subtests to reduce
number of outstanding connections

I also set up a test rig at https://github.com/danp/sarama-rack. It let me send some messages to a cluster in docker-compose and see things working. Maybe that will help others validate this as well.
Feel free to integrate my changes however you like, or let me know if there's anything else I can do.

Any ETA on this merge? One thing to note -  without this change, if you make the mistake of setting the RackId on the consumer client config with a rack aware kafka cluster, you may or may not consume messages. Luck of the draw

Hey, This is no longer my priority and I am not finding time to get to
this...can someone take this PR over ?
On Wed, Oct 28, 2020 at 7:27 AM salsorrentino<EMAIL_ADDRESS>wrote:

Any ETA on this merge? One thing to note - without this change, if you
make the mistake of setting the RackId on the consumer client config with a
rack aware kafka cluster, you may or may not consume messages. Luck of the
draw
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Shopify/sarama/pull/1696#issuecomment-717969708, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAQW5OPT3KDYAHH6L6ZQTR3SNAS4DANCNFSM4M5Q4LWQ
.

Hi there,
I spent some time validating this and, hopefully, improving it. Hope you don't mind me taking it a bit further, @dneralla!
My branch is here. These are the main changes from this PR:

Addressed feedback
Tightened up diff a bit (removed unrelated whitespace changes, etc)
Changed to only using a preferredReadReplica field, -1 indicates no preference
Moved up setting preferredReadReplica in parseResponse, which sets if if there are no records in the response
Set FetchResponse.PreferredReadReplica to -1 when decoding versions prior to 11
Add more tests to cover error cases and capture the behavior I observed that -1 is returned when no change should be made
Change consumeMsgs in functional tests to use subtests to reduce number of outstanding connections

I also set up a test rig at https://github.com/danp/sarama-rack. It let me send some messages to a cluster in docker-compose and see things working. Maybe that will help others validate this as well.
Feel free to integrate my changes however you like, or let me know if there's anything else I can do.

After some very basic testing with (sarama-rack)[https://github.com/danp/sarama-rack] some weeks ago, it seemed that the fork of @danp worked as expected. Can someone else try the changes? It's very easy to setup with docker.
I can try to test this feature more extensively as well.
Any certain tests you suggest that we run @dnwe ?
The thing i believe that most people will agree with, is that as soon as this feature is tested and released in sarama it is going to save a LOT of Inter-AZ/Cross-AZ Traffic Costs to a lot of applications using sarama as the base library.

@danp could you please submit a PR with changes you've implemented on top of this one?

Sure, opened https://github.com/Shopify/sarama/pull/1822.

Superceded by #1822


Log conversation data

Btw, the error in the dialogue agent test is because the test scaffold in test/agent doesn't mock engine.db, so you'll have to add that.

Updated:

Use uuid
Insert data incrementally in turnFinished()
Add migrate code

Oh and fix lint please!

Updated:

Add conversation id, previous turn id, dialogue id, and timestamp
Fix lint/tab problems

Add mock db and local table for testing

Added vote and comment fields in the database.


Clarify int main(void) example

The semicolon can elicit a syntax error from C compilers such as MSVC.
Specifying a return is useful: in ISO C ("C99") and later, the return 0; is implied.
However, in ANSI C (AKA "C89") the compiler is not required to add return 0;.
This document references ANSI C, so clarity on the matter and consistent style is best.
Also, it is best to identify the struct with the bit layout in question.

@workingjubilee : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#label:"aq-pr-triaged"

@corob-msft : Thanks for your contribution! The author(s) have been notified to review your proposed change.


Add a new CSS Animation

Describe the project you want to add with tech stack

Adding a page loading animation using HTML and CSS
Expected behavior

Various moving objects including delivery box, air-plane and border
Screenshots (optional)

Additional context (optional )

I want to do this issue.

please assign me this issue

Hey! Can you Assign this to me under Hacktoberfest?
Describe : I'll be adding a Sliding Text Animation.
I'll be adding a page-loading animation using HTML and CSS
Expected behaviour : Text will be sliding one after another.
Screenshot
It'll look something like this

Please! Assign this to me

I would love to contribute to this open source project!


Configure Azure Pipelines

This pipeline configuration builds the application and produces a build for the Release configuration.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: LorenaMuresan sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.


Next Permutation.cpp

The simple code is for next permutation of any arrangement of integers of characters. Leetcode 31.

Is this merged to hackoctober fest?
On Tue, 18 Oct 2022, 00:41 savarapu manohar joshi, @.***>
wrote:

Merged #14
https://github.com/Manohar-mj/Complete-Interview-Preparation---Self-Paced/pull/14
into main.
—
Reply to this email directly, view it on GitHub
https://github.com/Manohar-mj/Complete-Interview-Preparation---Self-Paced/pull/14#event-7605108556,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AT5J2Q7LP3MNKG42PD2ISVDWDWQGPANCNFSM6AAAAAARHGDURY
.
You are receiving this because you authored the thread.Message ID:
<Manohar-mj/Complete-Interview-Preparation---Self-Paced/pull/14/issue_event/7605108556
@github.com>


[BUG] export/download causes internal server error with FastCGI

Export/download causes internal server error when PHP is running with FastCGI. The problem comes from the exit called at the end of the download method in src/Maatwebsite/Excel/Writers/LaravelExcelWriters.php. Calling fastcgi_finish_request before exit seems to fix this bug:
protected function _download(Array $headers = [])
{
    // Set the headers
    $this->_setHeaders(
        $headers,
        [
            'Content-Type'        => $this->contentType,
            'Content-Disposition' => 'attachment; filename="' . $this->filename . '.' . $this->ext . '"',
            'Expires'             => 'Mon, 26 Jul 1997 05:00:00 GMT', // Date in the past
            'Last-Modified'       => Carbon::now()->format('D, d M Y H:i:s'),
            'Cache-Control'       => 'cache, must-revalidate',
            'Pragma'              => 'public'
        ]
    );

    // Check if writer isset
    if (!$this->writer)
        throw new LaravelExcelException('[ERROR] No writer was set.');

    // Download
    $this->writer->save('php://output');

    // ======== FIX =========
    if (function_exists('fastcgi_finish_request')) {
        fastcgi_finish_request();
    }
    // =======================

    // End the script to prevent corrupted xlsx files
    exit;
}

Package version : 2.1.23
Laravel version : 5.4.36

We are closing all old tickets, because version 2.* is reaching the End-of-Life stage. Our full attention is now allocated to version 3.0. It's still possible to use 2.*, but there will be no active support anymore.


Problem when using on Firefox 68.10.1 for Android

Describe the bug
After completing the captcha nothing seems to happen and a verification-expired error appears.
To Reproduce
Steps to reproduce the behavior:

Fill in the form
Click to Submit
Check the "I am not a robot" verification challenge
See error

Expected behavior
Successful verification and form submission.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: Android
Browser Firefox
Version 68.10.1

Additional context
Add any other context about the problem here.

@letria my sincere apologies. I missed this issue.
I would like to know if Airform works fine for you now?


Update WinGet Installer and Uninstaller Functions

Changed how WinGet installs and uninstalls are handled, by utilizing Install-WinGetPackage and Uninstall-WinGetPackage.

I learned today that PS Modules utilizing WinGet are installed alongside WinGet as of a few years ago. With the changes to how Winget Updating is handled by my other PR #1829, this should be a better solution to handling installs going forward. In the cases I tested, checking Status was the best way to handle issues with a package installer not having machine scope capabilities.
If anyone has any better ways to handle this, let me know.
This should fix issues, like: #1805 #1756

In my opinion it is a dangerous assumption to make that all Win10 and Win11 installs with winget also have the Cmdlets installed.
I've just spun up a new Win11 VM and winget was installed, but the Cmdlets were not. In this case, the winutil would now be useless until the user fixes the missing dependencies manually. All in all I'm not against using the Cmdlets but I think a bunch of legwork has to be done to make sure fallbacks are in place for when the Cmdlets are missing. (e.g. auto install the cmdlets or fallback to using the executable). I've written a commend I a previous PR already about the problems using Cmdlets https://github.com/ChrisTitusTech/winutil/pull/1682#discussion_r1527164290

Ah, yes. I did mean this as an addition to my other fix, which should always fetch the newest version of WinGet, but even then your change is a great addition. Thank you!

I just spun up a Win11 VM again to test if the Cmdlets will be installed automatically together with the newest winget release (v1.7.10861) downloaded and installed directly from the winget-cli gh repo but it still seems that the Cmdlets won't be installed automatically so if you want to utilize this functions, maybe the following snipped I wrote might be a good starting point to extend your PR to auto-update winget and install the cmdlet as well.
I have no time to write and test a full PR including error handling etc at the moment and contextually it fits into your PR so maybe you want to give it a go :)
$response = Invoke-RestMethod -Uri "https://api.github.com/repos/microsoft/Winget-cli/releases/latest" -Method Get -ErrorAction Stop
$installer = $response.assets | Where-Object {$_.name -eq "Microsoft.DesktopAppInstaller_8wekyb3d8bbwe.msixbundle"} | Select-Object Name, browser_download_url, @{Name = "path"; expr = {"$ENV:TEMP\"+$_.name}}

Write-Host "Downloading", $installer.name, "..."
Invoke-WebRequest -Uri $installer.browser_download_url -OutFile $installer.path

Write-Host "Installing", $installer.name, "..."
Add-AppxPackage -Path $installer.path -ForceUpdateFromAnyVersion

Install-PackageProvider -Name NuGet -Force
Install-Module -Name Microsoft.WinGet.Client -Force

I just spun up a Win11 VM again to test if the Cmdlets will be installed automatically together with the newest winget release (v1.7.10861) downloaded and installed directly from the winget-cli gh repo but it still seems that the Cmdlets won't be installed automatically so if you want to utilize this functions, maybe the following snipped I wrote might be a good starting point to extend your PR to auto-update winget and install the cmdlet as well.
I have no time to write and test a full PR including error handling etc at the moment and contextually it fits into your PR so maybe you want to give it a go :)
$response = Invoke-RestMethod -Uri "https://api.github.com/repos/microsoft/Winget-cli/releases/latest" -Method Get -ErrorAction Stop
$installer = $response.assets | Where-Object {$_.name -eq "Microsoft.DesktopAppInstaller_8wekyb3d8bbwe.msixbundle"} | Select-Object Name, browser_download_url, @{Name = "path"; expr = {"$ENV:TEMP\"+$_.name}}

Write-Host "Downloading", $installer.name, "..."
Invoke-WebRequest -Uri $installer.browser_download_url -OutFile $installer.path

Write-Host "Installing", $installer.name, "..."
Add-AppxPackage -Path $installer.path -ForceUpdateFromAnyVersion

Install-PackageProvider -Name NuGet -Force
Install-Module -Name Microsoft.WinGet.Client -Force

Thank you! I'll take a look when I get off work!

Do we want to merge this and add the improvement to the installer in another PR or add it to this one? @ruxunderscore

@ChrisTitusTech Let's hold this back and I can work on these tonight.

This should be the final update. @Marterich I implemented your suggestion during the winget-cli install. I also changed some commands to work better after a bit of testing on clean Windows VMs.

@ChrisTitusTech This should be good to merge into the test branch now. I got the code tidied up and better tested.

@ChrisTitusTech This should be good to merge into the test branch now. I got the code tidied up and better tested.

Awesome thanks Rux!


Add clintonb to django-commons

Fix #118

Is this plan valid?

Is this plan valid?

This is strange, not sure exactly what caused this.

What is even stranger is that running locally the plan shows different results. I think I have a different terraform version

Ok, looks better now!


How to perform identical transformation on images and targets in dense prediction tasks?

In dense prediction (or Pix2Pix) tasks such as segmentation, depth estimation etc, we often need to perform the same transformation on images and targets. e.g. Either both the image and the corresponding target should be horizontally flipped, or neither of them.
How do I achieve that with FFCV?

We already had a similar question @andrewilyas . Do you have a link to the original issue ?

Hello @shariqfarooq123 @Callidior we are currently planning improvements of FFCV to make this kind of task. One suggested potential API from @andrewilyas would be the addition of a transform called SynchronizeSeed(seed_offset=0) that you can insert in both pipelines and that guarantees that forces the seed to a value that is identical (assuming seed_offset is the same).
Is it something that would be useful and solve your use case ? Do you have a suggestion for a neater API that would solve what you need ?

Thanks for the response!
Yes, something like that could be useful and would solve this.
For now, I ended up setting the seed which is dependent on the indices inside the operation, as done in Mixup.
For maximum flexibility, it would be good to have the ability to implement custom "joint" operations / joint pipelines. Such operations would then take the tuple of all the fields e.g. JointOp(imA_at_index_i, imB_at_index_i).

@shariqfarooq123 Could provide an example of how this would look like from the point of view of a user. I'm not really clear about that.

@GuillaumeLeclerc The proposed SynchronizeSeed transform would be a straightforward and easy to implement solution for the problem. However, I see a few potential issues.
In some cases, it would be necessary to insert this transform multiple times into a pipeline, before each transformation to be synchronized. This would be the case when one pipeline contains more transforms than the other, as in the following example:
pipelines = {
    'image': [
        SynchronizeSeed(),
        RandomResizedCropRGBImageDecoder((224, 224)),
        tf.ColorJitter(0.2, 0.3),
        SynchronizeSeed(),
        RandomHorizontalFlip(),
        ToTensor(),
        ToTorchImage(),
        ToDevice(0)
    ],
    'mask': [
        SynchronizeSeed(),
        RandomResizedCropRGBImageDecoder((224, 224)),
        SynchronizeSeed(),
        RandomHorizontalFlip(),
        ToTensor(),
        ToDevice(0)
    ]
}

Here, the synchronization needs to be inserted multiple times, since the image pipeline contains an additional color jitter transform. In this case, this could easily be solved by reordering the transforms, but that doesn't always need to be the case.
Second, I am concerned that repeatedly fixing seeds might have a negative impact on the randomness of the remaining transformations in the pipeline.
I would also be in favor of "joint pipelines" as mentioned by shariqfarooq123 but it would probably require larger restructuring of the library to make joint pipelines a possibility. As a trade-off, I could imagine "stateful transforms", which can share an internal state between pipelines.
Let me illustrate this with an example. In the following, two separate instances of RandomHorizontalFlip are used and they are thus independent:
pipelines = {
    'image': [..., RandomHorizontalFlip(), ...],
    'mask': [..., RandomHorizontalFlip(), ...],
}

My proposal would be to make the behavior different if the same instance of the transform is used in both pipelines:
flip = RandomHorizontalFlip(stateful=True)
pipelines = {
    'image': [..., flip, ...],
    'mask': [..., flip, ...],
}

In this case, the flip in the two pipelines would be synchronized, as the same instance is used. This instance could maintain an internal state to remember the randomly sampled parameters. The state could be initialized during the first call to the transform from one of the pipelines and then re-used in the other ones. It would, of course, need to be reset after the batch or the epoch (if states depend on sample indices).
I have no idea how easy this would be to implement in FFCV, but it still seems easier than joint pipelines while being more user-friendly and less error-prone than having a SynchronizedSeed transform.

Thank you @Callidior for the careful thoughts you put in this issue.
About the SynchonizeSeed there would be an argument so that you can have it multiple times in the pipeline as long as each group uses a different one. Based on my understanding of PRNGs, I don't see how using one synchronization would be worse than multiple.
As @juliustao pointed, there is a broader problem here. When using multiple workers, there is no guarantee that they will process samples in order and use random samples for different elements in the batch across different supposedly synchronized transforms. This could be solved by https://github.com/numba/numba/issues/2649.
I like the idea of joint transformation and I could see multiple potential APIs. One is the one I described with a state and another would be stateless where we pass the two input simultaneously to the transformation. In both cases there are two potentially major problems:

all existing and future have to be modified and are now more complicated to write this raise the barrier for newcomers to both using and contributing to FFCV.
Memory allocation becomes trickier. Before each instance of a transform had it's memory now it more complicated as a variable amount of pipelines use a transform. Also how to handle someone who might want to apply the same flip on potentially different resolution images, now the memory allocated isn't even uniform accross users of a same transformation.

Thoughts ? (Also @andrewilyas @lengstrom )

It seems like joint transformations are the only way to ergonomically + efficiently allow for shared state (even if the seed mechanism worked perfectly it feels like an opaque/strange way to share state)

@lengstrom Any suggestion on how to solve the issues related to Joint transforms I raised?

Would it be too difficult to allocate memory per pipeline using the transform? For each pipeline, declare_state_and_memory would be called and dedicated memory be allocated. The destination of the transform is already passed to the function created by generate_code, so this would just be a different destination for each pipeline. Only the memory needed for storing the state would need to be shared across pipelines and could be allocated separately.
Or do I get the memory concept totally wrong?

No you are right @Callidior. I guess the only problem is how to make it compatible with all previous and future augmentations

@Callidior I'm currently refactoring the codegen of FFCV to allow for joined operations and other features. I hope it's going to be done soon. I still think we also will add SyncSeed to enable simple use cases where a user just want to synchronize two operations without having to write a joined version of it

@GuillaumeLeclerc Great! I think the possibility of joint transformations that can share state across pipelines will be an important feature for FFCV. Regarding SyncSeed, the concerns with respect to multiple parallel data loading processes still remain, right?

This (and more) made possible by 307ffe0cf3702f5c991b9ed5ca12ac314599edb9. I hope we can make a release with it soon :tada:

@GuillaumeLeclerc Is there an example on how to use this new feature? I am also looking to perform identical transforms on images and targets. Thanks!