[SPARK-16213][SQL] Reduce runtime overhead of a program that creates an primitive array in DataFrame

What changes were proposed in this pull request?
This PR reduces runtime overhead of a program the creates an primitive array in DataFrame by using the similar approach to #15044. Generated code performs boxing operation in an assignment from InternalRow to an Object[] temporary array (at Lines 051 and 061 in the generated code before without this PR). If we know that type of array elements is primitive, we apply the following optimizations:

Eliminate a pair of isNullAt() and a null assignment
Allocate an primitive array instead of Object[] (eliminate boxing operations)
Create UnsafeArrayData by using UnsafeArrayWriter to keep a primitive array in a row format instead of doing non-lightweight operations in constructor of GenericArrayData
The PR also performs the same things for CreateMap.

Here are performance results of DataFrame programs by up to 18.8x over without this PR.
Without SPARK-16043
OpenJDK 64-Bit Server VM 1.8.0_91-b14 on Linux 4.4.11-200.fc22.x86_64
Intel Xeon E3-12xx v2 (Ivy Bridge)
Read a primitive array in DataFrame:     Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
Int                                           3805 / 4150          0.0      507308.9       1.0X
Double                                        3593 / 3852          0.0      479056.9       1.1X

With SPARK-16043
Read a primitive array in DataFrame:     Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
Int                                            202 /  3050          0.0       26998.2       1.0X
Double                                         198 /  204          0.0       26449.4       1.0X

Note : #15780 is enabled for these measurements
An motivating example
val df = sparkContext.parallelize(Seq(0.0d, 1.0d), 1).toDF
df.selectExpr("Array(value + 1.1d, value + 2.2d)").show

Generated code without this PR
/* 005 */ final class GeneratedIterator extends org.apache.spark.sql.execution.BufferedRowIterator {
/* 006 */   private Object[] references;
/* 007 */   private scala.collection.Iterator[] inputs;
/* 008 */   private scala.collection.Iterator inputadapter_input;
/* 009 */   private UnsafeRow serializefromobject_result;
/* 010 */   private org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder serializefromobject_holder;
/* 011 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter serializefromobject_rowWriter;
/* 012 */   private Object[] project_values;
/* 013 */   private UnsafeRow project_result;
/* 014 */   private org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder project_holder;
/* 015 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter project_rowWriter;
/* 016 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter project_arrayWriter;
/* 017 */
/* 018 */   public GeneratedIterator(Object[] references) {
/* 019 */     this.references = references;
/* 020 */   }
/* 021 */
/* 022 */   public void init(int index, scala.collection.Iterator[] inputs) {
/* 023 */     partitionIndex = index;
/* 024 */     this.inputs = inputs;
/* 025 */     inputadapter_input = inputs[0];
/* 026 */     serializefromobject_result = new UnsafeRow(1);
/* 027 */     this.serializefromobject_holder = new org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(serializefromobject_result, 0);
/* 028 */     this.serializefromobject_rowWriter = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(serializefromobject_holder, 1);
/* 029 */     this.project_values = null;
/* 030 */     project_result = new UnsafeRow(1);
/* 031 */     this.project_holder = new org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(project_result, 32);
/* 032 */     this.project_rowWriter = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(project_holder, 1);
/* 033 */     this.project_arrayWriter = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter();
/* 034 */
/* 035 */   }
/* 036 */
/* 037 */   protected void processNext() throws java.io.IOException {
/* 038 */     while (inputadapter_input.hasNext()) {
/* 039 */       InternalRow inputadapter_row = (InternalRow) inputadapter_input.next();
/* 040 */       double inputadapter_value = inputadapter_row.getDouble(0);
/* 041 */
/* 042 */       final boolean project_isNull = false;
/* 043 */       this.project_values = new Object[2];
/* 044 */       boolean project_isNull1 = false;
/* 045 */
/* 046 */       double project_value1 = -1.0;
/* 047 */       project_value1 = inputadapter_value + 1.1D;
/* 048 */       if (false) {
/* 049 */         project_values[0] = null;
/* 050 */       } else {
/* 051 */         project_values[0] = project_value1;
/* 052 */       }
/* 053 */
/* 054 */       boolean project_isNull4 = false;
/* 055 */
/* 056 */       double project_value4 = -1.0;
/* 057 */       project_value4 = inputadapter_value + 2.2D;
/* 058 */       if (false) {
/* 059 */         project_values[1] = null;
/* 060 */       } else {
/* 061 */         project_values[1] = project_value4;
/* 062 */       }
/* 063 */
/* 064 */       final ArrayData project_value = new org.apache.spark.sql.catalyst.util.GenericArrayData(project_values);
/* 065 */       this.project_values = null;
/* 066 */       project_holder.reset();
/* 067 */
/* 068 */       project_rowWriter.zeroOutNullBytes();
/* 069 */
/* 070 */       if (project_isNull) {
/* 071 */         project_rowWriter.setNullAt(0);
/* 072 */       } else {
/* 073 */         // Remember the current cursor so that we can calculate how many bytes are
/* 074 */         // written later.
/* 075 */         final int project_tmpCursor = project_holder.cursor;
/* 076 */
/* 077 */         if (project_value instanceof UnsafeArrayData) {
/* 078 */           final int project_sizeInBytes = ((UnsafeArrayData) project_value).getSizeInBytes();
/* 079 */           // grow the global buffer before writing data.
/* 080 */           project_holder.grow(project_sizeInBytes);
/* 081 */           ((UnsafeArrayData) project_value).writeToMemory(project_holder.buffer, project_holder.cursor);
/* 082 */           project_holder.cursor += project_sizeInBytes;
/* 083 */
/* 084 */         } else {
/* 085 */           final int project_numElements = project_value.numElements();
/* 086 */           project_arrayWriter.initialize(project_holder, project_numElements, 8);
/* 087 */
/* 088 */           for (int project_index = 0; project_index < project_numElements; project_index++) {
/* 089 */             if (project_value.isNullAt(project_index)) {
/* 090 */               project_arrayWriter.setNullDouble(project_index);
/* 091 */             } else {
/* 092 */               final double project_element = project_value.getDouble(project_index);
/* 093 */               project_arrayWriter.write(project_index, project_element);
/* 094 */             }
/* 095 */           }
/* 096 */         }
/* 097 */
/* 098 */         project_rowWriter.setOffsetAndSize(0, project_tmpCursor, project_holder.cursor - project_tmpCursor);
/* 099 */       }
/* 100 */       project_result.setTotalSize(project_holder.totalSize());
/* 101 */       append(project_result);
/* 102 */       if (shouldStop()) return;
/* 103 */     }
/* 104 */   }
/* 105 */ }

Generated code with this PR
/* 005 */ final class GeneratedIterator extends org.apache.spark.sql.execution.BufferedRowIterator {
/* 006 */   private Object[] references;
/* 007 */   private scala.collection.Iterator[] inputs;
/* 008 */   private scala.collection.Iterator inputadapter_input;
/* 009 */   private UnsafeRow serializefromobject_result;
/* 010 */   private org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder serializefromobject_holder;
/* 011 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter serializefromobject_rowWriter;
/* 012 */   private org.apache.spark.sql.catalyst.expressions.UnsafeArrayData project_value;
/* 013 */   private org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder project_holder;
/* 014 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter project_createArrayWriter;
/* 015 */   private UnsafeRow project_result;
/* 016 */   private org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder project_holder1;
/* 017 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter project_rowWriter;
/* 018 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter project_arrayWriter;
/* 019 */
/* 020 */   public GeneratedIterator(Object[] references) {
/* 021 */     this.references = references;
/* 022 */   }
/* 023 */
/* 024 */   public void init(int index, scala.collection.Iterator[] inputs) {
/* 025 */     partitionIndex = index;
/* 026 */     this.inputs = inputs;
/* 027 */     inputadapter_input = inputs[0];
/* 028 */     serializefromobject_result = new UnsafeRow(1);
/* 029 */     this.serializefromobject_holder = new org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(serializefromobject_result, 0);
/* 030 */     this.serializefromobject_rowWriter = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(serializefromobject_holder, 1);
/* 031 */     project_value = new org.apache.spark.sql.catalyst.expressions.UnsafeArrayData();
/* 032 */     project_holder = new org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(project_value, 2);
/* 033 */     project_createArrayWriter = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter();
/* 034 */     project_result = new UnsafeRow(1);
/* 035 */     this.project_holder1 = new org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(project_result, 32);
/* 036 */     this.project_rowWriter = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(project_holder1, 1);
/* 037 */     this.project_arrayWriter = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter();
/* 038 */
/* 039 */   }
/* 040 */
/* 041 */   protected void processNext() throws java.io.IOException {
/* 042 */     while (inputadapter_input.hasNext()) {
/* 043 */       InternalRow inputadapter_row = (InternalRow) inputadapter_input.next();
/* 044 */       double inputadapter_value = inputadapter_row.getDouble(0);
/* 045 */
/* 046 */       project_holder.reset();
/* 047 */       project_createArrayWriter.initialize(project_holder, 2, 8);
/* 048 */
/* 049 */       boolean project_isNull1 = false;
/* 050 */
/* 051 */       double project_value1 = -1.0;
/* 052 */       project_value1 = inputadapter_value + 1.1D;
/* 053 */       project_createArrayWriter.write(0, project_value1);
/* 054 */
/* 055 */       boolean project_isNull4 = false;
/* 056 */
/* 057 */       double project_value4 = -1.0;
/* 058 */       project_value4 = inputadapter_value + 2.2D;
/* 059 */       project_createArrayWriter.write(1, project_value4);
/* 060 */       project_holder1.reset();
/* 061 */
/* 062 */       // Remember the current cursor so that we can calculate how many bytes are
/* 063 */       // written later.
/* 064 */       final int project_tmpCursor = project_holder1.cursor;
/* 065 */
/* 066 */       if (project_value instanceof UnsafeArrayData) {
/* 067 */         final int project_sizeInBytes = ((UnsafeArrayData) project_value).getSizeInBytes();
/* 068 */         // grow the global buffer before writing data.
/* 069 */         project_holder1.grow(project_sizeInBytes);
/* 070 */         ((UnsafeArrayData) project_value).writeToMemory(project_holder1.buffer, project_holder1.cursor);
/* 071 */         project_holder1.cursor += project_sizeInBytes;
/* 072 */
/* 073 */       } else {
/* 074 */         final int project_numElements = project_value.numElements();
/* 075 */         project_arrayWriter.initialize(project_holder1, project_numElements, 8);
/* 076 */
/* 077 */         for (int project_index = 0; project_index < project_numElements; project_index++) {
/* 078 */           if (project_value.isNullAt(project_index)) {
/* 079 */             project_arrayWriter.setNullDouble(project_index);
/* 080 */           } else {
/* 081 */             final double project_element = project_value.getDouble(project_index);
/* 082 */             project_arrayWriter.write(project_index, project_element);
/* 083 */           }
/* 084 */         }
/* 085 */       }
/* 086 */
/* 087 */       project_rowWriter.setOffsetAndSize(0, project_tmpCursor, project_holder1.cursor - project_tmpCursor);
/* 088 */       project_result.setTotalSize(project_holder1.totalSize());
/* 089 */       append(project_result);
/* 090 */       if (shouldStop()) return;
/* 091 */     }
/* 092 */   }
/* 093 */ }

How was this patch tested?
Added unit tests into DataFrameComplexTypeSuite

Test build #61250 has started for PR 13909 at commit 37e4ce2.

Test build #61250 has finished for PR 13909 at commit 37e4ce2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61250/
Test FAILed.

Test build #61264 has started for PR 13909 at commit 681be95.

Test build #61264 has finished for PR 13909 at commit 681be95.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61264/
Test PASSed.

Test build #61282 has started for PR 13909 at commit 4238268.

How much overhead we can reduce with this?

Test build #61282 has finished for PR 13909 at commit 4238268.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61282/
Test FAILed.

Test build #61285 has started for PR 13909 at commit c2da5bd.

Test build #61285 has finished for PR 13909 at commit c2da5bd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61285/
Test PASSed.

@viirya I should have a benchmark. When https://github.com/apache/spark/pull/13758 is merged and https://issues.apache.org/jira/browse/SPARK-16223 is solved, I will prepare benchmark result.

Test build #68396 has started for PR 13909 at commit 990d6c8.

@cloud-fan @hvanhovell could you please review this instead of #13758.
cc: @viirya
It would be good to close #13758 when this would be merged.

Test build #68396 has finished for PR 13909 at commit 990d6c8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68396/
Test FAILed.

Merged build finished. Test FAILed.

Test build #68399 has started for PR 13909 at commit 425522c.

Test build #68399 has finished for PR 13909 at commit 425522c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68399/
Test FAILed.

Test build #68403 has started for PR 13909 at commit 19a4e1a.

Test build #68403 has finished for PR 13909 at commit 19a4e1a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68403/
Test FAILed.

Looks like this also hits the 64KB problem...

Test build #68406 has started for PR 13909 at commit 46a660b.

@viirya What do you mean "the 64KB problem"?

The exception on jenkins tests:
[info]   java.util.concurrent.ExecutionException: java.lang.Exception: failed to compile: org.codehaus.janino.JaninoRuntimeException: Code of method "processNext()V" of class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator" grows beyond 64 KB

Yes, for this test.
The root cause for other failures is to put a benchmark with test() instead of ignore().

Why with #15780 this failure did not occur? That pr seems not related to 64 KB bytecode size issue?

Test build #68406 has finished for PR 13909 at commit 46a660b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68406/
Test FAILed.

Yes, #15780 did not directly change the size of generated code. However, since it changes nullable in Dataframe schema, the code size is changed.
Let me show you the following example. While you can see if-statements for nullcheck without #15780, you see no if-statements for null check with #15780. Precise schema information can improve quality of generated code and reduce code size.
without #15780
/* 039 */   protected void processNext() throws java.io.IOException {
/* 040 */     while (inputadapter_input.hasNext()) {
/* 041 */       InternalRow inputadapter_row = (InternalRow) inputadapter_input.next();
/* 042 */       int inputadapter_value = inputadapter_row.getInt(0);
/* 043 */
/* 044 */       boolean project_isNull1 = false;
/* 045 */
/* 046 */       int project_value1 = -1;
/* 047 */       project_value1 = inputadapter_value + 1;
/* 048 */       project_values[0] = project_value1;
/* 049 */
/* 050 */       boolean project_isNull4 = false;
/* 051 */
/* 052 */       int project_value4 = -1;
/* 053 */       project_value4 = inputadapter_value + 2;
/* 054 */       project_values[1] = project_value4;
/* 055 */       final ArrayData project_value = org.apache.spark.sql.catalyst.expressions.UnsafeArrayData.fromPrimitiveArray(project_values);
/* 056 */       project_holder.reset();
/* 057 */
/* 058 */       // Remember the current cursor so that we can calculate how many bytes are
/* 059 */       // written later.
/* 060 */       final int project_tmpCursor = project_holder.cursor;
...

With #15780
/* 037 */   protected void processNext() throws java.io.IOException {
/* 038 */     while (inputadapter_input.hasNext()) {
/* 039 */       InternalRow inputadapter_row = (InternalRow) inputadapter_input.next();
/* 040 */       int inputadapter_value = inputadapter_row.getInt(0);
/* 041 */
/* 042 */       final boolean project_isNull = false;
/* 043 */       this.project_values = new Object[2];
/* 044 */       boolean project_isNull1 = true;
/* 045 */       int project_value1 = -1;
/* 046 */
/* 047 */       if (!false) {
/* 048 */         project_isNull1 = false; // resultCode could change nullability.
/* 049 */         project_value1 = inputadapter_value + 1;
/* 050 */
/* 051 */       }
/* 052 */       if (project_isNull1) {
/* 053 */         project_values[0] = null;
/* 054 */       } else {
/* 055 */         project_values[0] = project_value1;
/* 056 */       }
/* 057 */
/* 058 */       boolean project_isNull4 = true;
/* 059 */       int project_value4 = -1;
/* 060 */
/* 061 */       if (!false) {
/* 062 */         project_isNull4 = false; // resultCode could change nullability.
/* 063 */         project_value4 = inputadapter_value + 2;
/* 064 */
/* 065 */       }
/* 066 */       if (project_isNull4) {
/* 067 */         project_values[1] = null;
/* 068 */       } else {
/* 069 */         project_values[1] = project_value4;
/* 070 */       }
/* 071 */
/* 072 */       final ArrayData project_value = new org.apache.spark.sql.catalyst.util.GenericArrayData(project_values);
/* 073 */       this.project_values = null;
/* 074 */       project_holder.reset();
/* 075 */
/* 076 */       project_rowWriter.zeroOutNullBytes();
/* 077 */
/* 078 */       if (project_isNull) {
/* 079 */         project_rowWriter.setNullAt(0);
/* 080 */       } else {
/* 081 */         // Remember the current cursor so that we can calculate how many bytes are
/* 082 */         // written later.
/* 083 */         final int project_tmpCursor = project_holder.cursor;
...

Test build #68418 has started for PR 13909 at commit c6c6cbc.

Test build #68418 has finished for PR 13909 at commit c6c6cbc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68418/
Test FAILed.

Merged build finished. Test FAILed.

@kiszk Thanks for clarifying it. Looks like two generated codes above should be swapped?

You are right. I swapped two comments.

Test build #68440 has started for PR 13909 at commit 03d1887.

Test build #68440 has finished for PR 13909 at commit 03d1887.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68440/
Test FAILed.

Test build #68463 has started for PR 13909 at commit 3ae0da7.

Test build #68463 has finished for PR 13909 at commit 3ae0da7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68463/
Test FAILed.

Test build #68468 has started for PR 13909 at commit 6941b58.

Test build #68468 has finished for PR 13909 at commit 6941b58.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68468/
Test FAILed.

Test build #68483 has started for PR 13909 at commit 5d12ee3.

Test build #68483 has finished for PR 13909 at commit 5d12ee3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68483/
Test PASSed.

ping @cloud-fan @hvanhovell

Test build #68520 has started for PR 13909 at commit 307d1b2.

Test build #68520 has finished for PR 13909 at commit 307d1b2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68520/
Test FAILed.

Test build #68523 has started for PR 13909 at commit 357171f.

Test build #68523 has finished for PR 13909 at commit 357171f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68523/
Test FAILed.

Test build #68526 has started for PR 13909 at commit a82ed38.

Test build #68526 has finished for PR 13909 at commit a82ed38.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68526/
Test PASSed.

Merged build finished. Test PASSed.

@cloud-fan @hvanhovell could you please review this since there is no failure now?

Test build #68587 has started for PR 13909 at commit ce11595.

Test build #68588 has started for PR 13909 at commit df9fa8e.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68588/
Test FAILed.

Test build #68589 has started for PR 13909 at commit eb1e07a.

Test build #68587 has finished for PR 13909 at commit ce11595.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68587/
Test PASSed.

Test build #68589 has finished for PR 13909 at commit eb1e07a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68589/
Test FAILed.

Test build #68594 has started for PR 13909 at commit 3ef6fd4.

Test build #68594 has finished for PR 13909 at commit 3ef6fd4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68594/
Test PASSed.

Merged build finished. Test PASSed.

Test build #68662 has started for PR 13909 at commit 22e5356.

Test build #68662 has finished for PR 13909 at commit 22e5356.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68662/
Test PASSed.

Except for the two comments regarding the test, I don't have other question for now. cc @hvanhovell @cloud-fan for more checks.

Test build #68779 has started for PR 13909 at commit 9745571.

Test build #68779 has finished for PR 13909 at commit 9745571.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68779/
Test PASSed.

I'd like to introduce a new expression: CreatePrimitiveArray. In the optimizer if we find the element type is primitive type, replace CreateArray with CreatePrimitiveArray.
BTW, we don't need to require all expressions to be non-nullable, we can just put a default value to the primitive array if that element is null, and call UnsafeArrayData.setNullAt later. (we need to implement ArrayData.setNullAt though)

Test build #68856 has started for PR 13909 at commit b713712.

Test build #68857 has started for PR 13909 at commit b454066.

Test build #68857 has finished for PR 13909 at commit b454066.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68857/
Test FAILed.

Merged build finished. Test FAILed.

Test build #68856 has finished for PR 13909 at commit b713712.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68856/
Test PASSed.

Merged build finished. Test PASSed.

Test build #68941 has started for PR 13909 at commit 64ec081.

Test build #68941 has finished for PR 13909 at commit 64ec081.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68941/
Test FAILed.

retest this please.

Test build #68962 has started for PR 13909 at commit 64ec081.

Test build #68962 has finished for PR 13909 at commit 64ec081.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68962/
Test PASSed.

@cloud-fan , @hvanhovell could you please review this again?

Test build #68992 has started for PR 13909 at commit 1fecc74.

Test build #68992 has finished for PR 13909 at commit 1fecc74.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68992/
Test PASSed.

Merged build finished. Test PASSed.

@cloud-fan , @hvanhovell @viirya could you please review this again except if the current Array[Byte] is OK?

Test build #69254 has started for PR 13909 at commit 8b2c787.

Test build #69254 has finished for PR 13909 at commit 8b2c787.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69254/
Test FAILed.

Jenkins, retest this please

Test build #69272 has started for PR 13909 at commit 8b2c787.

Test build #69272 has finished for PR 13909 at commit 8b2c787.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69272/
Test PASSed.

ping @cloud-fan

ping @cloud-fan @hvanhovell

Test build #69616 has started for PR 13909 at commit b29d7cf.

Test build #69616 has finished for PR 13909 at commit b29d7cf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69616/
Test PASSed.

ping @cloud-fan

Test build #69723 has started for PR 13909 at commit 41a5435.

Test build #69723 has finished for PR 13909 at commit 41a5435.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69723/
Test FAILed.

Test build #69728 has started for PR 13909 at commit 185ddf5.

Test build #69728 has finished for PR 13909 at commit 185ddf5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69728/
Test PASSed.

Merged build finished. Test PASSed.

@cloud-fan thank you for your thoughtful review comments. The latest one looks better.

ping @cloud-fan

After some more thoughts, will it be better to use UnsafeArrayWriter directly? The array elements are evaluated one by one, so theoretically creating a primitive array and copying it to UnsafeArray is also sub-optimal.
With UnsafeArrayWriter, we can write array elements to unsafe array one by one, no copy anymore.

oh nvm, for non-primitive element type, new GenericArrayData is faster than UnsafeArrayData, your approach should be good.

For primitive element type, should we optimize non UnsafeArrayWriter path? I mean that we just fill zero into nullableand then perform bulkcopy only for data part by adding a new API toUnsafeArrayData`. This seems to be better than element-wise copy.
What do you think?

creating a primitive array and setting element values is also element-wise copy right? And we need an extra bulkcopy to write the primitive array to unsafe array. By using unsafe array writer, we can save the extra bulkcopy

@cloud-fan Yeah, to keep primitive array and to use UnsafeArrayWrite can avoid extra bulkcopy.
To achieve this avoidance, I think that we need to create GenericArrayData after enabling https://github.com/apache/spark/pull/13758 instead of UnsfeArrayData. To create GenericArrayData with  https://github.com/apache/spark/pull/13758 can keep a primitive array inside it.
Does it mean what you want to do?

In CreateArray we can just evaluate each expression and write their results to unsafe array with the writer, no intermediate data need to be created.

Yea we can avoid the intermediate primitive array. Maybe we can benchmark against it also.

I understand what you want to do. I agree with this in this case. This is because the child of Project is an array creation.
== Physical Plan ==
*Project [array((value#2 + 1.1), (value#2 + 2.2)) AS array((value + 1.1), (value + 2.2))#10]
+- *SerializeFromObject [input[0, double, false] AS value#2]
   +- Scan ExternalRDDScan[obj#1]

I still have two questions.

Is this optimization general? If we have additional operations between Project and an array creation, what physical plan is generated?
How can we know this case at an array creation? An evaluation order of a physical plan tree is bottom-up.

Let me clarify these questions.

I think it's nothing to do with physical operators, we are talking about CreateArray right? Avoiding to create the intermedia primitive array should be faster generally.

Ah, I understand my misunderstanding. In this discussion, you mean "intermediate array" is "new int[]".
Yes, let me do it.

Test build #69969 has started for PR 13909 at commit 08beb88.

I finished it for createArray. I will do it for createMap tomorrow.

Test build #69969 has finished for PR 13909 at commit 08beb88.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69969/
Test FAILed.

Test build #69989 has started for PR 13909 at commit b0a7dab.

Test build #69989 has finished for PR 13909 at commit b0a7dab.

This patch fails PySpark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69989/
Test FAILed.

Test build #69995 has started for PR 13909 at commit 438944b.

Test build #69995 has finished for PR 13909 at commit 438944b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69995/
Test PASSed.

Test build #70010 has started for PR 13909 at commit f418062.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70010/
Test FAILed.

Jenkins, retest this please

Test build #70013 has started for PR 13909 at commit f418062.

Test build #70013 has finished for PR 13909 at commit f418062.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70013/
Test PASSed.

Test build #70327 has started for PR 13909 at commit d24c7b1.

Test build #70327 has finished for PR 13909 at commit d24c7b1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70327/
Test PASSed.

ping @cloud-fan

sorry for the delay. Yea looks like we can't reuse the byte array of unsafe data in expressions, which may get cached expectedly and leads to wrong result.
I'm a little concerned about the hacks in BufferHolder and the array writer. The code is so coupled with unsafe row writer and we have to hack it so that we can write unsafe array directly. What if we actually write an unsafe row with a single array field and return the array column? Then we don't need the hacks, but waste some bits for the row format overhead, which seems acceptable.

Test build #70480 has started for PR 13909 at commit c159f03.

@cloud-fan thank you for your comments. I implemented to create UnsafeArrayData using UnsafeRow. I hope that this makes the creation simpler.

Test build #70480 has finished for PR 13909 at commit c159f03.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70480/
Test PASSed.

Test build #70537 has started for PR 13909 at commit 0af0828.

Test build #70537 has finished for PR 13909 at commit 0af0828.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70537/
Test FAILed.

Jenkins, retest this please

Test build #70540 has started for PR 13909 at commit 0af0828.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70540/
Test FAILed.

Jenkins, retest this please

Test build #70545 has started for PR 13909 at commit 0af0828.

Test build #70545 has finished for PR 13909 at commit 0af0828.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70545/
Test PASSed.

Merged build finished. Test PASSed.

Test build #70555 has started for PR 13909 at commit f6e9a83.

Test build #70555 has finished for PR 13909 at commit f6e9a83.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70555/
Test PASSed.

LGTM, I left some comments about code style, thanks for working on it!
BTW, can you update the benchmark result? I think we have finalized the solution now.

Test build #70568 has started for PR 13909 at commit 327c8ac.

Test build #70568 has finished for PR 13909 at commit 327c8ac.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70568/
Test FAILed.

Test build #70570 has started for PR 13909 at commit 28df09f.

Test build #70570 has finished for PR 13909 at commit 28df09f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70570/
Test PASSed.

LGTM except for few minor comments.

Test build #70578 has started for PR 13909 at commit 293b344.

Test build #70578 has finished for PR 13909 at commit 293b344.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70578/
Test PASSed.

Test build #70584 has started for PR 13909 at commit 69d5e33.

Test build #70584 has finished for PR 13909 at commit 69d5e33.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70584/
Test FAILed.

Test build #70606 has started for PR 13909 at commit 2556ba5.

LGTM, pending jenkins

Test build #70607 has started for PR 13909 at commit 34bff15.

Test build #70608 has started for PR 13909 at commit 4a0409a.

Test build #70606 has finished for PR 13909 at commit 2556ba5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70606/
Test PASSed.

Test build #70607 has finished for PR 13909 at commit 34bff15.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70607/
Test PASSed.

Test build #70608 has finished for PR 13909 at commit 4a0409a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70608/
Test PASSed.

Merged build finished. Test PASSed.

Test build #70614 has started for PR 13909 at commit dcce4c5.

Should use dataType.containsNull for allowNull?

no we shouldn't,  containsNull is kind of an optimizing hint that guarantees the value will not be null, and it's guaranteed by the source, not here in CreateArray.

@cloud-fan Hmm, I agree that it's guaranteed by the source, but I thought that's why we should use containsNull for allowNull to guarantee the value will not be null when containsNull == false in this array for the caller of this expression. ditto for CreateMap.

@ueshin That will be a big mess if we validate the null here and there, I think we should only do it at the source side. If we do not trust the source side, there are a lot of other places that need to add validation logic.

Test build #70614 has finished for PR 13909 at commit dcce4c5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70614/
Test PASSed.

@cloud-fan Hmm, I didn't mean we don't trust the source side.
Currently we DO validate the null regardless of allowNull value.
The difference by the allowNull is setting null or throwing an exception.
I think which we should do when containsNull == false is throwing an exception.
(Anyway, eval.isNull of most of children will be "false" when nullable == false, so Janino will remove the $isNullAssignment clause.

I think the validation logic should be consistent across the system, you can't add the null check in CreateArray just because it's easy to do. Currently we do null check based on nullable property only at source side and we should follow it. And the previous code doesn't have null check either.
BTW, this change has external influence, the error message for null map key has been changed.


enable_manage_gitlab_token = false does not store runner token

Hello,
when starting the runner with enable_manage_gitlab_token = false the script is supposed to set the token in SSM. However there is an error due to a missing permission.
A workaround is to put the token value manually into SSM.
This is the error from the EC2 instance user-data:
aws ssm put-parameter --overwrite --type SecureString --name gitlab-runner-manager-runner-token --value=XXX --region eu-central-1

An error occurred (AccessDeniedException) when calling the PutParameter operation: User: arn:aws:sts::XXX:assumed-role/gitlab-runner-manager-instance/i-XXX is not authorized to perform: ssm:PutParameter on resource: arn:aws:ssm:eu-central-1:XXX:parameter/gitlab-runner-manager-runner-token because no identity-based policy allows the ssm:PutParameter action

From the docs:
"Boolean to enable the management of the GitLab token in SSM. If `true` the token will be stored in SSM, which means the SSM property is a terraform managed resource. If `false` the Gitlab token will be stored in the SSM by the user-data script during creation of the the instance. However the SSM parameter is not managed by terraform and will remain in SSM after a `terraform destroy`."

I checked the code and found that enable_manage_gitlab_token steers the creation of the ssm:PutParameter and ssm:GetParameter authorization only. And as explained in the docs the token will be stored in an SSM parameter always. This is what you have seen in your logs.
Suggestion is to remove this enable_manage_gitlab_token variable as the policy is always needed and no switch is needed.

Hmm I am not sure if there is a misunderstanding. There is a definitely a difference in the behavior in that the SSM parameter is not removed when 'enable_manage_gitlab_token' is set to false. That allows to reuse the exact same runner token the next time terraform apply is called. I do not agree that the parameter should be removed. It does however, to my understanding, not function correctly the first time terraform apply runs, because the token is not correctly stored by the user script when the token parameter in SSM is null. Thats when the error in my first post is logged and this can probably be fixed by adding the correct permission. After using the workarround of creating the SSM parameter manually, subsequent calls to terraform destroy / terraform apply function correctly and the SSM parameter is no longer touched, which is the desired behavior for enable_manage_gitlab_token = false

All right, so let's check what's going on here. I searched in version 5.0.2 for usages of var.enable_manage_gitlab_token. There are 2 usages: main.tf line 472 and 482. The resources created there are policies. These policies are never referenced.
In gitlab-runner.tpl I found this code
# fetch Runner token from SSM and validate it
token=$(aws ssm get-parameters --names "${secure_parameter_store_runner_token_key}" --with-decryption --region "${secure_parameter_store_region}" | jq -r ".Parameters | .[0] | .Value")

valid_token=true
if [[ `echo $token` != "null" ]]
then
  valid_token_response=$(curl -s -o /dev/null -w "%%{response_code}" --request POST -L "${runners_gitlab_url}/api/v4/runners/verify" --form "token=$token" )
  [[ `echo $valid_token_response` != "200" ]] && valid_token=false
fi

if [[ `echo ${runners_token}` == "__REPLACED_BY_USER_DATA__" && `echo $token` == "null" ]] || [[ `echo $valid_token` == "false" ]]
then
  token=$(curl --request POST -L "${runners_gitlab_url}/api/v4/runners" \
    --form "token=${gitlab_runner_registration_token}" \
    --form "tag_list=${gitlab_runner_tag_list}" \
    --form "description=${giltab_runner_description}" \
    --form "locked=${gitlab_runner_locked_to_project}" \
    --form "run_untagged=${gitlab_runner_run_untagged}" \
    --form "maximum_timeout=${gitlab_runner_maximum_timeout}" \
    --form "access_level=${gitlab_runner_access_level}" \
    | jq -r .token)
  aws ssm put-parameter --overwrite --type SecureString  --name "${secure_parameter_store_runner_token_key}" --value="$token" --region "${secure_parameter_store_region}"
fi

Thus the instance always needs the policy to access the SSM.
The runner token SSM parameter is created in line 12 and if you run a terraform destroy the SSM parameter will be removed. At least I do not see any special logic.
Am I overlooking something?

I am new to terraform and AWS so I am not sure I can help with debugging. But per the description of the parameter, the desired functionality would be to not create or destroy the SSM parameter with terraform apply and terraform destroy and instead only read / write it in the EC2 instance, correct? At least how I understand the description, the desired behavior is that the token is stored and retrieved from SSM but not being removed on terraform destroy, thus being not managed by terraform but still being used for creating the runner. When terraform apply´ is called, the token is stored in SSM with a value of null, regardless wether enable_manage_gitlab_token` is true or false. Would it help to grant the GetParameter and PutParameter policies to the instance but remove the creation of the SSM parameter from terraform apply when its false?

According to the docs it should work the way you described it. But it looks totally different in the code.
Can you please shed some light on this, @npalm


refactor(elements|ino-datepicker): separate validation and configuration from component

Closes #257
Proposed Changes

Move validation logic to validation.ts
Move type-specific options to type-specific-options.ts
Add some more test to ensure the functionality
Remove focus workaround from the past
Remove updateFlatpickr method to support typing

⚠ This PR is currently based on #271, so I'll wait until that one is merged and rebase this one afterward.
As there are some problems with #271, I rebased this one onto the current master so this can be reviewed now.

This can be reviewed now

This refactoring might have fixed the bug https://github.com/inovex/elements/issues/215 as well. Can you confirm this? @pfecht

@Sl1nd mentioned another bug in the old datepicker component. We should check if this bug is fixed here as well.

What about the pending comment about onCreate? otherwised it looks good to me 👍

@pfecht: Which comment do you mean exactly?

@pfecht: Which comment do you mean exactly?

I didn't submit the review a couple of weeks ago 🙈

@pfecht: Good idea to use the onReady and onDestroy methods of the flatpickr, didn't think about that.
I've implemented the changes you suggested and made some more changes because I realized that I'm actually implementing the factory pattern without thinking about it. It might seem like a lot but I mostly encapsulated all picker types to its own file and renamed some stuff.
I would appreciate your review one (last?) time 🙏🏼

@Sl1nd mentioned another bug in the old datepicker component. We should check if this bug is fixed here as well.

The issue is not fixed by this PR. Updating validation state is missing in 'min' and 'max' watcher. Should we integrate the fixes into this PR?


Swagger UI is not showing fileupload button for RequestBody of type Byte Array

Question
@ApiOperation(value = WOPIConstants.API_FILE_UPDATE_CONTENT, response = boolean.class)
@ApiResponses(value = {
        @ApiResponse(code = HttpStatus.SC_OK, message = WOPIConstants.API_FILE_UPLOADED)
}
)
@PostMapping(value = WOPIConstants.REST_END_POINT_PUT_UPDATED_FILE, consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
public boolean postFile(@RequestPart("files") Flux<FilePart> filePartFlux, @PathVariable final String name) {
    boolean isSuccessful = false;
    try {
        LOGGER.info(WOPIConstants.INFO_WOPISERVICE_PUTFILE_NAME + name);
        Flux<String> stringFlux = filePartFlux.flatMap(filePart ->
                filePart.content().map(dataBuffer -> {
                    byte[] bytes = new byte[dataBuffer.readableByteCount()];
                    LOGGER.info(WOPIConstants.INFO_WOPISERVICE_PUTFILE_RECEIVEDBYTES + bytes.length);
                    dataBuffer.read(bytes);
                    DataBufferUtils.release(dataBuffer);
                    byte[] encoded = Base64.getEncoder().encode(bytes);
                    return new String(encoded, StandardCharsets.UTF_8);
                }));
        final String encodedData = stringFlux.blockFirst();
        PutFileContentReq putFileContentReq;
        if (!"DATA".contains(name)) {
            putFileContentReq = new PutFileContentReq(insKey, encodedData);
        } else {
            putFileContentReq = new PutFileContentReq(name, encodedData);
        }
        LOGGER.info(WOPIConstants.INFO_WOPISERVICE_PUTFILE_PUTFILECONTENTREQ + putFileContentReq.toString());
        PutFileContentRes putFileContentRes = pegaClient.putUpdatedFileContent(baseURL, basicAuthHeader, putFileContentReq);
        LOGGER.info(WOPIConstants.INFO_WOPISERVICE_PUTFILE_PUTFILECONTENTREs + putFileContentRes.toString());
        isSuccessful = putFileContentRes.isSuccessful();
    } catch (Exception e) {
        LOGGER.error(WOPIConstants.EXCEPTION_WOPISERVICE_POSTFILE + e);
    }
    return isSuccessful;
}

Using springfox-swagger2 and springfox-swagger-ui Version : 2.9.2
Above mentioned is my code of a rest service which accepts a file in request body. Is there a way in Swagger UI to get fileupload button
In postman I was able to send file in the following way

In swagger Ui it is coming like this

Is there a way that I can annotate to determine that requestbody is a byte array of a file so that  I can get File upload button in Swagger UI

@dilipkrish will this be available in v3

Yes it should be available as soon as it's released


Declare the KubeSpray version in the Code

The KubeSpray version now for kubean is dynamic, it changed in CI.
The KubeSpray it an important dependency for the kubean.
Should we declare it at the code like go mod ?

/kind feature

https://github.com/kubean-io/kubean-helm-chart/blob/main/charts/kubean/templates/manifest.cr.yaml
Declared in manifest CR.


Adding Medal: MeDAL: Medical Abbreviation Disambiguation Dataset for Natural Language Understanding Pretraining

Could you fix the dummy data before we merge ?
Looks like the dummy train.csv is missing

Thanks @Narsil @lhoestq for adding MeDAL :)


InlineAbbreviations.pdf

Hi!
I downloaded InlineAbbreviations.pdf and it does not open regardless if I try with Mac Preview or Adobe Reader
although github does render it correctly.

Be careful that your downloading process isn't scrambling line endings as might get auto-confused as text (since I made the PDF text editor compatible so there are no binary bytes). Or your A/V is triggering and altering the file. Try downloading the ZIP of the repo directly and just extracting from there.
I don't own a Mac, but I have no problems opening it with the following (in no specific order) on PC/Windows 10 and, for some of these, WSL2 Ubuntu via XWindows and on a native Ubuntu box: Adobe Acrobat, PDF XChange, Nitro, Foxit, PDF Studio Viewer, Javelin3, Slim PDF Reader, PDF Architect, Xodo, MuPDF, GS, Chrome, FireFox, Edge, Xpdf, PDFBox Debugger app. Output, of course, varies.

I am just downloading the file using Chrome.
Only If I click on the file and let Github render it and then click on Raw the download is fine.

That's how GitHub works - the same thing happens with MD, source code or any other file. The HTML is the GH page for the file, not the file itself. You always need to use the Raw to get file.


Fixing the app creation scripts

after the renaming in the parameters.json

@abhidnya13 : if this works for you, I'll let you merge ....

cc: @rayluo

Thanks @jmprieur :shipit:

+1 on this :-)


Add 20 second timeout to IDS resolution and more detailed logs (#BRI-7171)

@bradtgmurray not necessarily asking for review, just your thoughts on if this is a good idea or could be implemented better

Also see https://linear.app/beeper/issue/BRI-7171#comment-d03210aa

How do these logs get plumbed through back to the checkpoint when we try to send a message so we have data on the frequency on each of these cases?

It seems that logs in barcelona-mautrix are currently only being sent to mautrix-imessage and oslog, so I don't think they ever make it to any sort of analytics checkpoint. As far as I can tell, we can't really send the checkpoint a success with extra data, so I don't think that notifying the checkpoint of this partial failure is possible. We could add analytics event of some sort (Growthbook? Sentry? What do we use here?), if you think that would be best.

I think this is fine to merge as is and we can figure out how to further improve.
I was just wondering alound how to get more info into the "info" field of a checkpoint when a sent message fails for this reason. ex: https://metabase.beeper-tools.com/question#eyJkYXRhc2V0X3F1ZXJ5Ijp7InR5cGUiOiJxdWVyeSIsInF1ZXJ5Ijp7InNvdXJjZS10YWJsZSI6MTQ0MywiZmlsdGVyIjpbImFuZCIsWyJjb250YWlucyIsWyJmaWVsZCIsNDAxNjksbnVsbF0sIlNNUyIseyJjYXNlLXNlbnNpdGl2ZSI6ZmFsc2V9XSxbIj0iLFsiZmllbGQiLDQwMTc3LG51bGxdLCJpbWVzc2FnZWNsb3VkIl0sWyJ0aW1lLWludGVydmFsIixbImZpZWxkIiw0ODY0MSxudWxsXSwtNywiZGF5Iix7ImluY2x1ZGVfY3VycmVudCI6ZmFsc2V9XV19LCJkYXRhYmFzZSI6MTN9LCJkaXNwbGF5IjoidGFibGUiLCJ2aXN1YWxpemF0aW9uX3NldHRpbmdzIjp7fX0=


Font rendering error

I discovered an error while drawing using my font (incorrectly drawn characters). I think the error is because of incorrect parsing of curves. Can you take a look at it?
Here is my font: http://www.ivank.net/Bentham.otf
I tested it here: http://nodebox.github.io/opentype.js/
Image: http://www.ivank.net/bentham.png

That looks weird -- I'll take a look soon.

While trying to find out if this is the same issue as #117 or a separate one, I wasn't able to reproduce this.

Closing this issue now, please reopen or open a new one, should there still be issues regarding that font.

I have "solved" it back in 2016 by writing my own library Typr.js :) https://github.com/photopea/Typr.js


List instead of regex

Right now for excluding the files it is required to use regex. For the other tools like flake8 the exclude contain a list of paths which should be excluded.
The regex is really difficult to read write and also it is hard to hold it inside the .toml file. The regex pattern contains also \, \\, \/ which are not allowed in a config files, you have to makes workaround for it. Also the regex is not the best option for the paths.
The solution can be a list which holds a paths and also file names. Then it easy to read and it is definitely more user friendly.
Example: excluded=["src/design/*", "docs/*", "src/my_file.py"]

Hi @LisPrzemyslaw
I've added a new option, excludes as well as exclude
excludes, as in excludes one or more paths
I couldn't add it instead of as it could break for people
Both can be used alone, or with each other
At least all the above should work, but please let me know if you spot a bug!
This will be available now under 0.60.0
Thanks!
Stephen


feat: Ideas for DT Kit

[ ] Elaborate on Provisioning and why it matters
[ ] setting asset/properties/@type when allowing access to multiple submodels
[ ] Comment on server-side filtering in EDC-Mgmt-Assets or AAS-Registry-API
[ ] Tutorials for all three roles (opco, prov, cons)
[ ] Remove explicit dependency to EDC
[ ] Harmonize spelling, introduce abbreviations on each page and be consistent afterwards.

In general, readability can be improved here and there, some typos and syntax glitches can be removed, tenses should be kept consistent.

https://github.com/eclipse-tractusx/eclipse-tractusx.github.io/blob/c622ff831ff69004b7b9939e14f967c686ab0c25/docs-kits/kits/Digital Twin Kit/page_adoption-view.md?plain=1#L62

What do you mean by expanding the API from different use cases? In which manner?

https://github.com/eclipse-tractusx/eclipse-tractusx.github.io/blob/c622ff831ff69004b7b9939e14f967c686ab0c25/docs-kits/kits/Digital Twin Kit/page_adoption-view.md?plain=1#L65

If you have no picture in mind, it his hard to follow the "noth-" and "southward" direction.

https://github.com/eclipse-tractusx/eclipse-tractusx.github.io/blob/c622ff831ff69004b7b9939e14f967c686ab0c25/docs-kits/kits/Digital Twin Kit/page_adoption-view.md?plain=1#L77

In general, avoid abbreviation(s), here: use "information" instead of "info"

https://github.com/eclipse-tractusx/eclipse-tractusx.github.io/blob/c622ff831ff69004b7b9939e14f967c686ab0c25/docs-kits/kits/Digital Twin Kit/page_adoption-view.md?plain=1#L84

What about citing the CX-0003 for Semantic Aspect Meta Models?

https://github.com/eclipse-tractusx/eclipse-tractusx.github.io/blob/b5f31baafac0af1ce0d1a8a29c73063936444a26/docs-kits/kits/Digital Twin Kit/page_software-operation-view.md?plain=1#L21

Not just BPN Discovery Servers. Also EDC discovery included.
Should we rename "identifiers" to "specific asset identifiers", since actually this is what it is pointing to.
I propose: "A microservice resolving a type of specific asset identifiers, e.g. manufactuerPartId against a set of BPN-, and EDC-Discovery Servers hosted at Catena-X operating companies."
The same holds true in an analogue manner for the following descriptions

https://github.com/eclipse-tractusx/eclipse-tractusx.github.io/blob/b5f31baafac0af1ce0d1a8a29c73063936444a26/docs-kits/kits/Digital Twin Kit/page_software-operation-view.md?plain=1#L28

Is the meaning of abbreviations, e.g., "OpCo", "IDTA", "IDSA", "AAS" known to the reader?
Otherwise, please introduce those.

https://github.com/eclipse-tractusx/eclipse-tractusx.github.io/blob/b5f31baafac0af1ce0d1a8a29c73063936444a26/docs-kits/kits/Digital Twin Kit/Software Development View/page_software-development-view.md?plain=1#L18

Since the link is dead, please update the link.

https://github.com/eclipse-tractusx/eclipse-tractusx.github.io/blob/b5f31baafac0af1ce0d1a8a29c73063936444a26/docs-kits/kits/Digital Twin Kit/Software Development View/page_software-development-view.md?plain=1#L25

Since this is a citation, please cite the source as well.

https://github.com/eclipse-tractusx/eclipse-tractusx.github.io/blob/b5f31baafac0af1ce0d1a8a29c73063936444a26/docs-kits/kits/Digital Twin Kit/Software Development View/page_software-development-view.md?plain=1#L27

"What exactly is defined in the Catena-X standard CX - 0002."
Is this a question or a reference to the previous sentence?
If you refer to the previous sentence, I propose: "This subset is defined in ..."

https://github.com/eclipse-tractusx/eclipse-tractusx.github.io/blob/b5f31baafac0af1ce0d1a8a29c73063936444a26/docs-kits/kits/Digital Twin Kit/Software Development View/page_software-development-view.md?plain=1#L33

Name "SerialPartTypization" instead of "Nameplate".
I suggest to refer to the description of the Semantic Aspect Meta Models (SAMM), e.g., SerialPartTypization

https://github.com/eclipse-tractusx/eclipse-tractusx.github.io/blob/b5f31baafac0af1ce0d1a8a29c73063936444a26/docs-kits/kits/Digital Twin Kit/Software Development View/page_software-development-view.md?plain=1#L66

Please adjust: "It allows to start a Discovery Process with only a type of AssetId, e.g., manufacturerPartId and the actual content of an AssetId.
We should think about if we cahnge AssetId and similar to AssetId?

https://github.com/eclipse-tractusx/eclipse-tractusx.github.io/blob/b5f31baafac0af1ce0d1a8a29c73063936444a26/docs-kits/kits/Digital Twin Kit/Software Development View/page_software-development-view.md?plain=1#L67

I do not understand the meaning of this sentence. Please clarify/simplify this sentence.

https://github.com/eclipse-tractusx/eclipse-tractusx.github.io/blob/b5f31baafac0af1ce0d1a8a29c73063936444a26/docs-kits/kits/Digital Twin Kit/Software Development View/page_software-development-view.md?plain=1#L100

You may rewrite this sentence: If this prior knowledge is given under all circumstances, Discovery Finder and BPN Discovery (steps 2-3) are not needed.

https://github.com/eclipse-tractusx/eclipse-tractusx.github.io/blob/b5f31baafac0af1ce0d1a8a29c73063936444a26/docs-kits/kits/Digital Twin Kit/Software Development View/page_software-development-view.md?plain=1#L180

What refers "She" to? I suggest: This data provider may create one ...

https://github.com/eclipse-tractusx/eclipse-tractusx.github.io/blob/b5f31baafac0af1ce0d1a8a29c73063936444a26/docs-kits/kits/Digital Twin Kit/Software Development View/page_software-development-view.md?plain=1#L186

Sentence incomplete. Maybe, it is not needed anymore? If needed please keep the tense, namely: "If a Data Provider wants to ..."

https://github.com/eclipse-tractusx/eclipse-tractusx.github.io/blob/b5f31baafac0af1ce0d1a8a29c73063936444a26/docs-kits/kits/Digital Twin Kit/Software Development View/page_software-development-view.md?plain=1#L191

That was quite right for the relase v3.1. From the v3.2 release onwards an data asset in the EDC can be a whole backend-system, which serves the content of submodels.
This is the description for v3.1 https://eclipse-tractusx.github.io/docs-kits/kits/Traceability Kit/Software Development View/Specification Traceability Kit/#data-provisioning: "EDC Asset ID: The EDC asset id under which the submodel was registered in the EDC. It must have the following format -"

https://github.com/eclipse-tractusx/eclipse-tractusx.github.io/blob/b5f31baafac0af1ce0d1a8a29c73063936444a26/docs-kits/kits/Digital Twin Kit/Software Development View/page_software-development-view.md?plain=1#L246

Does "BPNL:" need to be part of the ExternalReference:keys:[value]?

https://github.com/eclipse-tractusx/eclipse-tractusx.github.io/blob/b5f31baafac0af1ce0d1a8a29c73063936444a26/docs-kits/kits/Digital Twin Kit/Software Development View/page_software-development-view.md?plain=1#L254

With or without leading urn:uuid:?

https://github.com/eclipse-tractusx/eclipse-tractusx.github.io/blob/b5f31baafac0af1ce0d1a8a29c73063936444a26/docs-kits/kits/Digital Twin Kit/Software Development View/page_software-development-view.md?plain=1#L259
https://github.com/eclipse-tractusx/eclipse-tractusx.github.io/blob/b5f31baafac0af1ce0d1a8a29c73063936444a26/docs-kits/kits/Digital Twin Kit/Software Development View/page_software-development-view.md?plain=1#L305

"type": "GlobalReference" or "type": "Submodel"

https://github.com/eclipse-tractusx/eclipse-tractusx.github.io/blob/b5f31baafac0af1ce0d1a8a29c73063936444a26/docs-kits/kits/Digital Twin Kit/Software Development View/page_software-development-view.md?plain=1#L344

Is "DT" introduced as abbreviation to Digital Twin?


Implement logic for POST /users/session-verify

POST /users/session-verify
Validate JWT access token and return the appropriate status code to the user.

Use passport.js with a JWT strategy

b8f1b5e2effb23b87aab8db64baaeea85b6c714d


net/http: ServeContent()/ServeFile() doesn't return expected response when WriteTimeout happens

What version of Go are you using (go version)?

$ go version
go version go1.15.7 linux/amd64

Does this issue reproduce with the latest release?
YES
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://goproxy.cn,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build783936539=/tmp/go-build -gno-record-gcc-switches"

What did you do?

We are running simple proxy for retrieving blob files. Just in case, we set WriteTimeout for proxy server. The logic is described by the following code.
package main

import (
	"log"
	"net/http"
	"time"
)

// The content of filePath is in JSON format.
//
//
// {"architecture":"amd64","config":{"Hostname":"","Domainname":"","User":"0","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt"],"Cmd":null,"Image":"sha256:cbdac00ef1904d59ccd11add20108257111b8f65a854170f2d31caa78c4bb1b2","Volumes":null,"WorkingDir":"/","Entrypoint":["/go-runner"],"OnBuild":null,"Labels":{"description":"go based runner for distroless scenarios","maintainers":"Kubernetes Authors"}},"container_config":{"Hostname":"","Domainname":"","User":"0","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt"],"Cmd":["/bin/sh","-c","#(nop) COPY file:7fed31705f1c51a2c3f437d928dc68009c048406d85a556147a4a7aea5b15f7c in /usr/local/bin/kube-apiserver "],"Image":"sha256:cbdac00ef1904d59ccd11add20108257111b8f65a854170f2d31caa78c4bb1b2","Volumes":null,"WorkingDir":"/","Entrypoint":["/go-runner"],"OnBuild":null,"Labels":{"description":"go based runner for distroless scenarios","maintainers":"Kubernetes Authors"}},"created":"2020-12-18T12:21:06.637698261Z","docker_version":"19.03.8","history":[{"created":"1970-01-01T00:00:00Z","author":"Bazel","created_by":"bazel build ..."},{"created":"2020-11-24T01:39:04.46965334Z","created_by":"LABEL maintainers=Kubernetes Authors","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2020-11-24T01:39:04.46965334Z","created_by":"LABEL description=go based runner for distroless scenarios","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2020-11-24T01:39:04.46965334Z","created_by":"WORKDIR /","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2020-11-24T01:39:04.46965334Z","created_by":"COPY /workspace/go-runner . # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2020-11-24T01:39:04.46965334Z","created_by":"ENTRYPOINT [\"/go-runner\"]","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2020-12-18T12:21:06.637698261Z","created_by":"/bin/sh -c #(nop) COPY file:7fed31705f1c51a2c3f437d928dc68009c048406d85a556147a4a7aea5b15f7c in /usr/local/bin/kube-apiserver "}],"os":"linux","rootfs":{"type":"layers","diff_ids":["sha256:e7ee84ae4d1363ccf59b14bf34a79c245705dfd55429918b63c754d84c85d904","sha256:597f1090d8e9bd4f1847ea4b72a3c3ea1f0997011120816c9dae2fe858077b32","sha256:007430fdb20c04acd56f66f3032b747f8e5ac47edb6855a37ae9f37d7be15b81"]}}#
var filePath = "/tmp/75c7f711208082c548b935ab31e681ea30acccdce6b7abeecabae5bbfd326627"

type handler struct{}

func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	time.Sleep(2 * time.Second) // handle some logic
	http.ServeFile(w, r, filePath)
}

func main() {
	s := &http.Server{
		Addr:         "<IP_ADDRESS>:9999",
		Handler:      &handler{},
		WriteTimeout: 1 * time.Second,
	}
	log.Fatal(s.ListenAndServe())
}

If the handler takes time and the WriteTimeout happens, the http.ServeFile still can sends the data without HTTP Status line and header. We use the following code as client to reproduce the issue.
package main

import (
	"io"
	"io/ioutil"
	"net/http"
)

func main() {
	resp, err := http.Get("http://<IP_ADDRESS>:9999")
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()
	io.Copy(ioutil.Discard, resp.Body)
}

The result is like:
➜  /tmp go run client.go
panic: Get "http://<IP_ADDRESS>:9999": net/http: HTTP/1.x transport connection broken: malformed HTTP status code "based"

goroutine 1 [running]:
main.main()
        /tmp/client.go:12 +0x125
exit status 2

Go HTTP Server will call SetWriteDeadline() in readRequest. When we call http.ServeContent/ServeFile to write response, the go runtime will use (*response).ReadFrom to make it fast.
But we found that the chunkWriter.flush doesn't return error so that the ReadFrom doesn't know if the flush works.
// https://github.com/golang/go/blob/go1.15.7/src/net/http/server.go#L581

// ReadFrom is here to optimize copying from an *os.File regular file
// to a *net.TCPConn with sendfile.
func (w *response) ReadFrom(src io.Reader) (n int64, err error) {
       ...
       ...
       w.w.Flush()  // get rid of any previous writes
       w.cw.flush() // make sure Header is written; flush data to rwc
      ...
}

In this case, both the status and the header are lost because netpoll checker return I/O timeout. But the server still send data to client and the data is not valid http response, called malformed HTTP status. The WriteTimeout doesn't work well in ServeContent/ServeFile mode and return confusing response.  I think it should be aligned with normal mode, described by the following code.
// server.go
package main

import (
        "log"
        "net/http"
        "time"
)

type handler struct{}

func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
        time.Sleep(2 * time.Second) // handle some logic
        w.Write([]byte("oops"))
}

func main() {
        s := &http.Server{
                Addr:         "<IP_ADDRESS>:9999",
                Handler:      &handler{},
                WriteTimeout: 1 * time.Second,
        }
        log.Fatal(s.ListenAndServe())
}

// bash shell
➜  /tmp go run client.go
panic: Get "http://<IP_ADDRESS>:9999": EOF

goroutine 1 [running]:
main.main()
        /tmp/client.go:12 +0x125
exit status 2

What did you expect to see?
I expected to see EOF error
What did you see instead?
I saw that malformed HTTP status.
cc @zhuangqh @Ace-Tang

We have bothered by this bug for a long time.
After dig into code I found that when response writing headers, it ultimately using FD.Write which then called pollDesc.prepareWrite where to check if writer connection has expired. but when response writing body it using io.Copy which ultimately called TCPConn.readFrom who using sendFile to transfer file but don't check if the connection has expired. That's how come the http headers don't transfer to client but the body are.
I think we should invoke dstFD.pd.prepare to check if the connection has expired at:
https://github.com/golang/go/blob/ecf4ebf10054f70e51a0ce759b2ae91aa4febd1a/src/internal/poll/sendfile_linux.go#L20

We have bothered by this bug for a long time.
After dig into code I found that when response writing headers, it ultimately using FD.Write which then called pollDesc.prepareWrite where to check if writer connection has expired. but when response writing body it using io.Copy which ultimately called TCPConn.readFrom who using sendFile to transfer file but don't check if the connection has expired. That's how come the http headers don't transfer to client but the body are.
I think we should invoke dstFD.pd.prepare to check if the connection has expired at:
https://github.com/golang/go/blob/ecf4ebf10054f70e51a0ce759b2ae91aa4febd1a/src/internal/poll/sendfile_linux.go#L20

Maybe the patch of "  w.cw.flush() " is better, when the Head failed to send ,  shoud just cancel whole respose procedure，for a body withoud Head is malformed of HTTP protocal at all。

Maybe the patch of "  w.cw.flush() " is better, when the Head failed to send ,  shoud just cancel whole respose procedure，for a body withoud Head is malformed of HTTP protocal at all。

@dmitshur @toothrot @cagedmantis This issue might be worthy of a backport, so we may want to consider adding CL 285914 to 1.16 even though rc1 is out.  I'd like @neild to take a look at the test, though.  Thanks.

@dmitshur @toothrot @cagedmantis This issue might be worthy of a backport, so we may want to consider adding CL 285914 to 1.16 even though rc1 is out.  I'd like @neild to take a look at the test, though.  Thanks.

Roger that, I've added feedback to improve the test :-)

Roger that, I've added feedback to improve the test :-)

@odeke-em, Thanks! I updated it and please take a look. Thank you~

@neild Are you able to take a look at the latest comments in CL 285914, since there's little time left in the Go1.16 milestone? Thanks.

As per the comment by Ian in https://github.com/golang/go/issues/43822#issuecomment-768639142, this issue is a worthy candidate. @gopherbot please backport this issue.


chore: [M3-8262] - Cleanup eventMessages flag and legacy code

Description 📝
This PR is a follow up to EventMessages v2 implementation. Now that the flag has been on in production and no major (or minor for that regard) bugs has been reported, it's time to clean up the legacy code and ordganize the code around the features EventMessages v2 introduced and implemented.
This refactor was facilited by targeting all instances of TODO eventMessagesV2 left in the codebase to this purpose.
This PR does not modify behavior or styling in any way (read shouldn't), it only reorganizes the code around the cleanup.
Changes  🔄

Delete eventMessagesV2 flag
Delete Events and NotificationCenter legacy code
Update NotificationCenter structure/naming conventions
Update tests (unit & e2e)

Preview 📷
Include a screenshot or screen recording of the change
:bulb: Use <video src="" /> tag when including recordings in table.

NotificationCenter
Events

How to test 🧪
Verify that no regression happens as a result of this cleanup
As an Author I have considered 🤔
Check all that apply

[x] 👀 Doing a self review
[ ] ❔ Our contribution guidelines
[ ] 🤏 Splitting feature into small PRs
[x] ➕ Adding a changeset
[x] 🧪 Providing/Improving test coverage
[ ] 🔐 Removing all sensitive information from the code and PR description
[ ] 🚩 Using a feature flag to protect the release
[ ] 👣 Providing comprehensive reproduction steps
[ ] 📑 Providing or updating our documentation
[ ] 🕛 Scheduling a pair reviewing session
[ ] 📱 Providing mobile support
[ ] ♿  Providing accessibility support

Long, but clear and consistent, component names.

@mjac0bs yeah i struggled a bit with a decent convention but "Notification" in itself isn't conveying enough context. My take was that the names here makes it easier to find references for contributors in their IDE, and by compile time/minification the name length matters little.


ci: add checks to publish workflow

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (7444005) 100.00% compared to head (9f75d72) 100.00%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff            @@
##            master        #6   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            4         4
  Lines           69        69
  Branches        11        11
=========================================
  Hits            69        69

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Kotlin | Screenshots in HTML

Screenshots in the HTML report would be nice.
https://github.com/gojuno/composer

viewing screenshots on FTL is sufficient for now.


feat: Twitter Login

Added the possibility to login via Twitter.
⚠I HAVE ADDED A FOLDER IN THE DOCUMENTS TO BETTER ORGANIZE THE FILES⚠
Bugs:

Twitter icon is not visible.
It does not link the account to the existing one.

Change Type

[x] New feature (non-breaking change which adds functionality)
[x] Documentation update

Testing
test if the twitter login works correctly
Test Configuration:
I've had some issues because Twitter does NOT allow the use of localhost, so I tested on a private domain to see if it worked.
Checklist

[x] My code adheres to this project's style guidelines
[x] I have performed a self-review of my own code
[x] I have commented in any complex areas of my code
[x] I have made pertinent documentation changes
[x] My changes do not introduce new warnings
[x] I have written tests demonstrating that my changes are effective or that my feature works
[x] Local unit tests pass with my changes
[x] Any changes dependent on mine have been merged and published in downstream modules.

I've read some documentation both on Twitter and on Twitter's Passport.js, and it seems that other people are also facing the same issue with receiving the email... I believe you need to request the app to be whitelisted and have a paid API, which I don't think anyone wants to do😅. Here are a few links to better understand.
GET account/verify_credentials, Github Issue 1, Github Issue 2, stack overflow and Twitter Forum

I will close this PR since no one is responding to the question I asked on the Twitter Developer Community. Therefore, it is impossible to proceed

I will close this PR since no one is responding to the question I asked on the Twitter Developer Community. Therefore, it is impossible to proceed

I will close this PR since no one is responding to the question I asked on the Twitter Developer Community. Therefore, it is impossible to proceed

sorry to hear :(


Alter contexts schema

this PR adjust the metric contexts schema to be not-null, with an empty context document as the default

closing this -- with some testing, it appears performance is slightly better in Postgres but considerably worse in Sqlite -- on net, best to keep context as nullable field


Confusion about weight sharing

Hi，thanks for your great repo. I'm a little confused about why the left and right branches share weights during training. How can the same network extract different features from two different views?  Looking forward to your reply. Thank you very much!

I am also confused about this.


Use bazel version of go for artifacts creation

And all scripts that calls go directly.

resolved by the mono repo.