[SPARK-27388][SQL] expression encoder for objects defined by properties

What changes were proposed in this pull request?
This PR adds expression encoders for beans, java.util.List, java.util.Map and java enum. This addition make it possible to encode Avro objects.
The Beans are objects defined by properties; A property is defined by a setter and a getter functions where the getter return type is equal to the setter unique parameter type and the getter and setter functions have the same name; if the getter name is prefixed by "get" then the setter name must be prefixed by "set"; see tests for bean examples.
Avro objects are beans and thus we can create an expression encoder for avro objects as follows:
implicit val exprEncoder = ExpressionEncoder[Foo]()

All avro types, including fixed types, and excluding complex union types, are suppported by this addition.
The avro fixed types are beans  with exactly one property: bytes.
Currently complex avro unions are not supported because a complex union is declared as Object and there cannot be an expression encoder for Object type (need to use a custom serializer like kryo for example)
How was this patch tested?
currently only 1 encodeDecodeTest was added to ExpressionEncoderSuite; the test uses an avro object with map, array and fixed fields, as in the test 3 (below).
I used the modified spark-sql package in a local project to test it
the tests are as follows (where Barcode is a large Avro object):

test with simple beans

class Bar {
  private var bar$: String = _
  def bar(value: String): Unit = {
    bar$ = value
  }
  def bar(): String = bar$
  override def toString() = { s"Bar($bar)" }
}

class Foo extends Bar {
  var a: Int = _
  var b: Bar = _

  def getA() = a

  def setA(x: Int) {
    a = x
  }
  def getB(): Bar = b

  def setB(x: Bar) {
    b = x
  }
  override def toString() = { s"Foo($a,$b, ${bar()})" }
}

    implicit val encoderFoo = ExpressionEncoder[Foo]
    val bar = new Bar
    bar.bar("ok")
    val a = new Foo
    a.setA(55)
    a.setB(bar)
    a.bar("BAR")
    val ds = List(a).toDS()
    println(ds.collect().toList)

    val df = List(a).toDF()

    val r = df.collect().foreach(println)

    println(df.schema)

Result =>

List(Foo(55,Bar(ok), BAR))
[[ok],55,BAR]
StructType(StructField(B,StructType(StructField(bar,StringType,true)),true), StructField(A,IntegerType,false), StructField(bar,StringType,true))

test with a large Avro schema (Barcode) with nested avro objects, java enums and arrays:

    implicit val barcodeEncoder = ExpressionEncoder[Barcode]()

    val ds: Dataset[Barcode] = 0.until(10000).map(i => {
      val crpAddDesc = new java.util.ArrayList[CrpAddDesc]()
      crpAddDesc.add(CrpAddDesc.newBuilder()
        .setCrpAddEngDesc(s"Crp description1 $i")
        .build())
      crpAddDesc.add(CrpAddDesc.newBuilder()
        .setCrpAddEngDesc(s"Crp description2 $i")
        .build())
      val crpAttrs = new java.util.ArrayList[CrpAttributes]()
      crpAttrs.add(CrpAttributes.newBuilder()
        .setCrpCode(s"crp attr1 $i")
        .setCrpAddDesc(crpAddDesc)
        .build())
      crpAttrs.add(CrpAttributes.newBuilder()
        .setCrpCode(s"crp attr2 $i")
        .build())
      val barcode = Barcode.newBuilder()
        .setBarcode(s"Bar$i")
        .setPrdTaxVal(Money.newBuilder()
          .setUnscaledAmount(i.toLong)
          .setScale(0)
          .setCurrency(Currency.EUR)
          .setCurrencyAlphaCode("EUR")
          .build())
        .setCrpAttributes(crpAttrs)
        .build()
      barcode
    })
      .toDS()

    val x = ds.map(a => {
      (
        a.getBarcode,
        a.getPrdTaxVal.getCurrency,
        a.getCrpAttributes.get(0).getCrpCode,
        a.getCrpAttributes.get(1).getCrpCode)
    })

    println(x.collect().toList.drop(100).head)

result => (Bar100,EUR,crp attr1 100,crp attr2 100)

test with Avro schema having map, array and fixed types

    implicit val logEncoder = ExpressionEncoder[Log]()
    val ds: Dataset[Log] = List(Log.newBuilder()
      .setIps(List("<IP_ADDRESS>", "<IP_ADDRESS>").asJava)
      .setAdditional(Map(
        "foo" -> new java.lang.Integer(1),
        "bar" -> new java.lang.Integer(2)).asJava)
      .setTimestamp("12345678")
      .setMessage("test map")
      .setMagic(new Magic("magic".getBytes))
      .build())
      .toDS()

    println(ds.collect().toList)

result: List({"ips": ["<IP_ADDRESS>", "<IP_ADDRESS>"], "timestamp": "12345678", "message": "test map", "magic": [109, 97, 103, 105, 99], "additional": {"foo": 1, "bar": 2}})

Log Avro schema:
{"namespace": "example.avro", "type": "record", "name": "Log",
         "fields": [ {"name": "ips", "type": {"type": "array", "items": "string"}},
             {"name": "timestamp", "type": "string"},
             {"name": "message", "type": "string"},
             {"name": "magic", "type": {"type": "fixed", "name": "Magic", "size": 4}},
             {"name": "additional", "type": {"type": "map", "values": "int"}} ] }

Barcode Avro schema:
{"type":"record","name":"Barcode","namespace":"referential.product.v2","doc":"Barcode Pivot","fields":[{"name":"barcode","type":["null","string"],"doc":"Barcode of the product","default":null},{"name":"transBarcode","type":["null","string"],"doc":"Origin EAN13","default":null},{"name":"barcodeType","type":["null","string"],"doc":"Barcode type","default":null},{"name":"itemKey","type":["null","string"],"doc":"Internal Code of the Product","default":null},{"name":"itemDesc","type":["null","string"],"doc":"Description of the product","default":null},{"name":"itemDesc18","type":["null","string"],"doc":"Description of the product (18c) on the ticket","default":null},{"name":"mainItemKey","type":["null","string"],"doc":"Item code of the parent product (ifexists)","default":null},{"name":"mainBarcode","type":["null","string"],"doc":"Main Barcode of the product","default":null},{"name":"mainSupplierKey","type":["null","string"],"doc":"Identifier of the Main Supplier","default":null},{"name":"supplierKey","type":["null","string"],"doc":"Identifier of the supplier","default":null},{"name":"itemType","type":["null","string"],"doc":"Type of item : [D] Direct, [L] Logistique, [M] Mix , [XD] Cross-docking, [PCK] Picking, [FT] Flow Through, [VFT] Virtual Flow Through [HD] Home Delivery","default":null},{"name":"brandTypeKey","type":["null","string"],"doc":"ID for the BrandType (MD,MN,PP)(seevalues)","default":null},{"name":"brandTypeDesc","type":["null","string"],"doc":"Description for the Brand Type","default":null},{"name":"createDate","type":["null","string"],"doc":"Creation date","default":null},{"name":"updateDate","type":["null","string"],"doc":"Update date.(YYYYMMDD)","default":null},{"name":"deleteDate","type":["null","string"],"doc":"Delete Date","default":null},{"name":"nonActiveDate","type":["null","string"],"doc":"Non active date(YYYYMMDD)","default":null},{"name":"prdStopFlag","type":["null","string"],"doc":"Flag STOP PRODUC T(for the sales and the purchase)","default":null},{"name":"salStopFlag","type":["null","string"],"doc":"Flag STOP ITEM on sales","default":null},{"name":"purStopFlag","type":["null","string"],"doc":"Flag STOP ITEM on purchase","default":null},{"name":"prdVatPct","type":["null","double"],"doc":"VAT percentage","default":null},{"name":"prdTaxVal","type":["null",{"type":"record","name":"Money","namespace":"common.lib.v1","fields":[{"name":"unscaledAmount","type":"long","default":0},{"name":"scale","type":"int","default":0},{"name":"currency","type":{"type":"enum","name":"Currency","symbols":["EUR","USD","BRL"]},"default":"EUR"},{"name":"currencyAlphaCode","type":["null","string"],"default":null}]}],"doc":"Other taxes amount","default":null},{"name":"prdVigVal","type":["null","common.lib.v1.Money"],"doc":"Vignette amount","default":null},{"name":"prdDutyVal","type":["null","common.lib.v1.Money"],"doc":"Duty amount","default":null},{"name":"prdUnitType","type":["null","string"],"doc":"Reference nature code","default":null},{"name":"frRacCode","type":["null","string"],"doc":"RAC code of the product (ANABEL2)","default":null},{"name":"npceBaseRist","type":["null","common.lib.v1.Money"],"doc":"Net price without tax for margin computation","default":null},{"name":"frTheoSrpPceOrig","type":["null","common.lib.v1.Money"],"doc":"Theoretical Break Even Price (SRP) without tax","default":null},{"name":"frTheoSrpPceMod","type":["null","common.lib.v1.Money"],"doc":"Theoretical Break Even Price (SRP) updated by RSRP+RFA","default":null},{"name":"structHyp","type":["null",{"type":"record","name":"ProductStructHyp","doc":"Product HyperMarket Structure","fields":[{"name":"hypUbKey","type":["null","string"],"doc":"Identifier of the UB (HyperMarket)","default":null},{"name":"hypUbDesc","type":["null","string"],"doc":"Description of the UB (HyperMarket)","default":null},{"name":"hypSubClassKey","type":["null","string"],"doc":"Identifier of the Sub Class (HyperMarket)","default":null},{"name":"hypSubClassDesc","type":["null","string"],"doc":"Description of the Sub Class (HyperMarket)","default":null},{"name":"hypClassKey","type":["null","string"],"doc":"Identifier of the Class (HyperMarket)","default":null},{"name":"hypClassDesc","type":["null","string"],"doc":"Description of the Class (HyperMarket)","default":null},{"name":"hypGrpClassKey","type":["null","string"],"doc":"Identifier of the Group Class (HyperMarket)","default":null},{"name":"hypGrpClassDesc","type":["null","string"],"doc":"Description of the Group Class (HyperMarket)","default":null},{"name":"hypDepartmentKey","type":["null","string"],"doc":"Identifier of the Department (HyperMarket)","default":null},{"name":"hypDepartmentDesc","type":["null","string"],"doc":"Description of the Department (HyperMarket)","default":null},{"name":"hypSectorKey","type":["null","string"],"doc":"Identifier of the Sector (HyperMarket)","default":null},{"name":"hypSectorDesc","type":["null","string"],"doc":"Description of the Sector (HyperMarket)","default":null}]}],"doc":"Structure Hyper","default":null},{"name":"structSup","type":["null",{"type":"record","name":"ProductStructSup","doc":"Product SuperMarket Structure","fields":[{"name":"supUbKey","type":["null","string"],"doc":"Identifier of the UB (SuperMarket)","default":null},{"name":"supUbDesc","type":["null","string"],"doc":"Description of the UB (SuperMarket)","default":null},{"name":"supSubClassKey","type":["null","string"],"doc":"Identifier of the Sub Class (SuperMarket)","default":null},{"name":"supSubClassDesc","type":["null","string"],"doc":"Description of the Sub Class (SuperMarket)","default":null},{"name":"supClassKey","type":["null","string"],"doc":"Identifier of the Class (SuperMarket)","default":null},{"name":"supClassDesc","type":["null","string"],"doc":"Description of the Class (SuperMarket)","default":null},{"name":"supGrpClassKey","type":["null","string"],"doc":"Identifier of the Group Class (SuperMarket)","default":null},{"name":"supGrpClassDesc","type":["null","string"],"doc":"Description of the Group Class (SuperMarket)","default":null},{"name":"supDepartmentKey","type":["null","string"],"doc":"Identifier of the Department (SuperMarket)","default":null},{"name":"supDepartmentDesc","type":["null","string"],"doc":"Description of the Department (SuperMarket)","default":null},{"name":"supSectorKey","type":["null","string"],"doc":"Identifier of the Sector (SuperMarket)","default":null},{"name":"supSectorDesc","type":["null","string"],"doc":"Description of the Sector (SuperMarket)","default":null}]}],"doc":"Structure Super","default":null},{"name":"structPrx","type":["null",{"type":"record","name":"ProductStructPrx","doc":"Product Proxi Structure","fields":[{"name":"prxUbKey","type":["null","string"],"doc":"Identifier of the UB (Proxi)","default":null},{"name":"prxUbDesc","type":["null","string"],"doc":"Description of the UB (Proxi)","default":null},{"name":"prxSubClassKey","type":["null","string"],"doc":"Identifier of the Sub Class (Proxi)","default":null},{"name":"prxSubClassDesc","type":["null","string"],"doc":"Description of the Sub Class (Proxi)","default":null},{"name":"prxClassKey","type":["null","string"],"doc":"Identifier of the Class (Proxi)","default":null},{"name":"prxClassDesc","type":["null","string"],"doc":"Description of the Class (Proxi)","default":null},{"name":"prxGrpClassKey","type":["null","string"],"doc":"Identifier of the Group Class (Proxi)","default":null},{"name":"prxGrpClassDesc","type":["null","string"],"doc":"Description of the Group Class (Proxi)","default":null},{"name":"prxHahDepartmentKey","type":["null","string"],"doc":"Identifier of the Department (Huit a Huit)","default":null},{"name":"prxHahDepartmentDesc","type":["null","string"],"doc":"Description of the Department (Huit a Huit)","default":null},{"name":"prxHahSectorKey","type":["null","string"],"doc":"Identifier of the Sector (Huit a Huit)","default":null},{"name":"prxHahSectorDesc","type":["null","string"],"doc":"Description of the Sector (Huit a Huit)","default":null},{"name":"prxShoDepartmentKey","type":["null","string"],"doc":"Identifier of the Department (Shopi)","default":null},{"name":"prxShoDepartmentDesc","type":["null","string"],"doc":"Description of the Department (Shopi)","default":null},{"name":"prxShoSectorKey","type":["null","string"],"doc":"Identifier of the Sector (Shopi)","default":null},{"name":"prxShoSectorDesc","type":["null","string"],"doc":"Description of the Sector (Shopi)","default":null}]}],"doc":"Structure Proxi","default":null},{"name":"structPrs","type":["null",{"type":"record","name":"ProductStructPrs","doc":"Product Proxi Structure (Shopi)","fields":[{"name":"prsUbKey","type":["null","string"],"doc":"Identifier of the UB (Shopi)","default":null},{"name":"prsUbDesc","type":["null","string"],"doc":"Description of the UB (Shopi)","default":null},{"name":"prsSubClassKey","type":["null","string"],"doc":"Identifier of the Sub Class (Shopi)","default":null},{"name":"prsSubClassDesc","type":["null","string"],"doc":"Description of the Sub Class (Shopi)","default":null},{"name":"prsClassKey","type":["null","string"],"doc":"Identifier of the Class (Shopi)","default":null},{"name":"prsClassDesc","type":["null","string"],"doc":"Description of the Class (Shopi)","default":null},{"name":"prsGrpClassKey","type":["null","string"],"doc":"Identifier of the Group Class (Shopi)","default":null},{"name":"prsGrpClassDesc","type":["null","string"],"doc":"Description of the Group Class (Shopi)","default":null},{"name":"prsDepartmentKey","type":["null","string"],"doc":"Identifier of the Department (Shopi)","default":null},{"name":"prsDepartmentDesc","type":["null","string"],"doc":"Description of the Department (Shopi)","default":null},{"name":"prsSectorKey","type":["null","string"],"doc":"Identifier of the Sector (Shopi)","default":null},{"name":"prsSectorDesc","type":["null","string"],"doc":"Description of the Sector (Shopi)","default":null}]}],"doc":"Product Proxi Structure (Shopi)","default":null},{"name":"structPrh","type":["null",{"type":"record","name":"ProductStructPrh","doc":"Product Proxi Structure (8a8)","fields":[{"name":"prhUbKey","type":["null","string"],"doc":"Identifier of the UB (8a8)","default":null},{"name":"prhUbDesc","type":["null","string"],"doc":"Description of the UB (8a8)","default":null},{"name":"prhSubClassKey","type":["null","string"],"doc":"Identifier of the Sub Class (8a8)","default":null},{"name":"prhSubClassDesc","type":["null","string"],"doc":"Description of the Sub Class (8a8)","default":null},{"name":"prhClassKey","type":["null","string"],"doc":"Identifier of the Class (8a8)","default":null},{"name":"prhClassDesc","type":["null","string"],"doc":"Description of the Class (8a8)","default":null},{"name":"prhGrpClassKey","type":["null","string"],"doc":"Identifier of the Group Class (8a8)","default":null},{"name":"prhGrpClassDesc","type":["null","string"],"doc":"Description of the Group Class (8a8)","default":null},{"name":"prhDepartmentKey","type":["null","string"],"doc":"Identifier of the Department (8a8)","default":null},{"name":"prhDepartmentDesc","type":["null","string"],"doc":"Description of the Department (8a8)","default":null},{"name":"prhSectorKey","type":["null","string"],"doc":"Identifier of the Sector (8a8)","default":null},{"name":"prhSectorDesc","type":["null","string"],"doc":"Description of the Sector (8a8)","default":null}]}],"doc":"Product Proxi Structure (8a8)","default":null},{"name":"structPmc","type":["null",{"type":"record","name":"ProductStructPmc","doc":"Product PromoCash Structure","fields":[{"name":"pmcUbKey","type":["null","string"],"doc":"Identifier of the UB (PromoCash)","default":null},{"name":"pmcUbDesc","type":["null","string"],"doc":"Description of the UB (PromoCash)","default":null},{"name":"pmcSubClassKey","type":["null","string"],"doc":"Identifier of the Sub Class (PromoCash)","default":null},{"name":"pmcSubClassDesc","type":["null","string"],"doc":"Description of the Sub Class (PromoCash)","default":null},{"name":"pmcClassKey","type":["null","string"],"doc":"Identifier of the Class (PromoCash)","default":null},{"name":"pmcClassDesc","type":["null","string"],"doc":"Description of the Class (PromoCash)","default":null},{"name":"pmcDepartmentKey","type":["null","string"],"doc":"Identifier of the Department (PromoCash)","default":null},{"name":"pmcDepartmentDesc","type":["null","string"],"doc":"Description of the Department (PromoCash)","default":null},{"name":"pmcSectorKey","type":["null","string"],"doc":"Identifier of the Sector (PromoCash)","default":null},{"name":"pmcSectorDesc","type":["null","string"],"doc":"Description of the Sector (PromoCash)","default":null}]}],"doc":"Structure PromoCash","default":null},{"name":"sectorKey","type":["null","string"],"doc":"Unique ID for Sector (Product Sector)","default":null},{"name":"sectorDesc","type":["null","string"],"doc":"Sector description","default":null},{"name":"departmentKey","type":["null","string"],"doc":"Unique ID of the department (Product Department)","default":null},{"name":"departmentDesc","type":["null","string"],"doc":"Department description","default":null},{"name":"grpClassKey","type":["null","string"],"doc":"Unique ID for Group Class (Product Group Family)","default":null},{"name":"grpClassDesc","type":["null","string"],"doc":"Group Class (Product Group Family) description","default":null},{"name":"classKey","type":["null","string"],"doc":"Unique ID for Class (Product Family)","default":null},{"name":"classDesc","type":["null","string"],"doc":"Class (Product Family) description","default":null},{"name":"subClassKey","type":["null","string"],"doc":"Unique ID of for product sub-class (Sub-Family)","default":null},{"name":"subClassDesc","type":["null","string"],"doc":"product sub-class (Sub-Family) description","default":null},{"name":"ubKey","type":["null","string"],"doc":"Unique ID for Unite Besoin (Product UB Family)","default":null},{"name":"ubDesc","type":["null","string"],"doc":"Description for level6 in French Unité de Besoin","default":null},{"name":"ubDescShort","type":["null","string"],"doc":"Short Description of the UB","default":null},{"name":"sizeKey","type":["null","string"],"doc":"Size code","default":null},{"name":"sizeDesc","type":["null","string"],"doc":"size description","default":null},{"name":"colorKey","type":["null","string"],"doc":"Color code","default":null},{"name":"colorDesc","type":["null","string"],"doc":"Color description","default":null},{"name":"prdCapaType","type":["null","string"],"doc":"Capacity unit code","default":null},{"name":"prdCapaVolume","type":["null","double"],"doc":"Capacity","default":null},{"name":"prdCapaFactor","type":["null","double"],"doc":"Number of lot components","default":null},{"name":"prdUnitDesc","type":["null","string"],"doc":"Reference nature description","default":null},{"name":"prdCapaDesc","type":["null","string"],"doc":"Capacity unit description","default":null},{"name":"prdTechNature","type":["null","string"],"doc":"Technical nature of the product","default":null},{"name":"prdLotHomoFlag","type":["null","string"],"doc":"Homogeneous lot indicator","default":null},{"name":"prdLotQtyFree","type":["null","double"],"doc":"Free Quantity in the lot","default":null},{"name":"prdLotAmtFree","type":["null","common.lib.v1.Money"],"doc":"Free Amount in the lot","default":null},{"name":"prdLotQtyTotal","type":["null","double"],"doc":"Total Quantity in the lot","default":null},{"name":"prdQtyFree","type":["null","double"],"doc":"Free Quantity in the product","default":null},{"name":"prdQtyTotal","type":["null","double"],"doc":"Total Quantity in the product","default":null},{"name":"prdEan7Prix","type":["null","string"],"doc":"EAN7 prix","default":null},{"name":"prdEan7Poids","type":["null","string"],"doc":"EAN7 poids","default":null},{"name":"prdVarWeightFlag","type":["null","string"],"doc":"Variable measure indicator","default":null},{"name":"catSubManagerKey","type":["null","string"],"doc":"First level of Category Management Hierarchy","default":null},{"name":"bemSubClassKey","type":["null","string"],"doc":"A unique identifier of a product BEM sub-class","default":null},{"name":"prxUbKey","type":["null","string"],"doc":"Identifier of the UB (Proxi)","default":null},{"name":"pdrUbKey","type":["null","string"],"doc":"Code UB Prodirest","default":null},{"name":"pmcUbKey","type":["null","string"],"doc":"Identifier of the UB (PromoCash)","default":null},{"name":"brandOwnerCode","type":["null","string"],"doc":"External Code of the brand owner","default":null},{"name":"brandOwnerKey","type":["null","string"],"doc":"Identifier of the brand owner","default":null},{"name":"brandOwnerDesc","type":["null","string"],"doc":"Description of the brand owner","default":null},{"name":"brandKey","type":["null","string"],"doc":"Brand code","default":null},{"name":"brandDesc","type":["null","string"],"doc":"Brand description","default":null},{"name":"brandSubTypeKey","type":["null","string"],"doc":"Sub-qualification key","default":null},{"name":"brandSubTypeDesc","type":["null","string"],"doc":"Sub-qualification Description","default":null},{"name":"dispoKey","type":["null","string"],"doc":"Disponibility code","default":null},{"name":"dispoDesc","type":["null","string"],"doc":"Disponibility description","default":null},{"name":"negoOrigKey","type":["null","string"],"doc":"Identifier of the origin of the négociation","default":null},{"name":"negoOrigDesc","type":["null","string"],"doc":"Description of the origin of the négociation","default":null},{"name":"pceLevelType","type":["null","string"],"doc":"Price Level type","default":null},{"name":"prdCompoDesc","type":["null","string"],"doc":"Description ot the product composition","default":null},{"name":"modelBarcode","type":["null","string"],"doc":"Barcode of the model (textile)","default":null},{"name":"modelDesc","type":["null","string"],"doc":"Description of the model (textile)","default":null},{"name":"classDescShort","type":["null","string"],"doc":"Class (Product Family) short description","default":null},{"name":"mainSupplierPrcKey","type":["null","string"],"doc":"Identifier of the Main Supplier for the previous exercise","default":null},{"name":"purStopDate","type":["null","string"],"doc":"Date of STOP ITEM on purchase","default":null},{"name":"bcpBrdTypeKey","type":["null","string"],"doc":"identifier of the qualification in BCP","default":null},{"name":"bcpBrdTypeDesc","type":["null","string"],"doc":"Description of the qualification in BCP","default":null},{"name":"bcpBrdQualifKey","type":["null","string"],"doc":"identifier of the brand qualification in BCP","default":null},{"name":"bcpBrdQualifDesc","type":["null","string"],"doc":"Description of the brand qualification in BCP","default":null},{"name":"bcpBrdSubQualifKey","type":["null","string"],"doc":"Identifier of the brand sub-qualification in BCP","default":null},{"name":"bcpBrdSubQualifDesc","type":["null","string"],"doc":"Description of the brand sub-qualification in BCP","default":null},{"name":"bcpHypAppelPrixCode","type":["null","string"],"doc":"National POS Price short code for Hypermarket","default":null},{"name":"bcpSupAppelPrixCode","type":["null","string"],"doc":"National POS Price short code for Supermarket","default":null},{"name":"supUgKey","type":["null","string"],"doc":"Caroline Super Stock Keeping Unit (UG) Key","default":null},{"name":"supUgDesc","type":["null","string"],"doc":"Caroline Super Stock Keeping Unit (UG) Description","default":null},{"name":"prdRefNature","type":["null","string"],"doc":"Product reference nature","default":null},{"name":"prdModKey","type":["null","string"],"doc":"Code of manufacturing mode","default":null},{"name":"prdModDesc","type":["null","string"],"doc":"Description of manufacturing mode","default":null},{"name":"productLineDesc","type":["null","string"],"doc":"Product Line Description","default":null},{"name":"brandMasterKey","type":["null","string"],"doc":"Identifier of the Master Brand","default":null},{"name":"brandMasterDesc","type":["null","string"],"doc":"Description of the Master Brand","default":null},{"name":"prdOrigOpKey","type":["null","string"],"doc":"Operational Identifier of the origin of the product","default":null},{"name":"prdOrigOpDesc","type":["null","string"],"doc":"Operational Description of the origin of the product","default":null},{"name":"mbcItemKey","type":["null","string"],"doc":"MBC Atica Hyper Stock Keeping Unit (AG) Key","default":null},{"name":"hypTypoKey","type":["null","string"],"doc":"Product Typology for Hypermarkets","default":null},{"name":"supTypoKey","type":["null","string"],"doc":"Product Typology for Supermarkets","default":null},{"name":"prxTypoKey","type":["null","string"],"doc":"Product Typology for Proximity Stores","default":null},{"name":"cacTypoKey","type":["null","string"],"doc":"Product Typology for Cash & Carry Stores","default":null},{"name":"prdPackFlag","type":["null","string"],"doc":"Divisible product pack flag","default":null},{"name":"trendCode","type":["null","string"],"doc":"Code of textile classification","default":null},{"name":"trendDesc","type":["null","string"],"doc":"Description of textile classification","default":null},{"name":"prdSeasonCode","type":["null","string"],"doc":"Code of the Season","default":null},{"name":"mbcSubClassKey","type":["null","string"],"doc":"MBC sub family","default":null},{"name":"mbcSizeCode","type":["null","string"],"doc":"Size of the article","default":null},{"name":"mbcColorCode","type":["null","string"],"doc":"Color of the article","default":null},{"name":"ubcKey","type":["null","string"],"doc":"Identifier of the Consumer Unit of Need (UBC OptiCAT)","default":null},{"name":"ubcDesc","type":["null","string"],"doc":"Description of the Consumer Unit of Need (UBC OptiCAT)","default":null},{"name":"hypUgKey","type":["null","string"],"doc":"Caroline Hyper Stock Keeping Unit (UG) Key","default":null},{"name":"hypUgDesc","type":["null","string"],"doc":"Caroline Hyper Stock Keeping Unit (UG) Description","default":null},{"name":"prxUgKey","type":["null","string"],"doc":"Proxy Stock Keeping Unit (UG) Key","default":null},{"name":"prxUgDesc","type":["null","string"],"doc":"Proxy Stock Keeping Unit (UG) Description","default":null},{"name":"pmcUgKey","type":["null","string"],"doc":"PromoCash Stock Keeping Unit (UG) Key","default":null},{"name":"pmcUgDesc","type":["null","string"],"doc":"PromoCash Stock Keeping Unit (UG) Description","default":null},{"name":"dctHypDirectFlag","type":["null","string"],"doc":"Indicates that permanent product should use Direct Flow in HyperMarket","default":null},{"name":"dctSupDirectFlag","type":["null","string"],"doc":"Indicates that permanent product should use Direct Flow in SuperMarket","default":null},{"name":"crpAttributes","type":["null",{"type":"array","items":{"type":"record","name":"CrpAttributes","doc":"CRP Product characteristics Attributes","fields":[{"name":"crpKey","type":["null","string"],"doc":"CRP Key","default":null},{"name":"crpCode","type":["null","string"],"doc":"CRP external code","default":null},{"name":"crpValueBegDate","type":["null","string"],"doc":"CRP value begin date YYYYMMDD","default":null},{"name":"crpValueEndDate","type":["null","string"],"doc":"CRP value finish date YYYYMMDD","default":null},{"name":"catCrpKey","type":["null","string"],"doc":"CRP category key","default":null},{"name":"catCrpDesc","type":["null","string"],"doc":"CRP category descrpition","default":null},{"name":"subCatCrpKey","type":["null","string"],"doc":"CRP subcategory key","default":null},{"name":"subCatCrpDesc","type":["null","string"],"doc":"CRP subcategory descrpition","default":null},{"name":"crpNatDesc","type":["null","string"],"doc":"CRP national description","default":null},{"name":"crpEngDesc","type":["null","string"],"doc":"CRP english description","default":null},{"name":"crpAddDesc","type":["null",{"type":"array","items":{"type":"record","name":"CrpAddDesc","doc":"CRP Product characteristics additionnal description","fields":[{"name":"crpAddNatDesc","type":["null","string"],"doc":"CRP national additional description","default":null},{"name":"crpAddEngDesc","type":["null","string"],"doc":"CRP english additional description","default":null}]}}],"doc":"Additional descriptions","default":null},{"name":"crpDesc","type":["null","string"],"doc":"CRP description","default":null},{"name":"crpValueFormat","type":["null","string"],"doc":"CRP Value format","default":null},{"name":"crpValueLength","type":["null","int"],"doc":"CRP Value length","default":null},{"name":"crpUnitNatSymbol","type":["null","string"],"doc":"Mesurement unit key national symbol","default":null},{"name":"crpUnitEngSymbol","type":["null","string"],"doc":"Mesurement unit key english symbol","default":null},{"name":"crpUnitNatDesc","type":["null","string"],"doc":"Mesurement unit key national description","default":null},{"name":"crpUnitEngDesc","type":["null","string"],"doc":"Mesurement unit key english description","default":null},{"name":"crpUnitNatValue","type":["null","string"],"doc":"CRP national value","default":null},{"name":"crpUnitEngValue","type":["null","string"],"doc":"CRP english value","default":null},{"name":"crpLink","type":["null","string"],"doc":"CRP link between values","default":null},{"name":"crpSourceKey","type":["null","string"],"doc":"CRP code from source","default":null},{"name":"crpSourceDesc","type":["null","string"],"doc":"CRP source description","default":null},{"name":"extDateTime","type":["null","string"],"doc":"extraction timestamp ISO8601 deprecated yyyy-MM-dd''T''HH:mm:ss.SSSZZ use yyyyMMdd''T''HHmmss.SSSZ","default":null}]}}],"doc":"CRP Product characteristics Attributes","default":null},{"name":"hypSalPceUnitType","type":["null","string"],"doc":"Hyper Sales Price level type (UVC, UCT)","default":null},{"name":"supSalPceUnitType","type":["null","string"],"doc":"Super Sales Price level type (UVC, UCT)","default":null},{"name":"prxSalPceUnitType","type":["null","string"],"doc":"Proxi Sales Price level type (UVC, UCT)","default":null},{"name":"pmcSalPceUnitType","type":["null","string"],"doc":"Cash and Carry Sales Price level type (UVC, UCT)","default":null},{"name":"hypPluCode","type":["null","string"],"doc":"Hyper PLU Code","default":null},{"name":"supPluCode","type":["null","string"],"doc":"Super PLU Code","default":null},{"name":"prxPluCode","type":["null","string"],"doc":"Proxi PLU Code","default":null},{"name":"pmcPluCode","type":["null","string"],"doc":"Cash and Carry PLU Code","default":null},{"name":"prdSupplierCode","type":["null","string"],"doc":"Reference of the product by the supplier","default":null},{"name":"prdMarketingDesc","type":["null","string"],"doc":"Marketing description of the product","default":null},{"name":"prdPackaging","type":["null","string"],"doc":"Packaging of the products","default":null},{"name":"prdMarketingSaleUnit","type":["null","string"],"doc":"Type of sale Unit","default":null},{"name":"prdVariableTypePrep","type":["null","string"],"doc":"Type of preparation variable","default":null},{"name":"prdPublishableFlag","type":["null","string"],"doc":"Publishable flag","default":null},{"name":"specialTaxType","type":["null","string"],"doc":"Product Special tax Type","default":null},{"name":"prdDutyType","type":["null","string"],"doc":"Product Duty tax Type","default":null},{"name":"vatCode","type":["null","string"],"doc":"Identier of VAT","default":null},{"name":"prdAlcoholProof","type":["null","string"],"doc":"Product Alcohol Proof","default":null},{"name":"prdShortDesc","type":["null","string"],"doc":"Product short description","default":null},{"name":"productServiceType","type":["null","string"],"doc":"Type of the Service","default":null},{"name":"productServiceDesc","type":["null","string"],"doc":"Description of the service","default":null},{"name":"productPurchaseSaleType","type":["null","string"],"doc":"Type of the product (Purchase/Sale)","default":null},{"name":"prdDryNetWeight","type":["null","double"],"doc":"Dry net weight of the sales unit","default":null},{"name":"prdPackWeight","type":["null","double"],"doc":"Weight of the packaged saled product","default":null},{"name":"prdPackHeight","type":["null","double"],"doc":"Height of the packaged saled product","default":null},{"name":"prdPackWidth","type":["null","double"],"doc":"Width of the packaged saled product","default":null},{"name":"prdPackLength","type":["null","double"],"doc":"Length of the packaged saled product","default":null},{"name":"srcTimestamp","type":["null","string"],"doc":"Source Timestamp","default":null},{"name":"trtTimestamp","type":["null","string"],"doc":"Processing Timestamp","default":null}]}

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Overall: isn't this what the Encoder for Java Beans is for and already supports? I am not sure it obvious to me this should go in ScalaReflection.
Separately I don't think it's worth further supporting JavaBean properties. That's pretty old and rare, I think.

JavaBean do not support Avro fixed types, because the fixed type has one
property and is named bytes; javaBean only accepts properties fixed with
set/get

I believe that the current implementation has a bug, indeed

Line 136
in sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/JavaTypeInference.scala
should be:
val properties = getJavaBeanReadableAndWritableProperties(other) and not
val properties = getJavaBeanReadableProperties(other) 3. toDS, toDF, always
uses the expression encoder I corrected the bug mentioned above and tested
it, by using Encoders.bean, and I had three issues: 1. the ds.map fails
with no encoder found because it is using the ScalaReflection to match for
an encoder 2. the Encoders.bean fails for java enums (assertion fails, not
a StructType since an enum is saved as String) 3. and for Avro fixed types
are not supported because the property in fixed types is not prefixed by
set/get regards Taoufik
On Mon, Apr 8, 2019 at 3:47 PM Sean Owen<EMAIL_ADDRESS>wrote:

Overall: isn't this what the Encoder for Java Beans is for and already
supports? I am not sure it obvious to me this should go in ScalaReflection.
Separately I don't think it's worth further supporting JavaBean
properties. That's pretty old and rare, I think.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_spark_pull_24299-23issuecomment-2D480838251&d=DwMCaQ&c=oYSInPoxeTs4m10ilnGu9Q&r=tJmPZfODPMHyXvtcDPDLh8Y6NivZyWO2Jcs2xoRat90&m=1XuL-1AY_mS1qCc4oZ3D2tNdNu8ktGbSIZBkjEfoOFU&s=03jB8i0A-2N65hOzloj_HlIMEx8b4_TWOuCjXoFMwbg&e=,
or mute the thread
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AG82a0A2aSqET6khk5U0VMPd4vP6-2D5EJks5ve0hjgaJpZM4cdqpS&d=DwMCaQ&c=oYSInPoxeTs4m10ilnGu9Q&r=tJmPZfODPMHyXvtcDPDLh8Y6NivZyWO2Jcs2xoRat90&m=1XuL-1AY_mS1qCc4oZ3D2tNdNu8ktGbSIZBkjEfoOFU&s=UeN37Pqqn63me2GLDt3OxMo7wZ-0G5_MOFDttZIyKQg&e=
.

--
This e-mail and any attachment are confidential and intended solely for the
use of the individual to whom it is addressed. If you are not the intended
recipient, please telephone or email the sender and delete this message and
any attachment from your system. Unauthorized publication, use,
dissemination, forwarding, printing or copying of this e-mail and its
associated attachments is strictly prohibited.
http://disclaimer.carrefour.com/ http://disclaimer.carrefour.com/
Let's
respect the environment together. Only print this message if necessary.

@srowen This PR is mainly about adding support to Avro objects and make our code easier

cc @gengliangwang @cloud-fan

Many thanks to @mazeboard for bringing this PR to my attention and for taking a crack at the problem of typed Avro Datasets in Spark!
I feel it's a matter of due diligence for me to point to another PR supporting Avro-typed Datasets in Spark, namely #22878, which (full transparency) is the work of @xuanyuanking and myself. The approaches taken here and there are different, and it would seem so are the coverage of the Avro spec. I'd like to take the time to compare/contrast.
I am more qualified to speak to the approach and capabilities introduced #22878 (which has a history going back to Spark-Avro), and so if I misread this PR in that process, @mazeboard, please do correct my understanding.
I apologize for the length of this reply ahead of time!
#24299
I'll summarize my reading of this PR's approach: to extend the existing Java Bean Encoder functionality to more broadly support tricky types generated by Avro SpecificRecord classes, especially Bytes, which Avro doesn't allow access to via typical getters/setters, and Enums, which have to be encoded as Strings and therefore have to be “discovered” via their class.
One stated limitation is complex Union types, which Avro will represent as nested Object. It’s stated that there isn’t an Encoder for Object type (I think a serializer/deserializer could perhaps be made using the Object and [NewInstance](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala#L431 , but I can’t say how it would work in this Reflection-driven approach). It’s said things could get tough with how Avro serializes those objects when things like Kyro are used. I can see that as a limitation to this Bean-based approach even if the DataTypes and Expressions were sorted out.
Correct my understanding if this is wrong, but because this approach is based on Reflection over the types of the generated SpecificRecord class (viewing Avro as a Bean), it would not be (in a first-class sense) “aware” of the AvroSchema which generates the Class. I think this distinction may matter, and I’ll discuss it more below.
#22878
To summarize the approach of #22878: create an AvroEncoder which generates a pair of serializer/deserializer Expressions based on the AvroSchema.
This work stands on the Avro efforts that were recently folded into Spark-proper from the (now deprecated) Databricks/Spark-Avro project, but which still does not provide support for a first-class Encoder, and the efficiency and Strong/Static typing that entails. Being based on Spark-Avro however, #22878 does gain benefits of an AvroSchema driven approach. To avoid confusion, I’m going to refer to this folded-in functionality as “Spark-Avro”, relating to this portion of the current Spark project, rather than to the former deprecated project.
AvroSchema
Because #22878 generates its Encoder through the AvroSchema, we gain a couple things:

An isomorphism between the Dataset Schema and the source of truth for the structure of any Avro, namely, its AvroSchema. The important thing here is we can generate an Encoder both from the Class and the AvroSchema in its different representations, like JSON, which opens support for Schema evolution, and use of Schema stores.
Direct support Encoders of any GenericRecord, where an AvroSchema is known, but no SpecificRecord class can be generated.

Coverage
Spark-Avro’s ability to move between AvroSchema and Spark Dataset Schema also gives us the ability to traverse the DataType to create our Encoder’s Ser/De Expressions rather than using Reflection. This gives us two immediate benefits

The qualities of Bytes and Enums are more transparently represented in the AvroSchema, and so the SerDe expressions can be more directly generated.
Nested Records and Complex Unions (as well as Avro Lists and Maps) are a solved-problem in Spark-Avro, so we can generate a Dataset Schema of arbitrary complexity and nesting.
Naturally, we don’t have to extend the Bean Encoder for properties.

These two items mean the AvroEncoder in #22878 can generate an Encoder having full coverage of Avro types, and this coverage of the various combinations of types that can appear when adding nested Records and Unions is well tested in the PR.
Last thoughts
The PR goes a long way in support of Avro while still being very concise, which is definitely advantageous from a maintainability perspective. My concerns with a Reflection-based approach are:

It’s unknown or perhaps would prove difficult to extend support for (especially) Union types, which are very common in Avro and the only way to express Nullability in Avro.
A Reflection-based approach also foregoes Datasets of GenericRecord objects.

Parting words for #22878:

While it’s length (in terms of lines of code) as a feature has been discussed, however, I’d say ultimately it’s very well tested, considering we’re ultimately testing a closed set of type combinations described by AvroSchema, and nested complex types are tested via inductive unit tests. Anecdotally, I’ll say we’ve been using it in a fork of Spark-Avro with great success over exceptionally complicated Schemas.
It fits quite natively in the new Spark-Avro “external” sub-project, and accomplishes the goal: providing an Encoder for Datasets typed by arbitrary Avro (Generic and Specific).

Again, I'm very happy to see where this discussion goes as it evolves (:

This PR is not an extension of the existing Java Bean Encoder: The PR adds support for bean objects, java.util.List, java.util.Map, and java enums to ScalaReflection;  unlike the existing javaBean Encoder, properties can be named without the set/get prefix.
Reminder of Avro types:
primitive types: null, boolean, int, long, float, double, bytes, string
complex types: Records, Enums, Arrays, Maps, Unions, Fixed
All Avro types are supported by this PR (may be a better test including all Avro types is required), including simple union types and excluding complex union types: Simple  Avro unions are [null, type1], all other unions are complex.
this PR supports simple unions but not complex unions for the simple reason that the Avro compiler will generate java code with type Object for all complex unions, and fields with simple unions will be typed as the non-null type of the union.
Currently, the ScalaReflection does not have an encoder for Object type, but we can modify the ScalaReflection to use a default encoder (ie. kryo) for Object type (currently I do not know if this advisable, but why not?).
I do not understand the issue related to being Reflection-driven approach: all common scala objects are encoded using reflection.
I may be wrong, but as I tried to explain in this PR, that we need to add types in ScalaReflection to be able to transform Datasets to other Datasets of embedded Avro types.
As an example the map function transforms the Dataset[A] to a Dataset[(B, C)]
{code}
val r: Dataset[A] = List(makeA).toDS()
val ds: Dataset[(B, C)] = r.map(e => (e.getB, e.getC))
{/code}
The map function will recursively use ScalaReflection to find encoders for B, C types (Do you know if this compiles with #22878 solution? Or does it complain with no encoder found?)
Finally, I did not understand the benefits of AvroSchema driven approach, for me an Avro object is completely defined by its properties (that are derived from the Avro schema); the Avro compiler generates java code with all the properties in the Avro schema.

Thanks @mazeboard for the reply and clarifying the support here for Unions. I've edited my post to just a discussion of Complex Union types for clarity:

unions of two types where Null is not one of the types, or [Unions] having more than two types

I'm inclined to wait a bit for some more direction or questions from the Spark committers.

just a note about the number of lines changes: the java code for the Avro objects used in the test have 1252 lines of code; so if we ignore those lines of code the overall changed lines of code is +288 -17

I would like to give an example to explain why the PR addition
must be in ScalaReflection
The test is done with spark-sql 2.4.0
The first transformation (map), from Dataset[Foo) to Dataset[Bar],
pass without error, but the second transformation, from Dataset[Foo]
to Dataset[(Foo, Bar)] raises an exception, No Encoder found; this
is explained by the fact that the tuple encoder will search recursively
in ScalaReflection and fails to find an encoder for Foo and Bar
import spark.implicits._

implicit val encoderFoo = Encoders.bean[Foo](classOf[Foo])
implicit val encoderBar = Encoders.bean[Bar](classOf[Bar])

val bar = new Bar
bar.setBar("ok")
val a = new Foo
a.setA(55)
a.setB(bar)
val ds = List(a).toDS()
val x = ds.map(_.b)               // Ok
val y = ds.map(x => (x, x.getB))   // Ko - No Encoder found for Bar and Foo

class Bar {
    private var bar$: java.lang.String = _
    def setBar(value: java.lang.String): Unit = {
       bar$ = value
    }
    def getBar(): java.lang.String = bar$
 }

 class Foo extends Bar {
      var a: java.lang.Integer = _
      var b: Bar = _
      def getA() = a
      def setA(x: java.lang.Integer) {a = x}
      def getB(): Bar = b
      def setB(x: Bar) { b = x}
 }

With the PR additon the following code runs without errors
import spark.implicits._

implicit val encoderFoo = ExpressionEncoder[Foo]

val bar = new Bar
bar.setBar("ok")
val a = new Foo
a.setA(55)
a.setB(bar)
val ds = List(a).toDS()
val x = ds.map(_.b)               // Ok
val y = ds.map(x => (x, x.getB))   // Ok

Currently we modified JavaTypeInference to be able to create encoders for Avro objects; we have now two solutions, the one in this PR and another one using Encoders.bean (fewer code changes); which one is better? how to proceed? we should create another PR for the Encoders.bean solution?

we created a new PR (https://github.com/apache/spark/pull/24367) for the modification of the bean encoder (JavaTypeInference)

Overall: isn't this what the Encoder for Java Beans is for and already supports? I am not sure it obvious to me this should go in ScalaReflection.
Separately I don't think it's worth further supporting JavaBean properties. That's pretty old and rare, I think.

Hi, I would like to hear from about the possibility to merge this PR, many developer teams need to have the possibility to work with Datasets of SpecificRecords (ie. Avro), I implemented a version that modifies the bean encoder but I believe that adding encoders to ScalaReflection is the way to go for reason I explained in this PR, please let me know if this can be considered as a viable solution


How to integrate this responsive framework with go router package

I am using a go router package to enhance my router to my flutter project. This is how I implement the responsive framework in go router
routes: [
      ShellRoute(
        builder: (context, state, child) => ResponsiveWrapper.builder(
          ClampingScrollWrapper.builder(context, child),
          maxWidth: 1200,
          minWidth: 480,
          breakpoints: const [
            ResponsiveBreakpoint.resize(480, name: MOBILE),
            ResponsiveBreakpoint.autoScale(600, name: TABLET),
            ResponsiveBreakpoint.resize(1000, name: DESKTOP),
            ResponsiveBreakpoint.autoScale(2460, name: '4K')
          ],
        ),
        routes: [
          GoRoute(
            path: APP_PAGE.home.toPath,
            name: APP_PAGE.home.toName,
            builder: (_, __) => RootPage(),
          ),
          GoRoute(
            path: APP_PAGE.splash.toPath,
            name: APP_PAGE.splash.toName,
            builder: (context, state) => SplashPage(),
          ),
          GoRoute(
            path: APP_PAGE.login.toPath,
            name: APP_PAGE.login.toName,

            // TODO add login page
            builder: (context, state) => SizedBox.shrink(),
          ),
          GoRoute(
            path: APP_PAGE.game.toPath,
            name: APP_PAGE.game.toName,
            builder: (context, state) {
              final questGameKey = state.params['questGameKey'] ?? 'unknown';
              return GamePage(questGameKey: questGameKey);
            },
          ),
        ],
      ),
    ],

The problem will occur when I used a Responsive condition it said there:
"A responsive condition was caught referencing a nonexistent breakpoint. A ResponsiveCondition requires a parent ResponsiveWrapper or remove breakpoint references."
Excpected: The responsive condition must be working because I already use the ResponsiveWrapper in my go router builder.

This is interesting!
I'm not familiar with ShellRoute, is that the new nested navigation API the go_router team implemented?
I've been meaning to learn more, haven't got around yet.

@moshOntong-IT Did you find a solution? Was aboout integrating responsive framework with go_router in a new project.

@moshOntong-IT Did you find a solution? Was aboout integrating responsive framework with go_router in a new project.

Yeah I did
This is how I solve the problem
BlackboltWidget(         title: 'One sound, One word',         routerConfig: goRouter,         builder: (context, child) => ResponsiveWrapper.builder(           ClampingScrollWrapper.builder(context, child!),           maxWidth: 1200,           minWidth: 480,           breakpoints: const [             ResponsiveBreakpoint.resize(480, name: MOBILE),             ResponsiveBreakpoint.autoScale(600, name: TABLET),             ResponsiveBreakpoint.resize(1000, name: DESKTOP),             ResponsiveBreakpoint.autoScale(2460, name: '4K')           ],         ),       ),

@moshOntong-IT What is BlackboltWidget?

@moshOntong-IT What is BlackboltWidget?

Heheh dont worry Blackbolt Widget just a extend of Material Widget. It is my custom widget


vue-server-renderer: entry-server.js init method

What problem does this feature solve?
I propose to add ability to declare method (init) that will be called on server initialization (after server starting and recompilation). Then developer will have possibility to reload other services.
For now I creating vue frontend app with server-side rendering and I want to create backend api server. I want to bundle it together with vue ssr server – I can just import it in my entry-server.js (so it will be successfully bundled) and connect it to my express app, that I can provide via context param, but if I will add that route to my backend api server in export default method, route will be added every time when user requested some from server (and vue-server-renderer call my entry-server.js default export method). Also it will not added after server starting, but only after first request.
While my backend api server is bundled I can transparently call it methods in server-side api provider (on client-side api provider I use axios to make HTTP-requests). Writing 'providers' I mean create-api-*.js files inside vue-hackernews/src/api
I think that it will be very useful.
What does the proposed API look like?
Means code from vue-hackernews template
// server.js
...

const backendRouter = express.Router()
app.use(backendRouter)
renderer.init({ backendRouter })

...

app.get('*', isProd ? render : (req, res) => {
  readyPromise.then(() => render(req, res))
})

// entry-server.js
const backendApi = require('../backend/api')

function init(context) {
  context.backendRouter.use(backendApi);
}

export { init };

export default context => {
  ...
}

I think the API feels a bit confusing (took me a few times to grasp what you are trying to do). In fact, I think it's easier to externalize your backend API from your bundle so that both your bundle and your server code can just require the same backend API directly.


Encrypt/Decrypt large files will cause memory usage spike

First of all, in the README, this line of code in "Async and Streams":
decryptor = [[RNEncryptor alloc] initWithSettings:kRNCryptorAES256Settings
                                         password:@"blah"
                                          handler: ...];

The variable name should be "encryptor".
Secondly, when trying out this code myself, I used a 1.9G huge file to test the memory issue. And the app will always crash due to memory spike.
I profiled it, and found that as the encryption goes, the memory usage keeps going up, never goes down.
After a closer look, I see that it's this call that increases memory usage every time for 48 KB:
-[_NSPlaceholderData initWithBytes:length:copy:deallocator:]

So, I added a symbolic breakpoint for this function, and found that it's called by RNCryptorEngine.m at line 93 via
return [buffer subdataWithRange:NSMakeRange(0, dataOutMoved)];

So, looks like when using the Async and Streams method in ReadMe to encrypt a file, the memory usage will keep going up due to unknown issues related to this line of code.
I'm not quite familiar with RNCryptor yet. Is this a bug of RNCryptor? How can I fix this issue
Thanks a lot!

I believe the Swift version has more reliable memory usage, and no longer has a misleading example.

For anyone coming in here via Google; I solved this problem by applying the semaphore solution suggested on Stack Overflow:
https://stackoverflow.com/questions/14481605/dispatch-queues-and-asynchronous-rncryptor/14571838#14571838
Using code like that my iPad mini successfully encrypted a 2 GB file in one minute, with no memory issues.


Resizable text boxes and text editing features

Hello Graphite Community,
I'm exploring the capabilities of Graphite and am particularly interested in enhancing the text editing features within the software. I'm looking for a way to implement resizable text boxes where the text adjusts automatically to fit the size of the box. Additionally, I'd like the ability to configure line spacing within these text boxes. Could you provide insights or guidance on how to develop these features, or is there already work underway to integrate such functionalities into Graphite? Any pointers or collaboration on this would be greatly appreciated!
Thank you!

Presumably moved to https://github.com/GraphiteEditor/Graphite/discussions/1963. I'll answer there.


[Discussion] Drop the 2 from AngularFire2

Context: Igor Minar's keynote speech at NG-BE Conf (link to relevant part)
The official guidelines seem to be to refer to angular simply as "Angular", without any version number attached to it. Considering Angular 4 will come out in less than 3 months, then Angular 5 a few months later, then Angular 6 etc, it might be confusing having to keep using AngularFire2.
I understand simply dropping the 2 might not be possible due to AngularFire for Angular 1.x but I think it's important to address how to best tackle this while AF2 is still in beta, before moving to RC or stable.
Starting with Angular 4, one possible solution would be to merge it all in one single package ("angularfire") maintaining separate versions for each. As in, keep all current versions of AF1 and AF2 in that one package and then simply publish a new 4.0.0 version with the current AngularFire2 codebase. There might be some version number collisions between AF1 and AF2 right now, though - AF1 is currently at 2.1.0 and AF2 is at 2.0.0-beta.6.
Ideas?

Actually, merging the two packages might not be a good idea. Simply creating a new 4.0.0 version on the "angularfire" package with the most recent AF2 code seems like the best option to me. You would still keep the "angularfire2" package for any project currently using it, of course.

Merging the two packages won't be practical. Semantically, they could reside in the same GitHub repo, but they would still be an entirely different API, code base, documentation set, and testing infrastructure. About the only similarities are the names and the Travis integrations.
Keeping them separated is probably the simplest way to avoid a lot of confusion and frustration trying to apply documentation, code samples, Stack Overflow answers, et al to these libraries. In fact, a more divergent namespace would probably make all of this much clearer and easier to differentiate.
@jwngr @abehaskins @davideast to disagree or add flavor.

@jsayol thanks for initiating the discussion.

Keeping them separated is probably the simplest way to avoid a lot of confusion and frustration trying to apply documentation, code samples, Stack Overflow answers, et al to these libraries.

Yeah, that's a good point.
I don't really see an easy solution to this. Maybe keeping it as AngularFire2 might be best, I don't know.

Just adding this: https://twitter.com/IgorMinar/status/808401117822357504

Our primary goal should be to avoid confusion for end-users. Our secondary goal should be to make our lives as the library publishers easier.
Staying in lock step with Angular's version number is a fool's errand and seriously impairs our secondary goal (proof of that is that we already have AngularFire(1) version 2.x.x so we couldn't even do this if we wanted). It does make our primary goal easier though. I wish we could just call it something else since Angular2 is such a different beast and there will be little to no clear upgrade path from AngularFire(1) to AngularFire2. Towards that end, we could move to call this something entirely new like ngfire and just say that angularfire is for Angular(1) and ngfire is for Angular2+.
I'm not in love with any of the naming options really. I think it's going to be confusing for end users no matter what we do.

Staying in lock step with Angular's version number is a fool's errand

Definitely. I just realized that by suggesting to jump to 4.0.0 it might have looked like that's what I was proposing to do. It's not.
I like the idea of rebranding the library altogether. It sets a very clear boundary between AngularJS (1.x) and Angular (2+). For end-users, the transition from angularfire2 to the new package would be relatively painless: change the dependency in package.json and replace any import statements, which should be a matter of seconds in most IDEs. At first it might cause a bit of confusion to some users but in the long run it seems like the better option.
Sticking with angularfire2 might be the easiest option right now but I think it'll become increasingly more confusing as time passes and Angular starts hitting higher version numbers.
For what it's worth , both ngfire and ng-fire are available as npm package names.

How about firebang?

If you want to go the risque route, how about fng?
Pronounced eff-ing, of course.

Adding this discussion on twitter started by @ocombe since it feels relevant: https://twitter.com/toddmotto/status/809912496454766593
I linked to @toddmotto's tweet since it has some interesting replies but click around for the whole conversation.
Maybe for now it'd be best to wait and let things settle.
Does the Angular team have an official position when it comes to naming libraries? Maybe this discussion should be happening at that level since I'm sure many other Angular-related projects will be facing this very same issue.

The Angular team official position is: don't use ng2-, but you can use ng-, just don't use ng- for the directives because it's the "official" namespace (ngIf, ngFor, ...). Which means that ng- doesn't seem like a good prefix for the library name in my opinion.
That's why I think that ngx- is nice :)

Is there something prohibiting the name from being @angular/firebase in NPM and angular/firebase in GitHub?

From Twitter, it looks like AngularFire2 was going to be moving under the Firebase organization at Google. So this would mean that it would need to be @firebase/angular but also raised on Twitter was that this name looks like a fork of @angular/angular. @firebase/angularfire will continue to be the Angular 1.x version, so that's not an option either.
More broken projects/breaking changes coming as part of this too :(

you could name it JustAngularFire :D

@firebase/angularfire will continue to be the Angular 1.x version, so that's not an option either

Actually @firebase/angularfire is available on NPM, but on github firebase/angularfire is indeed the repo for v1 so that could certainly cause some confusion.
Maybe @jwngr's initial proposal of ngfire might be the best bet placing it under the Firebase org: @firebase/ngfire on NPM and firebase/ngfire on github.

Just like to add my 2 cents here, I like the sounds of the @ namespace, so weather it's being moved under the firebase organisation then I think @firebase/angularfire would be suited but otherwise @angular/fire or @angular/angularfire would both work well

@angular/firebase?

@wbhob Totally agree.
@angular/firebase would make the most sense like @angular/material2 (they still put a 2 for some reason..)
or @angular/cli
its really how the eco system /should look

I agree with @wbhob but I see a problem that we will end up with two packages with same name firebase and @angular/firebase and it can be confused. Is there any plan to remove the need for firebase package and make angularfire2 standalone or it will be always needed?
Also, firebase is used to handle the storage, so, for the current version (beta.8), we can't use only angularfire2.
I suggest we remove the need for firebase from our apps and make the angularfire2 handle the storage and have firebase in its package.json so the users will not know that they are using furebase package, and won't be any confusion. So, we can move to @angular/firebase without any confusion.

It won't cause any confusion because people will only get @angular/firebase if they are angular users. It is a completely different package. To install it, someone would have to directly install and reference the package as @angular/firebase.
Wilson Hobbs
CEO/Founder/Developer at Canal
11th Grade at The Lovett School
Personal Information
E-mail<EMAIL_ADDRESS>Cell:<PHONE_NUMBER>
Website: www.wilsonhobbs.com (https://www.wilsonhobbs.com)
Twitter/Instagram/Facebook: @wbhob
Canal Information
E-mail<EMAIL_ADDRESS>Web: www.getcanal.com (https://www.getcanal.com/)
Twitter/Instagram/Facebook: @getcanal
On Apr 5, 2017, 9:09 AM -0400, Bruno da Silva joão<EMAIL_ADDRESS>wrote:

I agree with @wbhob (https://github.com/wbhob) but I see a problem that we will end up with two packages with same name firebase and @angular/firebase and it can be confused. Is there any plan to remove the need for firebase package and make angularfire2 standalone or it will be always needed?
Also, firebase is used to handle the storage, so, for the current version (beta.8), we can't use only angularfire2.
I suggest we remove the need for firebase from our apps and make the angularfire2 handle the storage and have firebase in its package.json so the users will not know that they are using furebase package, and won't be any confusion. So, we can move to @angular/firebase without any confusion.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub (https://github.com/angular/angularfire2/issues/716#issuecomment-291855874), or mute the thread (https://github.com/notifications/unsubscribe-auth/AEPIEl_UcilcYzjfOkeQFEOdaqwFGsWBks5rs5KHgaJpZM4LJoS4).

But what about the class names? Would it be AngularFire,  AngularFirebase, firebase, Firebase?
Will the package be name @angular/firebase but the class named AngularFire? That is confuse to me... I think the class name should refer to its package name. If a new user knows only the @angular/firebase package name, they will assume the class name should be something like FirebaseModule and not AngularFireModule.
Also, for packages like  @angular/* they use the * as the class name and not prefix the class with Angular. So it won't be the same as using others angular packages.
Finally, in version beta.8 we need both packages to use database and storage, angularfire2 and firebase, so the user still need to install both and that is the confusion point between firebase and @angular/firebase.

Based on feedback from David East, it can't be under the @angular/ name space. It needs to go under @firebase/ at some point.
There are no plans that I have heard to remove the dependency on the firebase package. It does a lot more for AngularFire2 developers than just Storage. Many apps are already built using both libraries together. I don't expect that to change.

We have a plan for this. Closing for now. I think you'll all like it.


Add option to convert position into Area ID

This adds another command line flag to automatically convert the given position into an Area ID (--pos-to-area-id). If either this new flag or the --area-id argument are given then the CRA message with Area ID is sent. I also changed several of the uses of f64 to use f32 instead to make sure there weren't rounding issues in the Area ID calculations.
Is this an acceptable way to squeeze in this functionality? I was also thinking about adding these functions into https://github.com/swift-nav/swiftnav-rs, but that's not a dependency for ntripping and would potentially complicate the portability since it pulls in C code as well.

One last change added to error check the latitude/longitude values, but ended up pulling in anyhow


Slayer plugin NPE

Found in a support user's logs, appears to be due to addCounter() not guaranteeing that counter is set to a non-null value. (stemming from slayer xp gains when the plugin doesn't know the current task, for first-time RuneLite users for instance) Relevant line of code: https://github.com/runelite/runelite/blob/0da504769df58cb2f648d2fcc8048978c7997c5b/runelite-client/src/main/java/net/runelite/client/plugins/slayer/SlayerPlugin.java#L645
2022-01-02 14:01:45 [Client] WARN  n.runelite.client.eventbus.EventBus - Uncaught exception in event subscriber
java.lang.NullPointerException: null
	at net.runelite.client.plugins.slayer.SlayerPlugin.killed(SlayerPlugin.java:645)
	at net.runelite.client.plugins.slayer.SlayerPlugin.onStatChanged(SlayerPlugin.java:568)
	at net.runelite.client.eventbus.EventBus$Subscriber.invoke(EventBus.java:70)
	at net.runelite.client.eventbus.EventBus.post(EventBus.java:223)
	at net.runelite.client.callback.Hooks.post(Hooks.java:169)
	at client.mz(client.java:29351)
	at client.hf(client.java:7570)
	at client.gg(client.java:3685)
	at client.i(client.java:1843)
	at am.k(am.java:335)
	at am.run(am.java:314)
	at java.base/java.lang.Thread.run(Unknown Source)

I'm training slayer anyway so I'll take a look at this.
If this seems to be happening for players who haven't used runelite yet, could this occur if you start a slayer task through a different client and then continue the task through runelite?

I'm training slayer anyway so I'll take a look at this.
If this seems to be happening for players who haven't used runelite yet, could this occur if you start a slayer task through a different client and then continue the task through runelite?

You should be able to reproduce this by removing your task and then getting a slayer xp drop

fixed in 1f6ae893d732cf3a022fe6d9662cd2a42ff2b217


Infinite progress without a configured google account

Hi,
i'm using the last 0.4.3 version of this library, and i noticed this strange behaviour.
On a new phone without a google account configured, if i run my own application, when i launch the intent for choosing an account, progress bar never ends!
startActivityForResult(                     AuthUI.getInstance()                             .createSignInIntentBuilder()                             .setProviders(                                     AuthUI.EMAIL_PROVIDER)                             .build(),                     RC_CODE);
Additionally, configuring a google account doesn't solve the problem: progress bar never ends!
Only after clearing all app data and restarting, i can pick my account.
I think this happens on all Android version, but just to be precise i'm using a Samsung phone with Android 6.0.1.
Thanks.

@Fax3D your issue describes a problem with a Google account but your code only shows the EMAIL_PROVIDER enabled.  Are you still having this problem?  If so, can you show a screenshot of the issue?

It's been a long time since this issue was updated and I have not heard similar reports so I am going to close this issue. Feel free to re-open it or start a new one if you want to discuss more!


tests: Fix schema stuff for user_status code

Should be a pretty easy review if tests pass.

Merged, thanks @showell!

Merged, thanks @showell!


Adding Timing::makeEquivCells and Timing::equivCells Python APIs

adding void Timing::makeEquivCells() and std::vectorodb::dbMaster* Timing::equivCells(odb::dbMaster* master) Python APIs

You have missed the DCO.  Please see the directions on the Details like to fix that.

You need to run clang-format as well

Fix the clang-format issue and add sign-offs to each commit.


Doc'd writing integration tests for the system check framework.

From DjangoConUS 2023 sprints 🏃. I didn't create a ticket in the issue tracker for this.
This PR adds additional examples to the system checks framework testing section of the documentation to demonstrate how one can test the integration of a system check within a project.
Non-model field system checks must be registered/imported in apps.py or __init__.py in order to function. Testing check functions directly as unit tests doesn't capture this, whereas using call_command does.
I added an example of how to assert against checks which raise a SystemCheckError as well as one which outputs to stderr, and how to capture the message using StringIO.
Thanks @evildmp for the crash course on Sphinx links 😄

@marcgibbons Thanks :+1: I pushed small edits.


LightLLM  maye should provide a script to download the weight and convert the weight.

I found there is no script that can download the weight and convert the weight.

Currently I try this test_llama to run a test, but I am confused about the model weight. Use hugging face to download the weights? But load_hf_weight seems we needs covert the huggingface model weight.

@lambda7xx  have you encountered any failure running the test with weights downloaded from hugging face?

I use the follow code to download a llama-7b-half model. Currently its weight can not use for lightllm. we have to convert it

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("decapoda-research/llama-7b-hf")

my  error.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/lightllm/test/model/model_infer.py", line 28, in tppart_model_infer
    torch.cuda.set_device(rank_id)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py", line 326, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/lightllm/test/model/model_infer.py", line 28, in tppart_model_infer
    torch.cuda.set_device(rank_id)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py", line 326, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Could you please post your nvidia-smi output?
I guess your number of graphics cards is less than 8. You may set a smaller world_size value here.
https://github.com/ModelTC/lightllm/blob/3d84a6e625dbac541fb38957ec14726cb680ac84/test/model/test_llama.py#L11
@lambda7xx

my GPUs is A10. I found the weight format from huggingface is wrong.
ss 'lightllm.models.llama.layer_infer.model.LlamaTpPartModel'>, batch_size:20, input_len:64, output_len:16
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/lightllm/test/model/model_infer.py", line 34, in tppart_model_infer
    model_part = model_class(dist.get_rank(),
  File "/home/ubuntu/.local/lib/python3.10/site-packages/lightllm-1.0.0-py3.10.egg/lightllm/models/llama/layer_infer/model.py", line 20, in __init__
    with open(os.path.join(weight_dir, "config.json"), 'r') as json_file:
FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/.cache/huggingface/hub/models--decapoda-research--llama-7b-hf/blobs/config.json'
F

if we download the weight from  https://huggingface.co/decapoda-research/llama-7b-hf/tree/main, its format is bin

if we download the weight from https://huggingface.co/decapoda-research/llama-7b-hf/tree/main, its format is bin

https://github.com/ModelTC/lightllm/blob/3d84a6e625dbac541fb38957ec14726cb680ac84/lightllm/models/llama/layer_weights/hf_load_utils.py#L15-L18
bin format is also supported.

FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/.cache/huggingface/hub/models--decapoda-research--llama-7b-hf/blobs/config.json'

This indicates the model path you passed in might be wrong.
What model_dir value did you set here?
https://github.com/ModelTC/lightllm/blob/3d84a6e625dbac541fb38957ec14726cb680ac84/test/model/test_llama.py#L12
And you may run find /home/ubuntu/.cache/huggingface/hub/models--decapoda-research--llama-7b-hf -name config.json to check if you have set the right model_dir value.

let me try again.

@lambda7xx I use the same way with you to download the llma-7b-hf, however tokenizer is not download successfully, as your error log mentioned: can't load tokenizer. you should check if the tokenizer.model is existed or not.

@lambda7xx I use the same way with you to download the llma-7b-hf, however tokenizer is not download successfully, as your error log mentioned: can't load tokenizer. you should check if the tokenizer.model is existed or not.

And you can also download the whole repo with git @lambda7xx
# Make sure you have git-lfs installed (https://git-lfs.com)
git lfs install
git clone https://huggingface.co/decapoda-research/llama-7b-hf

# if you want to clone without large files – just their pointers
# prepend your git clone with the following env var:
GIT_LFS_SKIP_SMUDGE=1

@llehtahw  I use your command to download the weights.
I still have this problem
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/ubuntu/lightllm/lightllm/server/api_server.py", line 196, in <module>
    main()
  File "/home/ubuntu/lightllm/lightllm/server/api_server.py", line 163, in main
    httpserver_manager = HttpServerManager(args.model_dir,
  File "/home/ubuntu/lightllm/lightllm/server/httpserver/manager.py", line 18, in __init__
    self.tokenizer = get_tokenizer(model_weightdir, tokenizor_mode)
  File "/home/ubuntu/lightllm/lightllm/server/tokenizer.py", line 55, in get_tokenizer
    tokenizer = AutoTokenizer.from_pretrained(tokenizer_name, *args,
  File "/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 690, in from_pretrained
    raise ValueError(
ValueError: Tokenizer class LLaMATokenizer does not exist or is not currently imported.

 This might cause a significant slowdown. Consider using a fast tokenizer instead.
init MemoryManager size 120000 dtype torch.float16 head_num 32 head_dim 128 layer_num 32
init MemoryManager, self.mem_state.shape:torch.Size([120000]) and self._mem_cum_sum.shape:torch.Size([120000]) and self.indexs.shape:torch.Size([120000]) can_use_mem_size 120000
Process Process-1:
router init state: CUDA out of memory. Tried to allocate 938.00 MiB (GPU 0; 21.99 GiB total capacity; 21.07 GiB already allocated; 601.00 MiB free; 21.07 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF detoken init state: init ok

I change the 120000 to 12000, still fails.
Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
init MemoryManager size 12000 dtype torch.float16 head_num 32 head_dim 128 layer_num 32
init MemoryManager, self.mem_state.shape:torch.Size([12000]) and self._mem_cum_sum.shape:torch.Size([12000]) and self.indexs.shape:torch.Size([12000]) can_use_mem_size 12000
router init state: 'list' object has no attribute 'shape' detoken init state: init ok

It means I can't use one A10 24GPUs to server a 7b model. @llehtahw

router init state: 'list' object has no attribute 'shape' detoken init state: init ok

@hiworldwzj please have a look

I use your patch, then I use the command

python3 -m lightllm.server.api_server --model_dir /home/ubuntu/llama-7b-hf --tp 1 --max_total_token_num 120000

my error log

Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
init MemoryManager size 120000 dtype torch.float16 head_num 32 head_dim 128 layer_num 32
init MemoryManager, self.mem_state.shape:torch.Size([120000]) and self._mem_cum_sum.shape:torch.Size([120000]) and self.indexs.shape:torch.Size([120000]) can_use_mem_size 120000
Process Process-1:
router init state: CUDA out of memory. Tried to allocate 938.00 MiB (GPU 0; 21.99 GiB total capacity; 21.07 GiB already allocated; 601.00 MiB free; 21.07 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF detoken init state: init ok

I change the 120000 to 12000, still fails.
Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
init MemoryManager size 12000 dtype torch.float16 head_num 32 head_dim 128 layer_num 32
init MemoryManager, self.mem_state.shape:torch.Size([12000]) and self._mem_cum_sum.shape:torch.Size([12000]) and self.indexs.shape:torch.Size([12000]) can_use_mem_size 12000
router init state: 'list' object has no attribute 'shape' detoken init state: init ok

It seems I can't use one A10 24GPUs to server a 7b model. @llehtahw

max_total_token_num should be more smaller。please see https://github.com/ModelTC/lightllm/blob/main/docs/ApiServerArgs.md


Default Parser

Change to the plugin behavior when no parser is provided in the tool field. Previous behavior was to error out. Now, the following scenarios can occur with their corresponding behavior:
Note Simplified error message is in the format: Attempted parser <PARSER> with pattern <PATTERN>, but was unsuccessful.

No tool is defined (i.e. no parser or pattern) & at least 1 parser is successful:

ex call)recordCoverage(tools: [])
Try each possible parser with its corresponding default pattern
Print the successful parsers' logs and a simplified error message for each parser that failed.
Snippet of example output [Successful Go coverage, rest fail]:

No tool is defined (i.e. no parser or pattern) & all parsers fail

ex call) recordCoverage(tools: [])
Try each possible parser with its corresponding default pattern
Print the logs of each parser tried.
Snippet of example output [All fail]:

At least 1 pattern is defined, but none of them have a corresponding parser:

ex call) recordCoverage(tools: [[pattern: '*coverage.out'], [pattern: '*coverage.xml']])
Try each input pattern with each possible parser.
Follows same ruling as above:

1+ successful -> print verbose logs of successful combination, simplified error message of other combinations
All fail -> print verbose logs of all
Snippet of example output [2 patterns for Go & Python coverage successfully get auto detected]:

Note Scenario 3 can occur even if another tool with a Parser is defined. Treats the expansion of that tool as 1 set.
ex call) recordCoverage(tools: [[parser: 'JACOCO'], [pattern: '*coverage.xml']])
Testing done
Tested on a local instance of Jenkins running the plugin version. Ran the aforementioned scenarios via pipeline jobs and recorded their builds' console output.
Also created unit tests that test the same thing.
### Submitter checklist
- [x] Make sure you are opening from a **topic/feature/bugfix branch** (right side) and not your main branch!
- [x] Ensure that the pull request title represents the desired changelog entry
- [x] Please describe what you did
- [x] Link to relevant issues in GitHub or Jira
- [x] Link to relevant pull requests, esp. upstream and downstream changes
- [x] Ensure you have provided tests - that demonstrates feature works or fixes the issue

Can you please elaborate why this is useful? Is specifying the parser too complicated in your setup?

Can you please elaborate why this is useful? Is specifying the parser too complicated in your setup?

Yeah, our usecase is that we have a number of pipelines (1000+) migrating over that simply upload their coverage files to a server/service that determines the type of coverage file it is for them, so users aren't specifying anything about the specifics of the coverage file type. We want to make the migration to using the recordCoverage() call in their pipelines as hands-off and minimal steps as possible, which involved moving the "What type of coverage is this?" determination functionality into the coverage-plugin.


feat: source editor that supports multiple files

What changed / motivation ?
The StyleX website has an empty-ish page for the playground. This tackles one of the issues: A source editor pane that support tabs for multiple files.
Linked PR/Issues
This helps address issue #732
Additional Context

Implemented Change (w/o server running):

Screenshots:

Pre-flight checklist

[x] I have read the contributing guidelines
Contribution Guidelines
[x] Performed a self-review of my code

@nmn Hey Naman! I'm still working through the functionality for this feature, but I'm having some trouble getting the styling correct. I've added 'overflow: auto,' to the container component, but the result is above. Without that, the header causes the remainder of the text to overflow slightly and the preview pane is not entirely visible. Any suggestions?
Also, it's currently setting default tabs to those provided within playground-utils/files. Is it preferable to only have the src as a tab and then allow the user to create additional tabs? Sorry if these are silly questions,  but let me know your thoughts. Thanks in advance!

@aminaopio I suggest building a sidebar instead. Within the sidebar, you can use <details><summary> elements to create a tree view for the entire file-tree.
You can add a single icon button to show the sidebar that can be absolutely positioned appear over the code editor. This way you can avoid dealing with layout issues.
If you continue to deal with issues, lmk, and I'll help with the UI.

Hi @nmn, I ended up having to tuck the icon button into a header anyway, but let me know your thoughts. Maybe the UI could use some work....

I have a project with Rollup working properly here:
https://stackblitz.com/edit/sb1-tu4v6l?file=src%2FApp.tsx

It's a basic React app with Rollup and livereload
It uses a forked version of @stylexjs/rollup-plugin

It adds a few lines of code to write the individual transformed JS files and the collected styles from them to a previews folder.

Using this project, you should be able to have a four tab design with the following tabs:

The editor
The transformed JS code of the currently active file
The styles collected from the currently active file

OR the final CSS

The preview

The preview should "just work".
It should not refresh twice on every change

You can probably skip the eslint and typescript parts of the project for now. That should make the setup fairly small.

I'm having some trouble visualizing what you mean by 4 tab design @nmn

Do you want each of those tabs to be hidden within the sidebar and take up 100% of the screen when active?

Or something like this:

I was under the impression that what I'm implementing currently is just the multi-file support for the code editor alone

 A source editor pane that support tabs for multiple files

I was under the impression that what I'm implementing currently is just the multi-file support for the code editor alone

You can start with that. Do you want me to integrate the Rollup project into my playground PR so you can just rebase on top and do the work on adding tabs for the source editor?

Hey @aminaopio Sorry, I confused the two Playground related PRs. Please carry on with your work on adding tabs.

Hey @aminaopio Sorry, I confused the two Playground related PRs. Please carry on with your work on adding tabs.

Oh okay all good! I'll start implementing what you suggested once this one gets finished up and merged

You can start with that. Do you want me to integrate the Rollup project into my playground PR so you can just rebase on top and do the work on adding tabs for the source editor?

We can try this out on our own, but may need assistance. I'll get back to you on this one. Thanks!

@nmn Is this okay to merge? And can you go ahead an integrate the rollup project as well? Thanks!

@aminaopio I think you will need to rebase your branch before we can merge.

@aminaopio I think you will need to rebase your branch before we can merge.
The basic functionality seems to work, but the design could use a bit more love too.

The arrow icons can be replaced with real SVG icons. Docusaurus might also have an icon for this available.
The trash icon should probably be red, and likely the outline variant.
The filenames should not be editable all the time. Double-clicking (or hitting Enter) should enable edit mode and the border around the text input should be more prominent.

None of these changes are blockers, but we will need to get the other parts of the playground working before we are able to merge it anyway.

@nmn Ok thanks for the feedback, it should be good to go now. I imported icons from FontAwesome and double-clicking will enable edit mode for the filename (let me know if the white bg is too harsh/if the red trash icon should be a diff color).
We're hoping to implement the rest of the playground in parallel with working on other issues. Do you have any beginner-friendly ideas for contributing to the docs?

Do you have any beginner-friendly ideas for contributing to the docs?

We need help identifying issues in the docs. We've received feedback that the docs are not good enough, but we also don't have any feedback on what should be improved.
One of my ideas is to add a section called "Recipes" or "Patterns". It can contain a bunch of examples of how to do various patterns from other CSS libraries like variants, or applying styles on descendent when a parent is hovered by using variables.
I have written long answers in Github Discussions answering many such questions, so digging through all the discussion topics, it should be possible find a bunch of patterns that can be documented on the website itself.

Looking further at this PR, there are a few nits still:

The red icon doesn't look great on the pink background. Let's stick to the "primary" color (white or black)
There is no way to create a file within a folder.
Can't drag and drop files between folders (stretch goal)

@nmn, I've added the missing file operation functionalities to the playground and would appreciate your feedback.

It looks OK visually, of course the buttons are functional in my testing. I assume that's expected?

It looks OK visually

@nmn Anything specific you want us to improve with interface? Also, can you let us know what else we should focus on before this can get merged? I'm currently trying to get the drag and drop to work.

...of course the buttons are functional in my testing. I assume that's expected?

I'm a bit confused by this, if you could clarify, thanks!

The file tree needs a bit more work as the actions are not all working to create a file, create a file in a folder etc.
So, I think it's worth merging a version that lets you edit a single file for now. I'll rebase and merge it. Thanks!

The file tree needs a bit more work as the actions are not all working to create a file, create a file in a folder etc.

Hey @nmn, so currently it's not possible to create a file before the webcontainer finishes loading. Once it's fully loaded, it should work as expected. Let us know if it's different on your end and we can troubleshoot

We could make it so that you don't have to wait until the container finishes booting up?

This is fine. You can show a loading indicator in the file tree as well.
Please rebase this branch on the latest better-playground branch so I can test it with all the other changes.

This is fine. You can show a loading indicator in the file tree as well.

Done

Please rebase this branch on the latest better-playground branch so I can test it with all the other changes.

Don't think I did this part correctly @nmn but let me know!

@aminaopio This is the process for rebasing:
Before starting run git config --global rerere.enabled true to make rebasing slightly faster later. This config option will auto-resolve conflicts once you resolve them once.

On your local copy of the repo, checkout the main branch git checkout main
Make sure you pull from the original stylex repo: git pull upstream main

You can use git remote -v to see the "remote" repos configured.
Use git remote add upstream <style repo URL> to add the upstream remote
Now, git pull upstream main should work.
Do a git push origin main to update your fork as well.

Switch back to your feature branch with git checkout dev-amina-playground

OPTIONAL: squash the commits in your branch to make rebasing later easier.

Rebase with git rebase main. Fix all the conflicts. The more commits you have the longer this will take.

OPTIONAL: squash the commits in your branch to make rebasing later easier.

Tried to squash as much as I could, but commits from better-playground popped up as well...is that normal?

Rebase with git rebase main. Fix all the conflicts. The more commits you have the longer this will take.

I think it's done?

I'm so sorry. I suggested rebasing on main, but what I meant was rebasing onto better-playground.
Can you re-rebase onto the better-playground branch. It's all the same steps, but with that branch name instead of main.

I'm so sorry. I suggested rebasing on main, but what I meant was rebasing onto better-playground.

That's my bad, I didn't notice

Can you re-rebase onto the better-playground branch. It's all the same steps, but with that branch name instead of main.

We should be good to go @nmn. For future reference, what's the proper way to clean up commit history? Would running git rebase --interactive $commit and manually removing duplicate commits prior to rebasing cause any issues?

@aminaopio I'm still seeing a bunch of conflicts on this PR. I don't think the rebase worked. Are you sure you pulled the latest better-playground branch from the stylex repo?

If this is too challenging, you can try creating a new PR from scratch and copy paste the relevant changes.

@aminaopio I'm still seeing a bunch of conflicts on this PR. I don't think the rebase worked. Are you sure you pulled the latest better-playground branch from the stylex repo?

@nmn So it's been up-to-date:

Here's my "remote" repos:

The issue starts when I run git rebase better-playground inside of my branch (dev-amina-playground).  (Running git config --global rerere.enabled at the beginning didn't seem to do anything).

Where I may have messed up is that I accepted all the current changes inside of each of those commits instead of the incoming? I can go ahead and make a new PR, but I don't where in this process I'm messing up

+1 I recommend a new PR. In the past I've had issues where the Vercel CI job randomly fails when you touch the package-lock file after rebasing

New PR is here: https://github.com/facebook/stylex/pull/826


GET APIS completed

/shopcarts for listing all shoppingcarts
/shopcarts/{sid} for displaying all products in shopcart sid
/shopcarts?=sid for displaying all products in shopcart sid
/shopcarts/{sid}/products for displaying all products in shopcart sid
/shopcarts/{sid}/products?sku={sku} for displaying product sku in shopcart sid
/shopcarts/{sid}/products/{sku} for displaying product sku in shopcart sid
#13 #14 #15

@Pratik-Karnik I changed the comment


Feature Request: Control of LEDs?

Hi, thanks for putting this together. I haven't opened up the C545 yet but is there any possibility for control over the leds as well? Would love to be able to disable the "sleep" mode light entirely or after a few seconds/minute after this mode is enabled.

Got it thank you!


Typo in documentation about high-level function for downloading compendia

From @cgreene

is this function name a typo? https://alexslemonade.github.io/refinebio-py/high_level_functions.html#pyrefinebio.high_level_functions.download_quandfile_compendium

I think this is a typo in the docs because:
https://github.com/AlexsLemonade/refinebio-py/blob/9d9ee1fffe1db2ac5e45089a982434d44c4134b1/pyrefinebio/high_level_functions.py#L201

This typo was addressed in #42
https://github.com/AlexsLemonade/refinebio-py/pull/42/files#diff-2bbedd13cfba1db572820a90f3d61d13b85d171a926091b42acb056acf409773R208

Feel free to close once the change gets deployed to GH pages!

The docs have been deployed! This is fixed!


CON-2957 - reviews styling and translation fixes

Coverage remained the same at 25.958% when pulling c3f6f31a48a5b7c5bc51d6a6780291d4c421ade3 on CON-2957 into e17576e61aec0b054ad705f4e2d82ea181190055 on master.


Version FAR_0_7_0_3 (CPY)  crashes when trying to disabled the default OSD.

Firstly, I created this as a new issue because all the changes related to dinput8 in the 7.0 version
Control + Shift +  Backspace makes the game crash
Bask + Start (Gamepad) makes the came crash.
Control + Shift + O Works.
It is ok not to used. Actually I don't know what it is in there.
I have x96dbg so if you tell me what should I look for I might be able to help:
So far in x64dbg I have a EXCEPTION_ACCESS_VIOLATION steam_64api
000000003B40D72D | 00 00                    | add byte ptr ds:[rax],al                |
000000003B40D72F | 00 70 1C                 | add byte ptr ds:[rax+1C],dh             |
000000003B40D732 | 40 3B 00                 | cmp eax,dword ptr ds:[rax]              |
000000003B40D735 | 00 00                    | add byte ptr ds:[rax],al                |
000000003B40D737 | 00 80 1C 40 3B 00        | add byte ptr ds:[rax+3B401C],al         |
000000003B40D73D | 00 00                    | add byte ptr ds:[rax],al                |
000000003B40D73F | 00 60 D7                 | add byte ptr ds:[rax-29],ah             |
000000003B40D742 | 40 3B 00                 | cmp eax,dword ptr ds:[rax]              |
000000003B40D745 | 00 00                    | add byte ptr ds:[rax],al                |
000000003B40D747 | 00 40 D7                 | add byte ptr ds:[rax-29],al             |
000000003B40D74A | 40 3B 00                 | cmp eax,dword ptr ds:[rax]              |
000000003B40D74D | 00 00                    | add byte ptr ds:[rax],al                |
000000003B40D74F | 00 00                    | add byte ptr ds:[rax],al                |
000000003B40D751 | 00 00                    | add byte ptr ds:[rax],al                |
000000003B40D753 | 00 00                    | add byte ptr ds:[rax],al                |
000000003B40D755 | 00 00                    | add byte ptr ds:[rax],al                |
000000003B40D757 | 00 00                    | add byte ptr ds:[rax],al                |

manual FAR.ini (GlobalIlluminationWorkgroups=32)  have no impact on the game as it does with baldman v3. But who cares! With this mod my old SAVES are back! i`m so fkn happy!))))

same problem here :(

I think is better say now, cpy isn´t compatible.
I don't touched in the crack to know the crash reason, i think i don't need try fix, why?, why we can use the baldman v3.

I hope if the steampunks someday create a keygen to this game too (I Know, we already have 2 cracks), and everyone can play with a stable way with the mod

Sure, I will give it a try anyway :)

I lost my game saves with the other crack, but after some edittings of the GameSave files it works agains.
FAR works as well.