Istio Envoyfilter support for grpc server calls

Describe the feature request
Envoyfilter inline lua script functionality to support for grpc calls like it does for http calls.
we are trying to route some of the calls that are reaching out ingress-gateway to a ratelimiting grpc server. we thought of implementing this through a Envoyfilter with specifying out conditions in an inline lua filter. But the inline lua filter only allows for http calls to be made. we need to have the ability to have  GRPC calls too.
we had earlier enabled a simliar setup for an http server using the same Envoyfilter with lua inline scripts.
In that script we have the http call like
response_headers, response_body = request_handle:httpCall(
tostring(cluster),
request_headers,
request_body,
timeout
)
we would like to have a similar support for the GRPC calls as well?
Could Anyone help us with how we can acheive this ?
Describe alternatives you've considered
Additional context

Does your rate limiting gRPC server an implementation of Envoy's ratelimit service https://github.com/envoyproxy/envoy/blob/4c5f4310bb8a19a38e5377d7a2d2dc6aa4560f47/api/envoy/service/ratelimit/v3/rls.proto#L22?

@ramaraochavali - Yes it is an  implementation of Envoy's ratelimit service

Ok. If that is the case, did you try inserting a network/http Envoy ratelimit filter https://github.com/envoyproxy/envoy/blob/4c5f4310bb8a19a38e5377d7a2d2dc6aa4560f47/api/envoy/service/ratelimit/v3/rls.proto#L22 instead of calling the service your self in Lua script?

Our workflow doesnt support this... the ratelimitting provider is a seperate team used across our organisation and we will have to use that.
Is there any possiblility of making a grpc call from the lua filter?

Our workflow doesnt support this... the ratelimitting provider is a seperate team used across our organisation and we will have to use that.

Sorry. I did not follow this probably I am missing some detail. I am not suggesting to use a different ratelimiting service. The rate limiting provider (I assume implementation of the Envoy ratelimiting service) can be any where but as long as the ratlimiting filter is configured to use the service using the https://www.envoyproxy.io/docs/envoy/latest/api-v2/config/ratelimit/v2/rls.proto#envoy-api-msg-config-ratelimit-v2-ratelimitserviceconfig, Envoy would call that service and enable rate limiting.  It is similar to you calling in Lua filter.

Is there any possiblility of making a grpc call from the lua filter?

AFAIK, Not possible - I may be wrong here

@ramaraochavali - It would be of great help if you could share me a sample on how i can set the grpc service at the istio-ingress gateway level Envoy?
Also we saw an example of using a grpc json transcode ? would this be in any case helpful for us ? https://github.com/istio/istio/issues/7652

@ramaraochavali - Thanks for you Quick Response.. We greatly appreciate it.
we tried configuring Envoyfilter like you said. Could you please verify the below config is correct
kind: EnvoyFilter
metadata:
  name: filter-ratelimit
  namespace: istio-system
spec:
  workloadSelector:
    # select by label in the same namespace
    labels:
      istio: ingressgateway
  configPatches:
      # The Envoy config you want to modify
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: "envoy.http_connection_manager"
              subFilter:
                name: "envoy.router"
      patch:
        operation: INSERT_BEFORE
        value:
         name: envoy.rate_limit
         config:
           # domain can be anything! Match it to the ratelimter service config
           domain: test
           rate_limit_service:
             grpc_service:
               envoy_grpc:
                 cluster_name: rate_limit_service
               timeout: 0.25s
    - applyTo: CLUSTER
      match:
        cluster:
          service: "RATE_LIMITING_SERVER_ENDPOINT"
      patch:
        operation: ADD
        value:
          name: rate_limit_service
          type: STRICT_DNS
          connect_timeout: 0.25s
          lb_policy: ROUND_ROBIN
          http2_protocol_options: {}
          hosts:
            - socket_address:
                address: "RATE_LIMITING_SERVER_ENDPOINT"
                port_value: 81

The approach looks good to me. Please try and let me know if you run in to any issues

@ramaraochavali - Regarding this config we have 2 questions

How do we pass headers to the ratelimiting server [is there any option for that?]
We also want to skip this call when some specific header values? like how we do in the inline lua script

@ramaraochavali - Any update in this ? It would be of great help as this is blocked our prod deployment

@Adharsh-Muraleedharan Sorry, was busy with other stuff

https://github.com/istio/istio/issues/22068#issuecomment-608302282 - Would this help? It has example of how to pass headers using request_headers: - does it help?
If you want to skip the call when specific header is present? If you add that header to the request_headers and if there is no value for that header, Envoy would skip calling rate limiting service for that request. See https://github.com/envoyproxy/envoy/issues/10124 and https://github.com/envoyproxy/envoy/pull/10212 for details on when Envoy skips calling rate limiting service when header value is not present.

@ramaraochavali - Thanks for the reply
we tried based on your suggestion but we are still not able to hit our ratelimitting server via this config .. Could you please review the below config and let us know if there are any issues here?
kind: EnvoyFilter
metadata:
  name: filter-ratelimit
  namespace: istio-system
spec:
  workloadSelector:
    # select by label in the same namespace
    labels:
      istio: ingressgateway
  configPatches:
    # The Envoy config you want to modify
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: "envoy.http_connection_manager"
              subFilter:
                name: "envoy.router"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.rate_limit
          config:
            # domain can be anything! Match it to the ratelimter service config
            domain: {{XYZ.OURDOMAIN.COM}}
            failure_mode_deny: true
            rate_limit_service:
              grpc_service:
                envoy_grpc:
                  cluster_name: rate_limit_cluster
                timeout: 10s
    - applyTo: CLUSTER
      match:
        cluster:
          service: {{OUR_RATE_LIMITTING_SERVER}}
      patch:
        operation: ADD
        value:
          name: rate_limit_cluster
          type: STRICT_DNS
          connect_timeout: 10s
          lb_policy: ROUND_ROBIN
          http2_protocol_options: {}
          hosts:
            - socket_address:
                address: {{OUR_RATE_LIMITTING_SERVER}}
                port_value: 81
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: filter-ratelimit-svc
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
    - applyTo: HTTP_ROUTE
      match:
        context: GATEWAY
        routeConfiguration:
          vhost:
            name: {{XYZ.OURDOMAIN.COM}}
            route:
              name: http-echo-service
              action: ANY
      patch:
        operation: MERGE
        value:
          route:
            rate_limits:
              - actions: # any actions in here
                  # Multiple actions nest the descriptors
                  # - generic_key:
                  # descriptor_value: "test"
                  - {request_headers: {header_name: "method", descriptor_key: "GET"}}
                  - {request_headers: {header_name: "host", descriptor_key: "{{XYZ.OURDOMAIN.COM}}"}}
                  - {request_headers: {header_name: "x-request-id", descriptor_key: "ac5b684b-4bc6-4474-a943-0de4f1faf8df"}}
                  # - remote_address: {}
                  # - destination_cluster: {}

Can you please provide the config_dump output of the Envoy?

@ramaraochavali - is this what you are expecting
    {
        "name": "<IP_ADDRESS>_80",
        "address": {
            "socketAddress": {
                "address": "<IP_ADDRESS>",
                "portValue": 80
            }
        },
        "filterChains": [
            {
                "filters": [
                    {
                        "name": "envoy.http_connection_manager",
                        "typedConfig": {
                            "@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager",
                            "statPrefix": "outbound_<IP_ADDRESS>_80",
                            "rds": {
                                "configSource": {
                                    "ads": {}
                                },
                                "routeConfigName": "http.80"
                            },
                            "httpFilters": [
                                {
                                    "name": "mixer",
                                    "typedConfig": {
                                        "@type": "type.googleapis.com/istio.mixer.v1.config.client.HttpClientConfig",
                                        "transport": {
                                            "networkFailPolicy": {
                                                "policy": "FAIL_CLOSE",
                                                "baseRetryWait": "0.080s",
                                                "maxRetryWait": "1s"
                                            },
                                            "checkCluster": "outbound|15004||istio-policy.istio-system.svc.cluster.local",
                                            "reportCluster": "outbound|15004||istio-telemetry.istio-system.svc.cluster.local",
                                            "reportBatchMaxEntries": 100,
                                            "reportBatchMaxTime": "1s"
                                        },
                                        "serviceConfigs": {
                                            "default": {
                                                "disableCheckCalls": true
                                            }
                                        },
                                        "defaultDestinationService": "default",
                                        "mixerAttributes": {
                                            "attributes": {
                                                "context.proxy_version": {
                                                    "stringValue": "1.4.2"
                                                },
                                                "context.reporter.kind": {
                                                    "stringValue": "outbound"
                                                },
                                                "context.reporter.uid": {
                                                    "stringValue": "kubernetes://istio-ingressgateway-7c6c65958-vs976.istio-system"
                                                },
                                                "source.namespace": {
                                                    "stringValue": "istio-system"
                                                },
                                                "source.uid": {
                                                    "stringValue": "kubernetes://istio-ingressgateway-7c6c65958-vs976.istio-system"
                                                }
                                            }
                                        },
                                        "forwardAttributes": {
                                            "attributes": {
                                                "source.uid": {
                                                    "stringValue": "kubernetes://istio-ingressgateway-7c6c65958-vs976.istio-system"
                                                }
                                            }
                                        },
                                        "ignoreForwardedAttributes": true
                                    }
                                },
                                {
                                    "name": "envoy.cors"
                                },
                                {
                                    "name": "envoy.fault"
                                },
                                {
                                    "name": "envoy.rate_limit",
                                    "config": {
                                        "domain": "abcdefghi.freshpo.com",
                                        "failure_mode_deny": true,
                                        "rate_limit_service": {
                                            "grpc_service": {
                                                "envoy_grpc": {
                                                    "cluster_name": "rate_limit_cluster"
                                                },
                                                "timeout": "10s"
                                            }
                                        }
                                    }
                                },
                                {
                                    "name": "envoy.router"
                                }
                            ],
                            "tracing": {
                                "operationName": "EGRESS",
                                "clientSampling": {
                                    "value": 100
                                },
                                "randomSampling": {
                                    "value": 1
                                },
                                "overallSampling": {
                                    "value": 100
                                }
                            },
                            "httpProtocolOptions": {},
                            "serverName": "istio-envoy",
                            "streamIdleTimeout": "0s",
                            "useRemoteAddress": true,
                            "generateRequestId": true,
                            "forwardClientCertDetails": "SANITIZE_SET",
                            "setCurrentClientCertDetails": {
                                "subject": true,
                                "cert": true,
                                "dns": true,
                                "uri": true
                            },
                            "upgradeConfigs": [
                                {
                                    "upgradeType": "websocket"
                                }
                            ],
                            "normalizePath": true
                        }
                    }
                ]
            }
        ],
        "trafficDirection": "OUTBOUND"
    },
    {
        "address": {
            "socketAddress": {
                "address": "<IP_ADDRESS>",
                "portValue": 15090
            }
        },
        "filterChains": [
            {
                "filters": [
                    {
                        "name": "envoy.http_connection_manager",
                        "config": {
                            "codec_type": "AUTO",
                            "http_filters": {
                                "name": "envoy.router"
                            },
                            "route_config": {
                                "virtual_hosts": [
                                    {
                                        "domains": [
                                            "*"
                                        ],
                                        "name": "backend",
                                        "routes": [
                                            {
                                                "match": {
                                                    "prefix": "/stats/prometheus"
                                                },
                                                "route": {
                                                    "cluster": "prometheus_stats"
                                                }
                                            }
                                        ]
                                    }
                                ]
                            },
                            "stat_prefix": "stats"
                        }
                    }
                ]
            }
        ]
    }
]

do you have "rate_limit_cluster" cluster defined and pointing to your rate limit service?

@ramaraochavali - In the above config only we are setting the rate limit cluster .
like this where {{OUR_RATE_LIMITING_SERVER}} is replaced by the rate limiting server endpoint
Let us know if this is how we should define the "rate_limit_cluster" to make a GRPC call to our rate limiting endpoint ?
        operation: INSERT_BEFORE
        value:
          name: envoy.rate_limit
          config:
            # domain can be anything! Match it to the ratelimter service config
            domain: {{XYZ.OURDOMAIN.COM}}
            failure_mode_deny: true
            rate_limit_service:
              grpc_service:
                envoy_grpc:
                  cluster_name: rate_limit_cluster
                timeout: 10s
    - applyTo: CLUSTER
      match:
        cluster:
          service: {{OUR_RATE_LIMITING_SERVER}}
      patch:
        operation: ADD
        value:
          name: rate_limit_cluster
          type: STRICT_DNS
          connect_timeout: 10s
          lb_policy: ROUND_ROBIN
          http2_protocol_options: {}
          hosts:
            - socket_address:
                address: {{OUR_RATE_LIMITING_SERVER}}
                port_value: 81

@ramaraochavali - Even though we are mentioning the cluster details in the Envoyconfig it is not showing up on the envoy config dump.
The istio version we are using is 1.4.2

    - applyTo: CLUSTER
      match:
        cluster:
          service: {{OUR_RATE_LIMITING_SERVER}}

I think the ^^ is wrong.  It might not match.
Try adding a cluster like this example https://github.com/istio/api/blob/master/networking/v1alpha3/envoy_filter.proto#L154

@ramaraochavali -- we had already tried this but we are getting an error while applying this config
The Error is :
"spec.configPatches.match" must validate one and only one schema (oneOf). Found none valid
spec.configPatches.match.listener in body is required"
      match:
        context: GATEWAY
      patch:
        operation: ADD
        value:
          name: rate_limit_cluster
          type: STRICT_DNS
          connect_timeout: 10s
          lb_policy: ROUND_ROBIN
          http2_protocol_options: {}
          hosts:
            - socket_address:
                address: vpce-0b247209ae0145d88-4fa54j71.vpce-svc-0874c1e9512bd57dc.us-east-1.vpce.amazonaws.com
                port_value: 81

The full error is
"spec.configPatches.match" must validate one and only one schema (oneOf). Found none valid
spec.configPatches.match.listener in body is required

Not sure why it is failing now, may be @rshriram @hzxuzhonghu have any ideas on the cluster creation patch?

@ramaraochavali - Found someone facing a similar issue too
https://github.com/istio/istio/issues/22615

I see. One of the other you can try is - create a ServiceEntry for your ratelimiting service and Istio generates outbound cluster like outbound|..... (you can check the name in the configdump) and try giving that cluster name here.

@ramaraochavali thanks for the info. I am Adharsh's colleague. We already created service entry for our ratelimit service as it is an external one. does that appear in proxy-config IngressGateway Pod? If so, we can't find an entry for that. are we missing any?

AFAIK, It should. Can you please LMK the host name you used and the config_dump of gateway?

This is how we are creating this serviceentry for the rate limiting server
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: endpoint
  namespace: default
spec:
  hosts:
  - vpc-endpoint-for-ratelimit-svc
  location: MESH_EXTERNAL
  ports:
  - name: grpc
    number: 81
    protocol: GRPC
  resolution: DNS

But this does not create a mapping in the config_dump of the Gateway.

It seems your service entry definition is incorrect - endpoint is missing, are you seeing any errors in pilot logs?
 endpoints:
    - address: vpce-xxxxx.xxxxxxxxxxxxxxxxxxxx.vpce.amazonaws.com
      ports:

And BTW, I thought you got any error in cluster creation - How are you seeing rate_limit_cluster in list of clusters?

@ramaraochavali - In a service entry while using "location: MESH_EXTERNAL" do we give the endpoints option?
And the rate_limit_cluster is appearing coz the previously applied filter where we didn't use "Context" option is applied . If i remove that filter it would disappear here

@ramaraochavali - when i add with endpoint like you said we get an entry for our endpoint in the  config_dump of the Gateway
Also when i run istioctl proxy-config route this is the output we get. Do we expect our endpoint to also be displayed here as a route ?
    {
        "name": "http.80",
        "virtualHosts": [
            {
                "name": "blackhole:80",
                "domains": [
                    "*"
                ],
                "routes": [
                    {
                        "name": "default",
                        "match": {
                            "prefix": "/"
                        },
                        "directResponse": {
                            "status": 404
                        }
                    }
                ]
            }
        ],
        "validateClusters": false
    }
]

I also get this entry in proxy-config cluster dump
        "name": "outbound|81||vpce-xxxxxxxxxxxxxxxxxxx.us-east-1.vpce.amazonaws.com",
        "type": "STRICT_DNS",
        "connectTimeout": "1s",
        "loadAssignment": {
            "clusterName": "outbound|81||vpcexxxxxxxxxxxxxxxxxx.us-east-1.vpce.amazonaws.com",
            "endpoints": [
                {
                    "locality": {},
                    "lbEndpoints": [
                        {
                            "endpoint": {
                                "address": {
                                    "socketAddress": {
                                        "address": "vpce-xxxxxxxxxxxxxxxxxxxxx.us-east-1.vpce.amazonaws.com",
                                        "portValue": 81
                                    }
                                }
                            },
                            "loadBalancingWeight": 1
                        }
                    ],
                    "loadBalancingWeight": 1
                }
            ]
        },

OK. Can you try giving this cluster outbound|81||vpcexxxxxxxxxxxxxxxxxx.us-east-1.vpce.amazonaws.com in the rate limit filter?

We Tried Giving that in the Cluster section of the Filter. Could you please confirm if this is the correct config you need? we are not getting the call forwarded to our ratelimit server yet
    - applyTo: CLUSTER
      match:
        cluster:
          service: outbound|81||vpce-xxxxxxxxxxxxxxxxxxxxxxxx.us-east-1.vpce.amazonaws.com
      patch:
        operation: ADD
        value:
          name: rate_limit_cluster
          type: STRICT_DNS
          connect_timeout: 10s
          lb_policy: ROUND_ROBIN
          http2_protocol_options: {}
          hosts:
            - socket_address:
                address: vpce-xxxxxxxxxxxxxxxxxxxxxxxx.us-east-1.vpce.amazonaws.com
                port_value: 81

There is no need to add this cluster patch. Please use the cluster in the rate limit filter directly

@ramaraochavali - Could you please verify the below filter config
kind: EnvoyFilter
metadata:
  name: filter-ratelimit
  namespace: istio-system
spec:
  workloadSelector:
    # select by label in the same namespace
    labels:
      istio: ingressgateway
  configPatches:
    # The Envoy config you want to modify
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: "envoy.http_connection_manager"
              subFilter:
                name: "envoy.router"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.rate_limit
          config:
            # domain can be anything! Match it to the ratelimter service config
            domain: abcdefghi.xxxxxx.com
            failure_mode_deny: true
            rate_limit_service:
              grpc_service:
                envoy_grpc:
                  cluster_name: outbound|81||vpce-xxxxxxxxxxxxxxxxxx.us-east-1.vpce.amazonaws.com
                timeout: 10s
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: filter-ratelimit-svc
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
    - applyTo: HTTP_ROUTE
      match:
        context: GATEWAY
        routeConfiguration:
          vhost:
            name: abcdefghi.xxxxxx.com
            route:
              action: ANY
      patch:
        operation: MERGE
        value:
          route:
            rate_limits:
              - actions:
                  - {request_headers: {header_name: "method", descriptor_key: "GET"}}
                  - {request_headers: {header_name: "path", descriptor_key: "/path/"}}
                  - {request_headers: {header_name: "host", descriptor_key: "abcdefghi.xxxxxx.com"}}
                  - {request_headers: {header_name: "x-request-id", descriptor_key: "ac5b684b-4bc6-4474-a943-0de4f1faf8df"}}
                  - {request_headers: {header_name: "domain", descriptor_key: "XYZ"}}

Can you share your gateway envoy logs?

The ingress gateway logs only show the below and it reaches the backend service :
[2020-04-16T05:58:42.837Z] "GET /api/v2/app-service/health HTTP/1.1" 200 - "-" "-" 10 40 xx 16 "115.112.XX.51,172.16.xxx.71" "curl/7.54.0" "5ff9d011-db2a-497a-xxxx-6f1ddeeabec8" "demo.xxxx.com" "172.16.XX.45:8080" outbound|80|v1|app-service.namespace.svc.cluster.local 172.16.XX.4:44398 172.16.XX.4:80 172.16.XXX.71:42798 - -

@ramaraochavali @rshriram This is our analysis on this issue

At first, we checked proxy status

istioctl proxy-status istio-ingressgateway-598796f4d9-xm8hn.istio-system

o/p:
Clusters Match
Listeners Match
Routes Match (RDS last loaded at Wed, 15 Apr 2020 08:07:33 UTC)

**You can find RDS is not updated at all for applying the above Envoyfilter & ServiceEntry manifesthttps://github.com/istio/istio/issues/22536#issuecomment-614050491

we checked CDS

istioctl proxy-config clusters istio-ingressgateway-598796f4d9-xm8hn.istio-system

o/p:
vpce-xxxxx-xxxx.vpce-svc-xxxx.us-east-1.vpce.amazonaws.com     81        -          outbound      STRICT_DNS

we checked RDS

istioctl proxy-config routes istio-ingressgateway-598796f4d9-xm8hn.istio-system

o/p:
NOTE: This output only contains routes loaded via RDS.
NAME        VIRTUAL HOSTS
http.80     1
                  1

*** Not sure one virtual hosts shows without any NAME associated. does it also belong to http.80?

Checked the config using this command

istioctl proxy-config routes istio-ingressgateway-598796f4d9-xm8hn.istio-system -o json

o/p:
It does not contain cluster specific to Service Entry(VPC Endpoint Svc). Contains only INTERNAL services

is this expected one? if so where/any command to see list of external service registered in Istio?
(like kubectl get endpoints). If no routes present, how can istio forward the traffic to external svc?

We suspected resolution: DNS in our ServiceEntry yaml as we don't have any domain entry for this(since it is a internal VPC endpoint service only not available in DNS) but it shows the following failure
Error from server: error when creating "serviceentry.yaml": admission webhook "validation.istio.io" denied the request: configuration is invalid: vpce-0b247209ae0145d88-4fa54j71.vpce-svc-0874c1e9512bd57dc.us-east-1.vpce.amazonaws.com is not a valid IP

We also would love to know any debugging statements can be added in this type of envoyfilters to check whether the request is successfully posted or not and also get the response back and print it here?

@ramaraochavali It would be great if you can quickly help on this as it blocked us for further progress

Sorry for the delay.

If no routes present, how can istio forward the traffic to external svc?

If you are referring to rate limiting service here, it is called by Envoy directly based on the cluster provided in the filter configuration. So no routes are required.
Does your service run on TLS?
I need to get full envoy debug logs + config_dump +stats to debug further. The filter looks fine to me and the cluster details etc that provided seem right to me.

Thanks for your response @ramaraochavali

Does your service run on TLS?
 Kindly check this data and share your suggestions

@VinothChinnadurai  The debug log just shows some Lua calls.
And in the config_dump, I do not see envoy.rate_limit filter and cluster outbound|81||vpce-xxxxxxxxxxxxxxxxxx.us-east-1.vpce.amazonaw for your rate limit service.
Did you apply the service entry and envoy filter correctly? Do you see any errors in pilot logs?

@ramaraochavali i can find it here: https://gist.github.com/VinothChinnadurai/a260e5f4e648c9ab4b93202558dde023#file-gistfile1-txt-L7871 in the config_dump.
Did you apply the service entry and envoy filter correctly? Do you see any errors in pilot logs?

We applied it and there was no error thrown while applying it.
The debug log just shows some Lua calls

Even we cannot find anything specific to external DNS(rate limiter) service call here.. do we need to enable or set any here?

Sorry, I do see the config related to envoy filter and cluster.
Is that the complete log? Is your request successful? It seems it is after Lua filter nothing is executed. Can you please enable debug log of proxy and share it?

Please let me know how to enable debug log of proxy? currently we set only  access log
--set values.global.proxy.accessLogFile="/dev/stdout"

while installing Istio

For a pod, set annotation: sidecar.istio.io/logLevel: "debug"
For the whole mesh, install with --set values.global.proxy.logLevel=debug

Thanks for your quick response.. I tried the same  by editing istio-ingress gateway pod and added this annotation 'sidecar.istio.io/logLevel: "debug"'. But still nothing appear specific to ratelimitter call in debug log 'kubectl logs -f istio-ingressgateway-598796f4d9-xm8hn -n istio-system'
[2020-04-20T16:48:52.249Z] "GET /api/v2/activities HTTP/1.1" 503 UF,URX "-" "-" 0 91 104 - "xx.xx.xx.xx" "curl/7.61.1" "42baf8fe-6564-4956-8f30-b83ace56c979" "abcdefghi.freshpo.com" "xx.xx.xx.xx:8080" outbound|80||twilight-service.twilight-istio.svc.cluster.local - <IP_ADDRESS>:80 xx.xx.xx.xx:42596 - -
[2020-04-20T16:49:33.469Z] "GET /api/v2/activities HTTP/1.1" 503 UF,URX "-" "-" 0 91 34 - "xx.xx.xx.xx" "curl/7.61.1" "87eefa8a-27c8-4d9c-8fa7-8bb54fb279ed" "abcdefghi.freshpo.com" "xx.xx.xx.xx:8080" outbound|80||twilight-service.twilight-istio.svc.cluster.local - <IP_ADDRESS>:80 xx.xx.xx.xx:42664 - -

We cant make sure that ratelimitter call is really happening or skipping? i can see envoyfilter in list envoyfilter call(kubectl get envoyfilter --all-namespaces)
Please suggest

Can you please share entire log if possible?

@ramaraochavali - hope you are asking for the ingress-gateway logs?
We are only seeing the entries for the lua filter that we applied at the Gateway and the log of the request reaching the backend service like what vinoth had showed above.
The only different log we see is this " gRPC config stream closed: 13,"
would like to know what this is ?
The entire logs you can find it here :
https://gist.github.com/Adharsh-Muraleedharan/e6219c0661fd4e005b7d28d149e9eb76

" gRPC config stream closed: 13," would like to know what this is ?

This is not a problem.  This happens when pilot disconnects proxies (which periodically happens) to rebalance the load.
I can not see any debug logs from Ingress gateway Envoy.  We should see lot more logs with debug.

@ramaraochavali - In the pod config for instio-ingressgateway we can see the option like "sidecar.istio.io/logLevel: debug" being set.
we are printing out the logs using "kubectl logs -f  instio-ingressgateway-xxxxxxx -n istio-system"
Is this how you expect us to fetch the logs ? or should we add something else
The entire pod configuration is given below :
kind: Pod
metadata:
  annotations:
    kubernetes.io/psp: eks.privileged
    sidecar.istio.io/inject: "false"
    sidecar.istio.io/logLevel: debug
  creationTimestamp: "2020-04-14T04:40:38Z"
  generateName: istio-ingressgateway-598796f4d9-
  labels:
    app: istio-ingressgateway
    chart: gateways
    heritage: Tiller
    istio: ingressgateway
    pod-template-hash: 598796f4d9
    release: istio
    service.istio.io/canonical-name: istio-ingressgateway
    service.istio.io/canonical-revision: "1.5"
  name: istio-ingressgateway-598796f4d9-xm8hn
  namespace: istio-system
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: istio-ingressgateway-598796f4d9
    uid: 6203fca1-7d9a-11ea-bcfc-0e6d31531fe3
  resourceVersion: "28456133"
  selfLink: /api/v1/namespaces/istio-system/pods/istio-ingressgateway-598796f4d9-xm8hn
  uid: 17068003-7e0a-11ea-bcfc-0e6d31531fe3
spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - preference:
          matchExpressions:
          - key: beta.kubernetes.io/arch
            operator: In
            values:
            - amd64
        weight: 2
      - preference:
          matchExpressions:
          - key: beta.kubernetes.io/arch
            operator: In
            values:
            - ppc64le
        weight: 2
      - preference:
          matchExpressions:
          - key: beta.kubernetes.io/arch
            operator: In
            values:
            - s390x
        weight: 2
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: beta.kubernetes.io/arch
            operator: In
            values:
            - amd64
            - ppc64le
            - s390x
  containers:
  - args:
    - proxy
    - router
    - --domain
    - $(POD_NAMESPACE).svc.cluster.local
    - --proxyLogLevel=warning
    - --proxyComponentLogLevel=misc:error
    - --log_output_level=default:info
    - --drainDuration
    - 45s
    - --parentShutdownDuration
    - 1m0s
    - --connectTimeout
    - 10s
    - --serviceCluster
    - istio-ingressgateway
    - --zipkinAddress
    - zipkin.istio-system:9411
    - --proxyAdminPort
    - "15000"
    - --statusPort
    - "15020"
    - --controlPlaneAuthPolicy
    - NONE
    - --discoveryAddress
    - istio-pilot.istio-system.svc:15012
    - --trust-domain=cluster.local
    env:
    - name: JWT_POLICY
      value: third-party-jwt
    - name: PILOT_CERT_PROVIDER
      value: istiod
    - name: ISTIO_META_USER_SDS
      value: "true"
    - name: CA_ADDR
      value: istio-pilot.istio-system.svc:15012
    - name: NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: INSTANCE_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: HOST_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.hostIP
    - name: SERVICE_ACCOUNT
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.serviceAccountName
    - name: ISTIO_META_WORKLOAD_NAME
      value: istio-ingressgateway
    - name: ISTIO_META_OWNER
      value: kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-ingressgateway
    - name: ISTIO_META_MESH_ID
      value: cluster.local
    - name: ISTIO_AUTO_MTLS_ENABLED
      value: "true"
    - name: ISTIO_META_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: ISTIO_META_CONFIG_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: ISTIO_META_ROUTER_MODE
      value: sni-dnat
    - name: ISTIO_META_CLUSTER_ID
      value: Kubernetes
    image: docker.io/istio/proxyv2:1.5.0
    imagePullPolicy: IfNotPresent
    name: istio-proxy
    ports:
    - containerPort: 15020
      protocol: TCP
    - containerPort: 80
      protocol: TCP
    - containerPort: 443
      protocol: TCP
    - containerPort: 15029
      protocol: TCP
    - containerPort: 15030
      protocol: TCP
    - containerPort: 15031
      protocol: TCP
    - containerPort: 15032
      protocol: TCP
    - containerPort: 15443
      protocol: TCP
    - containerPort: 15011
      protocol: TCP
    - containerPort: 8060
      protocol: TCP
    - containerPort: 853
      protocol: TCP
    - containerPort: 15090
      name: http-envoy-prom
      protocol: TCP
    readinessProbe:
      failureThreshold: 30
      httpGet:
        path: /healthz/ready
        port: 15020
        scheme: HTTP
      initialDelaySeconds: 1
      periodSeconds: 2
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: "2"
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 128Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/istio
      name: istiod-ca-cert
    - mountPath: /var/run/secrets/tokens
      name: istio-token
      readOnly: true
    - mountPath: /var/run/ingress_gateway
      name: ingressgatewaysdsudspath
    - mountPath: /etc/istio/pod
      name: podinfo
    - mountPath: /etc/istio/ingressgateway-certs
      name: ingressgateway-certs
      readOnly: true
    - mountPath: /etc/istio/ingressgateway-ca-certs
      name: ingressgateway-ca-certs
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: istio-ingressgateway-service-account-token-rrjq5
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: ip-172-16-57-15.ec2.internal
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: istio-ingressgateway-service-account
  serviceAccountName: istio-ingressgateway-service-account
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - configMap:
      defaultMode: 420
      name: istio-ca-root-cert
    name: istiod-ca-cert
  - downwardAPI:
      defaultMode: 420
      items:
      - fieldRef:
          apiVersion: v1
          fieldPath: metadata.labels
        path: labels
      - fieldRef:
          apiVersion: v1
          fieldPath: metadata.annotations
        path: annotations
    name: podinfo
  - emptyDir: {}
    name: ingressgatewaysdsudspath
  - name: istio-token
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          audience: istio-ca
          expirationSeconds: 43200
          path: istio-token
  - name: ingressgateway-certs
    secret:
      defaultMode: 420
      optional: true
      secretName: istio-ingressgateway-certs
  - name: ingressgateway-ca-certs
    secret:
      defaultMode: 420
      optional: true
      secretName: istio-ingressgateway-ca-certs
  - name: istio-ingressgateway-service-account-token-rrjq5
    secret:
      defaultMode: 420
      secretName: istio-ingressgateway-service-account-token-rrjq5
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2020-04-14T04:40:38Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2020-04-14T04:40:41Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2020-04-14T04:40:41Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2020-04-14T04:40:38Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://c06c525a7e0c1e94c926bdca2ecfb41a735e4700b19f4b5f4456cc742768ce57
    image: istio/proxyv2:1.5.0
    imageID: docker-pullable://istio/proxyv2@sha256:89b5fe2df96920189a193dd5f7dbd776e00024e4c1fd1b59bb53867278e9645a
    lastState: {}
    name: istio-proxy
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: "2020-04-14T04:40:40Z"
  hostIP: <IP_ADDRESS>
  phase: Running
  podIP: <IP_ADDRESS>
  qosClass: Burstable
  startTime: "2020-04-14T04:40:38Z"

I have n't looked at in detail so not sure why it is not working - may be you can get to ingress pod and run this https://www.envoyproxy.io/docs/envoy/latest/operations/admin#post--logging in the envoy container.

@ramaraochavali
curl -X POST http://localhost:15000/logging
active loggers:
  admin: warning
  aws: warning
  assert: warning
  backtrace: warning
  client: warning
  config: warning
  connection: warning
  conn_handler: warning
  decompression: warning
  dubbo: warning
  file: warning
  filter: warning
  forward_proxy: warning
  grpc: warning
  hc: warning
  health_checker: warning
  http: warning
  http2: warning
  hystrix: warning
  init: warning
  io: warning
  jwt: warning
  kafka: warning
  lua: warning
  main: warning
  misc: error
  mongo: warning
  quic: warning
  quic_stream: warning
  pool: warning
  rbac: warning
  redis: warning
  router: warning
  runtime: warning
  stats: warning
  secret: warning
  tap: warning
  testing: warning
  thrift: warning
  tracing: warning
  upstream: warning
  udp: warning
  wasm: warning

Shall we pass the 'level=debug' as query param?

@ramaraochavali  set the query param as 'level=debug' and the output is here.
Please find it here.. we are hitting abcdefghi.xxx.com
https://gist.github.com/VinothChinnadurai/6fbacfe39c88bf3ce4de454c6f706c57

Can you tell me the request path you are trying to hit so that I can search in the logs?

curl -X GET http://microservices.test.internal/api/v2/activities --header "host:abcdefghi.xxxx.com"

hmm..at high level look, it seems it is directly trying to hit your service and calls is failing with TLS error. Will have to look take a look again. Will let you know

Sure @ramaraochavali . Thanks for all the followup responses.

I do not see the following rate limit descriptors applied to the route - that is why Envoy is not calling the rate limit service. Can you please check that part of the Envoy and ensure that they are added?
 patch:
        operation: MERGE
        value:
          route:
            rate_limits:
              - actions:
                  - {request_headers: {header_name: "method", descriptor_key: "GET"}}
                  - {request_headers: {header_name: "path", descriptor_key: "/path/"}}
                  - {request_headers: {header_name: "host", descriptor_key: "abcdefghi.xxxxxx.com"}}
                  - {request_headers: {header_name: "x-request-id", descriptor_key: "ac5b684b-4bc6-4474-a943-0de4f1faf8df"}}
                  - {request_headers: {header_name: "domain", descriptor_key: "XYZ"}}

Sorry @ramaraochavali . Where can we check this route? is it in envoy config dump?

@ramaraochavali - we changed the filter config
routeConfiguration:
portNumber: 80
vhost:
name: "*:80"
and applied but the config dump still shows the old config. we suspect that the istio pilot is not pushing the changes.
1. what is the command to manually sync the changes?
2. Anyway we would like to know the route config with virtual host as "*:80" is proper ?
the istioctl proxy-status results:
NAME                                                   CDS          LDS          EDS          RDS          PILOT                      VERSION
fieldservice-api-b6b6476fd-tq6wd.fieldservice          NOT SENT     SYNCED       SYNCED       SYNCED       istiod-7d9c7bdd6-h28z8     1.5.0
fieldservice-api-b6b6476fd-xjzsp.fieldservice          NOT SENT     SYNCED       SYNCED       SYNCED       istiod-7d9c7bdd6-h28z8     1.5.0
fieldservice-kafka-6b865748db-8bw9k.fieldservice       NOT SENT     SYNCED       SYNCED       SYNCED       istiod-7d9c7bdd6-h28z8     1.5.0
fieldservice-kafka-6b865748db-cnjnp.fieldservice       NOT SENT     SYNCED       SYNCED       SYNCED       istiod-7d9c7bdd6-h28z8     1.5.0
fieldservice-kafka-6b865748db-wxxz8.fieldservice       NOT SENT     SYNCED       SYNCED       SYNCED       istiod-7d9c7bdd6-h28z8     1.5.0
istio-ingressgateway-598796f4d9-xm8hn.istio-system     NOT SENT     SYNCED       SYNCED       SYNCED       istiod-7d9c7bdd6-h28z8     1.5.0
prometheus-b47d8c58c-k62lf.istio-system                NOT SENT     SYNCED       SYNCED       SYNCED       istiod-7d9c7bdd6-h28z8     1.5.0
symphony-api-86bd57f799-569k9.symphony                 NOT SENT     SYNCED       SYNCED       SYNCED       istiod-7d9c7bdd6-h28z8     1.5.0
symphony-db-migration-tj7mt.symphony                   SYNCED       NOT SENT     NOT SENT     NOT SENT     istiod-7d9c7bdd6-h28z8     1.5.0

@ramaraochavali We are waiting for your response. Kindly help us once you are available.

@VinothChinnadurai Sorry, I am busy today and won't have time to look at this config. I can probably take a look tomorrow.

Sure @ramaraochavali . Thanks for your response and we are waiting :)

@ramaraochavali sorry for multiple threads. Once you are available. Please check the route configuration and let us know what we are missing?

@VinothChinnadurai No problem. Sorry. I was about to send a message - I am little busy right now and I could not immediately figure out what is problem in applying the route rate limits via Envoy filter.
tagging @rshriram @hzxuzhonghu  if they can find some thing immediately - otherwise I will take a look when I get free

@VinothChinnadurai  My wild guess is you may be missing the route name -
           route:
              name: http-echo-service
              action: ANY

Good that you have posted here https://github.com/istio/istio/issues/22068 - they might have better idea as they were able to make it work

Sure.. Thanks @ramaraochavali for getting back to us.. We will try out this solution as well

@ramaraochavali thanks for all your responses. Really would like to thank a lot :) we have solved this problem!

@VinothChinnadurai Glad that you are able to make this work !

Would you folks be willing to document the envoy filter thing in our wiki for others to use ?

Would you folks be willing to document the envoy filter thing in our wiki for others to use ?

Sure, if you can resolve my issue. ~_~
https://github.com/istio/istio/issues/25193

looks like this is resolved. Note we are tracking better support for rate limit in https://github.com/istio/istio/issues/20952


Radio Buttons not working.

Do you have an example of a working radio button group?  I have tried names on radio inputs, id's on inputs and an id on fieldset. Nothing seems to work. Your documentation says is support radio buttons. Am I just missing something or is this still a known issue?  Everything else seems to work perfectly and I really want to continue using validate.js.

It works fine, except for one issue I haven't worked out yet.
The radio buttons with the same name are part of a set, so you just refer to that name in the config.
name: 'radio_button_name',
display: 'Radio button needs to be picked',
rules: 'required'


moving poseidon constants to icicle/hash/poseidon_constants

Describe the changes
moving poseidon constants to icicle/include/icicle/hash/poseidon_constants
Linked Issues
cuda-backend-branch: poseidon-constants-displacement

Approved


getFile method fails to generate a valid url

For some weird reason my bot stopped working after couple of tries.
Base on API documentation for getfile I should get a valid url for at least one hour for any file below 20mb and I don't see any more limitation on this method.
However I'm getting 404 response after the second try for each file format.
This issue started happening after couple of hours of testing my bot.
At the beginning I was just testing my bot with animation files ( GIFs ) and after 1-2 hours of testing which I've downloaded ~20 files it stopped working.
After that I tried with a photo and it worked for the first file and failed for further attempts.
Same thing happened to audio and video files, so after the first successful attempt it started giving 404.
I tried following solutions:

Using both http (http://api.telegram.org/file/bot<token>/<file_path>) and https (https://api.telegram.org/file/bot<token>/<file_path>)
Revoking my token and tried with a new token.
Creating a new bot and using its token
Rebooting my modem to get a new public IP ( in case it was somehow blacklisted/limited )
Using another device on a different network

But none of these changed anything.
Any idea what could be the reason ?

How have you fixed the problem? Have you figured out what it was?

@el-lsan
Facing the same problem.
Were you able to find out a solution for this?
Thanks.

@WaffleLapkin @arladmin
This was happening because I was calling deletemesssage method before download was made/finished.
so apparently if you remove your message the bot can't access to the file anymore.

Facing the same problem.
I always get 404. File is not saved on my local machine after I called getFile method. what should I do?

http://localhost:8081/file/botToken/documents/file_2

@malikovss The file is saved in the bots folder in the server's working directory.


Redesign

This is a quick design for the page, closely mirroring the actual Noun Project site. It should make it a bit easier to navigate. I've also changed the search term's querystring name from term to q, since that's a bit more standard for the search value.

Amazing! thank you very much for your contributions!
Initially this project was a fast simple app to search only public domain icons :)
I'll add a AUTHORS file mention you


Can install pinpoint behind proxy

Hi,
Our environment is behind proxy and while trying to build, we fail on trying to access maven url. Can we just copy the whole installation, unpack it and run, without need to build it?

hi @Benzion18
This is possible by installing in Maven offline mode after copy whole library in mave local repository.
But I am not sure that installing NPM is possible.
Simple way is that install Pinpoint another environment
And copy collector and web jar to your environment.
thanks :)

without need to build it?

Simply. use the files in the release

Hi,
I found how to work behind proxy. In mvnw file add this line:
export MAVEN_OPTS="-Dhttp.proxyHost= -Dhttp.proxyPort= -Dhttps.proxyHost= -Dhttps.proxyPort= "
But it failed further because the build requires java 9 and I have a problem to install it on my env.

@Benzion18

But it failed further because the build requires java 9 and I have a problem to install it on my env.

Just to let you know.
Yes, you will need java 9 to build it. But you won't need java 9 to use it.


"Exit Steam VR" doesn't quit Steam VR, instead is restarted.

Maybe quest 2 has an SDK that captures "exit" event and prevents to steam VR to restart?
Just thinking aloud.

Mmm couldn't find any Quest 2 SDK/API for doing that, but I was thinking that we can same the timestamp in a text file after the WaitForExit() method, so the next time the app is called we can prevent steam VR execution if the time passed from the timestamp of that file doesn't exceed nnn milliseconds/seconds.

I mean you could close the oculus app first and then SteamVR.

I have this same issue

Same issue, up


Error cpu:0

Hello
when running "python Feature_extraction.py --input_list script/vertex_path_GT.txt --model_path ../Feature_extraction/model_10000.ckpt"
I have many errors about Cpu:0 such as:
2020-12-22 23:24:43.600425: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv2/bn/cond/ExponentialMovingAverage: (NoOp): /job:localhost/replica:0/task:0/device:CPU:0
2020-12-22 23:24:43.600431: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv2/bn/cond/control_dependency: (Identity): /job:localhost/replica:0/task:0/device:CPU:0

Please provide more information about your bug.
Is the environment ok for GPU running? Maybe a simple test can bring more clues.

I run on windows
WARNING:tensorflow:From F:\DJ-RN-master\DJ-RN-master\pointnet\models\transform_nets.py:41: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.
WARNING:tensorflow:Variable += will be deprecated. Use variable.assign_add if you want assignment to the variable value or 'x = x + y' if you want a new python Tensor object.
WARNING:tensorflow:Variable += will be deprecated. Use variable.assign_add if you want assignment to the variable value or 'x = x + y' if you want a new python Tensor object.
WARNING:tensorflow:From Feature_extraction.py:52: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.
WARNING:tensorflow:From Feature_extraction.py:55: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.
WARNING:tensorflow:From Feature_extraction.py:59: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
2020-12-23 12:42:24.312104: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2020-12-23 12:42:24.502234: I tensorflow/core/common_runtime/direct_session.cc:296] Device mapping:
WARNING:tensorflow:From C:\Users\avajang\Anaconda3\lib\site-packages\tensorflow\python\training\saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
2020-12-23 12:42:24.907703: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/ExpandDims: (ExpandDims)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.907737: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/weights/Initializer/random_uniform/RandomUniform: (RandomUniform)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.907746: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/weights/Initializer/random_uniform/sub: (Sub)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.907753: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/weights/Initializer/random_uniform/mul: (Mul)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.907760: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/weights/Initializer/random_uniform: (Add)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.907766: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/weights: (VariableV2)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.907772: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/weights/Assign: (Assign)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.907778: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/weights/read: (Identity)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.907785: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/L2Loss: (L2Loss)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.907792: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/weight_loss: (Mul)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.907799: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/Conv2D: (Conv2D)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.907806: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/biases: (VariableV2)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.907812: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/biases/Assign: (Assign)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.907818: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/biases/read: (Identity)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.907825: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/BiasAdd: (BiasAdd)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.907893: W tensorflow/core/common_runtime/colocation_graph.cc:1016] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:1' assigned_device_name_='' resource_device_name_='/device:GPU:1' supported_device_types_=[CPU] possible_devices_=[]
VariableV2: CPU
Assign: CPU
Identity: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
transform_net1/tconv1/bn/beta (VariableV2) /device:GPU:1
transform_net1/tconv1/bn/beta/Assign (Assign) /device:GPU:1
transform_net1/tconv1/bn/beta/read (Identity) /device:GPU:1
save/Assign_43 (Assign) /device:GPU:1
2020-12-23 12:42:24.907912: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/bn/beta: (VariableV2)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.907919: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/bn/beta/Assign: (Assign)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.907925: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/bn/beta/read: (Identity)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.907957: W tensorflow/core/common_runtime/colocation_graph.cc:1016] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:1' assigned_device_name_='' resource_device_name_='/device:GPU:1' supported_device_types_=[CPU] possible_devices_=[]
VariableV2: CPU
Assign: CPU
Identity: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
transform_net1/tconv1/bn/gamma (VariableV2) /device:GPU:1
transform_net1/tconv1/bn/gamma/Assign (Assign) /device:GPU:1
transform_net1/tconv1/bn/gamma/read (Identity) /device:GPU:1
save/Assign_44 (Assign) /device:GPU:1
2020-12-23 12:42:24.907969: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/bn/gamma: (VariableV2)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.907976: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/bn/gamma/Assign: (Assign)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.907982: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/bn/gamma/read: (Identity)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.907989: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/bn/moments/mean: (Mean)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.907995: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/bn/moments/StopGradient: (StopGradient)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.908003: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/bn/moments/SquaredDifference: (SquaredDifference)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.908010: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/bn/moments/variance: (Mean)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.908017: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/bn/moments/Squeeze: (Squeeze)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.908024: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/bn/moments/Squeeze_1: (Squeeze)/job:localhost/replica:0/task:0/device:CPU:0
2020-12-23 12:42:24.908031: I tensorflow/core/common_runtime/placer.cc:54] transform_net1/tconv1/bn/cond/Switch: (Switch)/job:localhost/replica:0/task:0/device:CPU:0

It looks that your TF environment is not ready or not successfully complied, please ensure your TF is OK first.

Ok. Thanks


.Draw displays 0s in place of all letters when in projects with Scribble Deluxe

When Jr and Deluxe share a project, Jr's .Draw() displays only "0"s in place of every letter/ glyph.
A work around seems to be renaming "Draw" in these 2 lines. I changed them to lowercase just to try.
__ScribblejrClassWrapper.gml
__element.draw(_x, _y, _colour, _alpha, _sdfEffects);
__ScribblejrClass.gml
draw = (_fontScale == 1)? __Draw : __DrawScale;
I've had this problem for some time but I never looked into it until now. I saw it sometime around 1.3 but figured it was Beta so I'd come back later. Trying 1.5 It produced the same results. In a fresh 2024.8 project with only Jr it's fine, but once I added deluxe it would no longer render properly.
Feather also seems to throw a fit about .Draw() expecting 0 arguments.
Thanks!

You can safely ignore Feather as a general rule.
I'm currently using Scribble Jr and Scribble Deluxe in the same project (Sports Sports) and things are working fine. What version of Scribble Deluxe are you using?

I'm using Deluxe 9.3.1 but it also happened with whatever versions were current while Jr was in 1.3.
I opened up this blank project, removed both, re-added them, and Jr started working normally. I don't know why it was give the same 0s but it went away.
I added both to an unrelated test project and it worked without a problem.
Thinking maybe this was it, I removed and added them both to my main project, but that didn't work. On the first use of Jr's Draw() anywhere in the project it momentarily displays the string then changes to 0s. Doesn't matter if it's static text to show or not, or if it has any arguments.
I can't find any conflicting names or anything that might be causing it. I might have to rename those 2 lines and accept that my project is bizarre, or not use it.

I'll try to reproduce the issue my end. If you can send me a reproducing project that would help. I suspect the issue is a globally scoped variable etc. is interfering with the internal workings of Scribble Jr.

I couldn't find any conflicting global variables or macros. Since it works fine in other projects and renaming fixes it, we'll just assume I have a weird project lol. Thanks for your help.


Questions about int8 quantification.

Dear authors,
I am a beginner to the project. And I check the code in "include/rwkv/cuda/rwkv.cu". If my understanding is correct, only the computation inside functions cudac_mm8_one and cuda_mm8_threec are related to int8 quantification and the results are float point numbers. But the calculation in sigmoid and kernel_wkvc_forward are done in float point numbers.
My question is why are these parts not quantified? I have heard of some methods which can quantify the non-linear function with a look-up table. Considering the low speed of exp() function. Is there any methods to replace them with fast substitution?
Best,
zzczzc20

Hi!
To answer your question: it's a matter of scale.
The mm8 matvec are n^2 operations, so int8 is used to minimise cuda memory usage.
There's no appreciable memory advantage to quantizing other operations.
If you can get faster inference using the methods outlined, please submit a PR :)

Hi, I am willing to look into it. But I need to benchmark the model so that I can measure the quant loss. Is there any easy method to benchmark the model? Which benchmark do you use? Thanks very much.


Fullscreen Action.

Adding Toggle Full Screen to browser. It uses layoutService.container as the root element for full screen, which behaves the same as VS Code now. Keyboard lock works as expected no matter which DOM element requests full screen.
Once in full screen mode, all unbound keybindings are brought back.

@bpasero

now see warnings printed to the console even when running with electron (e.g. keybindingService.js:296 Ctrl/Cmd+Num keybindings should not be used by default in web. Offender: 000123 for workbench.action.focusSecondEditorGroup)

It's a regression of leaking web into Code itself. Fixed.

when running from web and going fullscreen, I cannot just copy/paste in the editor anymore for some reason, works fine when not being in fullscreen 😩. also possibly related, Cmd+A no longer selects all in input fields?

Now we only lock Escape and since it's in full screen mode, we can still hijack all keyboard events (except Escape).

@sbatten reviewed the code change so I merged it first to unblock him fix menu bar issues. If the code change still has some bad smell, we can fix it.

@rebornix the fullscreen state from WindowManager is incorrect when the user uses the browser to enter fullscreen. On Windows, this is a reasonable method and it seems we aren't intercepting F11 so our command is not run. We need to listen for these events and keep the WindowManager state up to date.

@rebornix this works crazy good!! how did you fix the issue with Cmd+A not working, I wonder?

@bpasero it wasn't working because keyboard lock was locking all keys to the app and messing with normal functions. Now we only lock escape

Awesome 👍


Compilation failed

steps：
mkdir -p build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j4
problem：

Finally, execute make and report the following error

supplement ： centos7.6   gcc5.4  cmake3.15.3   openssl1.1.1

Hi,
You need at least GCC 9 for latest arangodbs.
We use static builds with libmusl; The alpine build containers that bring these tools can be found here:
https://github.com/arangodb/oskar/tree/master/containers

嗨，
您至少需要GCC 9才能安装最新的arangodb。
我们在libmusl中使用静态版本；可以在这里找到带来这些工具的高山构建容器：https :
//github.com/arangodb/oskar/tree/master/containers

hello
Use as follows，This problem occurs
https://github.com/arangodb/oskar/tree/master/containers/buildUbuntu4.docker

Request assistance

Hi,
the actual error is much above; please re-run make and cut'n'paste the error.
Hint: you don't need a screenshot, just put tripple-backticks around the text you paste - `
like
this

Hi,
the actual error is much above; please re-run make and cut'n'paste the error.
Hint: you don't need a screenshot, just put tripple-backticks around the text you paste - `
like
this

In file included from ../../../../3rdParty/V8/v7.9.317/../v7.9.317/src/snapshot/embedded/embedded-file-writer.cc:5:
../../../../3rdParty/V8/v7.9.317/../v7.9.317/src/snapshot/embedded/embedded-file-writer.h:28:7: warning: 'class v8::internal::EmbeddedFileWriterInterface' has virtual functions and accessible non-virtual destructor [-Wnon-virtual-dtor]
28 | class EmbeddedFileWriterInterface {
|       ^~~~~~~~~~~~~~~~~~~~~~~~~~~
../../../../3rdParty/V8/v7.9.317/../v7.9.317/src/snapshot/embedded/embedded-file-writer.h:55:7: warning: base class 'class v8::internal::EmbeddedFileWriterInterface' has accessible non-virtual destructor [-Wnon-virtual-dtor]
55 | class EmbeddedFileWriter : public EmbeddedFileWriterInterface {
|       ^~~~~~~~~~~~~~~~~~
../../../../3rdParty/V8/v7.9.317/../v7.9.317/src/snapshot/embedded/embedded-file-writer.h:55:7: warning: 'class v8::internal::EmbeddedFileWriter' has virtual functions and accessible non-virtual destructor [-Wnon-virtual-dtor]
In file included from ../../../../3rdParty/V8/v7.9.317/../v7.9.317/src/snapshot/mksnapshot.cc:16:
../../../../3rdParty/V8/v7.9.317/../v7.9.317/src/snapshot/embedded/embedded-file-writer.h:28:7: warning: 'class v8::internal::EmbeddedFileWriterInterface' has virtual functions and accessible non-virtual destructor [-Wnon-virtual-dtor]
28 | class EmbeddedFileWriterInterface {
|       ^~~~~~~~~~~~~~~~~~~~~~~~~~~
../../../../3rdParty/V8/v7.9.317/../v7.9.317/src/snapshot/embedded/embedded-file-writer.h:55:7: warning: base class 'class v8::internal::EmbeddedFileWriterInterface' has accessible non-virtual destructor [-Wnon-virtual-dtor]
55 | class EmbeddedFileWriter : public EmbeddedFileWriterInterface {
|       ^~~~~~~~~~~~~~~~~~
../../../../3rdParty/V8/v7.9.317/../v7.9.317/src/snapshot/embedded/embedded-file-writer.h:55:7: warning: 'class v8::internal::EmbeddedFileWriter' has virtual functions and accessible non-virtual destructor [-Wnon-virtual-dtor]
LINK(target) /root/arangodb/build/3rdParty/V8/v7.9.317/arm64.release/mksnapshot
TOUCH fcf4bc5560c5ebd918bb4fa39879f42ca902cc99..intermediate
ACTION generating: "/root/arangodb/build/3rdParty/V8/v7.9.317/arm64.release/obj.target/icuuc/geni/snapshot.cc" ../v7.9.317/src/snapshot/embedded/embedded-empty.cc fcf4bc5560c5ebd918bb4fa39879f42ca902cc99..intermediate
CXX(target) /root/arangodb/build/3rdParty/V8/v7.9.317/arm64.release/obj.target/v8_snapshot/../v7.9.317/src/init/setup-isolate-deserialize.o
CXX(target) /root/arangodb/build/3rdParty/V8/v7.9.317/arm64.release/obj.target/v8_snapshot/../gypfiles/extras-libraries.o
CXX(target) /root/arangodb/build/3rdParty/V8/v7.9.317/arm64.release/obj.target/v8_snapshot/geni/snapshot.o
CXX(target) /root/arangodb/build/3rdParty/V8/v7.9.317/arm64.release/obj.target/v8_snapshot/../v7.9.317/src/snapshot/embedded/embedded-empty.o
AR(target) /root/arangodb/build/3rdParty/V8/v7.9.317/arm64.release/obj.target/../gypfiles/libv8_snapshot.a
[ 14%] No install step for 'v8_build'
[ 14%] Completed 'v8_build'
[ 14%] Built target v8_build
make: *** [Makefile:152: all] Error 2

supplement：
My server is ARM Architecture，
Because the official package or mirror does not support the ARM architecture，
So, I have been trying to compile on an ARM machine, and then it has been unsuccessful to compile

Hi,
yes, ARM is not official supported. For V8 you need at least an ARM v7 host (afaik) so default Raspian won't work at all.
Whats been failing on you here is the V8 compilation, its probably having a -WError in some of the ARM flags in the gyp files, hence the compilation aborts.
You can find out how CMake invokes gyp here:
https://github.com/arangodb/arangodb/blob/devel/3rdParty/V8/CMakeLists.txt#L425
The gyp control files (similar function to CMakeLists.txt) that control these compiler flags are here:
https://github.com/arangodb/arangodb/tree/devel/3rdParty/V8/gypfiles
They could also have linux/arm specific cases, most probably there -Werror needs to be removed.
The branching into the gyp created makefiles can be inspected by running make V=1 VERBOSE=1.

Hi,
I'm closing this as solved.
If any more questions arise, please let us know.


Strange UI after modifying input text

Testing #208409
Steps:

open inline chat
I typed add additional capabilities
Change the input field to
add additional capabilities and add assert library

I ended up with this:

I do like the basic idea of having a stack of request I did. But I think that it would be a better UI if I have previous / next button to cycle through them.

Change the input field to

So, you have pressed "send" after that, right?

Yes, by pressing enter.

I do like the basic idea of having a stack of request I did. But I think that it would be a better UI if I have previous / next button to cycle through them.

Tho, that doesn't represent what's happening. Inline chat is in fact multi-turn and all previous requests are send along with the last request. The UI never really showed this behaviour but behind the scenes it goes like "the following code is the result of the following message exchange. take that into account when answering to XYZ"


i-dot-ming: 7.01 -> 8.00

Description of changes
Also add updateScript.

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 191023 run on x86_64-linux 1

1 package built:

i-dot-ming

Thanks!


[refactor results 1] - refactor backward

Splitting a major refactor over many smaller PRs.
The end result is structured results

ready for review?

@sshleifer cleaned that up... if you guys are going to override it, you no longer need to worry about amp, etc...
but this hook is really meant for research into GANs or something where you might need to retain a graph or something


fix: Added metadata for miscellaneous models

I made a script to generate metadata for models from HF Hub, and added metadata for models in the keep category on this issue #1515

Oops, made a silly blunder

Let me know if I need to run this for more models @Samoed @KennethEnevoldsen


HADOOP-17609. Make SM4 support optional for OpenSSL native code.

https://issues.apache.org/jira/browse/HADOOP-17609
This replaces #2847.
After HDFS-15098, OpensslCipher does not work with OpenSSL >= 1.1.1 without SM4 support. RHEL/CentOS 8 provides such openssl package. The OpensslCipher on such environment should be usable if users do not need SM4 feature.
$ rpm -q openssl-devel
openssl-devel-1.1.1g-12.el8_3.x86_64

$ bin/hadoop checknative 2>/dev/null
Native library checking:
hadoop:  true /home/centos/dist/hadoop-3.4.0-SNAPSHOT/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
zstd  :  true /lib64/libzstd.so.1
bzip2:   true /lib64/libbz2.so.1
openssl: false Cannot find AES-CTR/SM4-CTR support, is your version of Openssl new enough?
ISA-L:   true /lib64/libisal.so.2
PMDK:    false The native code was built without PMDK support.

I manually tested this by hadoop checknative after -Pnative -Pdist build.
$ mvn clean install -DskipTests -DskipShade -Pnative -Pdist
$ hadoop-dist/target/hadoop-3.4.0-SNAPSHOT/bin/hadoop checknative
2021-05-17 11:19:03,102 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
2021-05-17 11:19:03,106 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2021-05-17 11:19:03,182 INFO nativeio.NativeIO: The native code was built without PMDK support.
Native library checking:
hadoop:  true /home/centos/srcs/hadoop/hadoop-dist/target/hadoop-3.4.0-SNAPSHOT/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
zstd  :  true /lib64/libzstd.so.1
bzip2:   true /lib64/libbz2.so.1
openssl: true /lib64/libcrypto.so
ISA-L:   true /lib64/libisal.so.2
PMDK:    false The native code was built without PMDK support.

I need additional fix to make falling back to JceSm4CtrCryptoCodec work. Tests should be fixed too.

On CentOS 8, only TestCryptoCodec#testOpensslSm4CtrCryptoCodec was skipped as expected.
$ mvn test -Pnative -Dtest=TestCryptoCodec,TestOpensslCipher
...
[INFO] Running org.apache.hadoop.crypto.TestCryptoCodec
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 11.538 s - in org.apache.hadoop.crypto.TestCryptoCodec
[INFO] Running org.apache.hadoop.crypto.TestOpensslCipher
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.143 s - in org.apache.hadoop.crypto.TestOpensslCipher

On Ubuntu 18.04, all tests of TestCryptoCodec were executed since SM4 is supported by OpenSSL.
$ mvn test -Pnative -Dtest=TestCryptoCodec,TestOpensslCipher
...
INFO] Running org.apache.hadoop.crypto.TestCryptoCodec
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.002 s - in org.apache.hadoop.crypto.TestCryptoCodec
[INFO] Running org.apache.hadoop.crypto.TestOpensslCipher
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.098 s - in org.apache.hadoop.crypto.TestOpensslCipher

On CentOS 8, OpensslAesCtrCryptoCodec is available while OpensslSm4CtrCryptoCodec is not.
$ bin/hadoop checknative
2021-05-17 13:02:39,196 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
2021-05-17 13:02:39,197 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library
2021-05-17 13:02:39,324 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
2021-05-17 13:02:39,327 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2021-05-17 13:02:39,393 INFO nativeio.NativeIO: The native code was built without PMDK support.
Native library checking:
hadoop:  true /home/centos/srcs/hadoop/hadoop-dist/target/hadoop-3.4.0-SNAPSHOT/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
zstd  :  true /lib64/libzstd.so.1
bzip2:   true /lib64/libbz2.so.1
openssl: true /lib64/libcrypto.so
ISA-L:   true /lib64/libisal.so.2
PMDK:    false The native code was built without PMDK support.

$ bin/hadoop key create key-sm4 -cipher 'SM4/CTR/NoPadding'
$ bin/hdfs dfs -mkdir /zone1
$ bin/hdfs crypto -createZone -path /zone1 -keyName key-sm4
$ bin/hdfs dfs -put README.txt /zone1/
2021-05-17 13:05:25,570 DEBUG util.PerformanceAdvisory: Crypto codec org.apache.hadoop.crypto.OpensslSm4CtrCryptoCodec is not available.
2021-05-17 13:05:25,570 DEBUG util.PerformanceAdvisory: Using crypto codec org.apache.hadoop.crypto.JceSm4CtrCryptoCodec.

$ bin/hadoop key create key-aes
$ bin/hdfs dfs -mkdir /zone2
$ bin/hdfs crypto -createZone -path /zone2 -keyName key-aes
$ bin/hdfs dfs -put NOTICE.txt /zone2/
2021-05-17 13:09:04,773 DEBUG crypto.OpensslAesCtrCryptoCodec: Using org.apache.hadoop.crypto.random.OpensslSecureRandom as random number generator.
2021-05-17 13:09:04,775 DEBUG util.PerformanceAdvisory: Using crypto codec org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec.

On Ubuntu 18.04, OpensslSm4CtrCryptoCodec was available and worked.
$ bin/hadoop key create key-sm4 -cipher 'SM4/CTR/NoPadding'
$ bin/hdfs dfs -mkdir /zone1
$ bin/hdfs crypto -createZone -path /zone1 -keyName key-sm4
$ bin/hdfs dfs -put README.txt /zone1/
2021-05-17 13:39:45,686 DEBUG crypto.OpensslSm4CtrCryptoCodec: Using org.apache.hadoop.crypto.random.OpensslSecureRandom as random number generator.
2021-05-17 13:39:45,687 DEBUG util.PerformanceAdvisory: Using crypto codec org.apache.hadoop.crypto.OpensslSm4CtrCryptoCodec.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 34s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
34m 21s

trunk passed

+1 :green_heart:
compile
20m 54s

trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
compile
18m 13s

trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

+1 :green_heart:
mvnsite
1m 33s

trunk passed

+1 :green_heart:
shadedclient
88m 58s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 52s

the patch passed

+1 :green_heart:
compile
20m  9s

the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04

-1 :x:
cc
20m  9s
/results-compile-cc-root-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt
root-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 45 new + 282 unchanged - 45 fixed = 327 total (was 327)

+1 :green_heart:
golang
20m  9s

the patch passed

+1 :green_heart:
javac
20m  9s

the patch passed

+1 :green_heart:
compile
18m 13s

the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

-1 :x:
cc
18m 13s
/results-compile-cc-root-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt
root-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 7 new + 320 unchanged - 7 fixed = 327 total (was 327)

+1 :green_heart:
golang
18m 13s

the patch passed

+1 :green_heart:
javac
18m 13s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
mvnsite
1m 31s

the patch passed

+1 :green_heart:
shadedclient
15m 16s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
17m  0s

hadoop-common in the patch passed.

+1 :green_heart:
asflicense
0m 55s

The patch does not generate ASF License warnings.

163m 55s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3019/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3019

Optional Tests
dupname asflicense compile cc mvnsite javac unit codespell golang

uname
Linux d873047096f0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 5dc10700a50adb51d32f4e1175d7fdd05cbca3a3

Default Java
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3019/1/testReport/

Max. process+thread count
1941 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3019/1/console

versions
git=2.25.1 maven=3.6.3

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 54s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
37m 37s

trunk passed

+1 :green_heart:
compile
29m 25s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
24m 29s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
1m  8s

trunk passed

+1 :green_heart:
mvnsite
1m 44s

trunk passed

+1 :green_heart:
javadoc
1m 12s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
1m 45s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
2m 52s

trunk passed

+1 :green_heart:
shadedclient
21m 31s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m  8s

the patch passed

+1 :green_heart:
compile
28m 43s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

-1 :x:
cc
28m 43s
/results-compile-cc-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt
root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 18 new + 309 unchanged - 18 fixed = 327 total (was 327)

+1 :green_heart:
golang
28m 43s

the patch passed

+1 :green_heart:
javac
28m 43s

the patch passed

+1 :green_heart:
compile
22m 57s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

-1 :x:
cc
22m 57s
/results-compile-cc-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt
root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 37 new + 290 unchanged - 37 fixed = 327 total (was 327)

+1 :green_heart:
golang
22m 57s

the patch passed

+1 :green_heart:
javac
22m 57s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m 21s

the patch passed

+1 :green_heart:
mvnsite
1m 40s

the patch passed

+1 :green_heart:
javadoc
1m  8s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
1m 47s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
3m 10s

the patch passed

+1 :green_heart:
shadedclient
21m 31s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
19m 20s

hadoop-common in the patch passed.

+1 :green_heart:
asflicense
0m 53s

The patch does not generate ASF License warnings.

225m 25s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3019/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3019

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc golang

uname
Linux 0638ff4f5c49 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 8188038a5eba90aa0b656bbfa0180cb40ca365e1

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3019/2/testReport/

Max. process+thread count
3136 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3019/2/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 51s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 3 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
35m  2s

trunk passed

+1 :green_heart:
compile
20m 44s

trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
compile
18m  1s

trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

+1 :green_heart:
checkstyle
1m  6s

trunk passed

+1 :green_heart:
mvnsite
1m 33s

trunk passed

+1 :green_heart:
javadoc
1m  4s

trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 37s

trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

+1 :green_heart:
spotbugs
2m 20s

trunk passed

+1 :green_heart:
shadedclient
15m 38s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 53s

the patch passed

+1 :green_heart:
compile
20m  6s

the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04

-1 :x:
cc
20m  6s
/results-compile-cc-root-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt
root-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 17 new + 310 unchanged - 17 fixed = 327 total (was 327)

+1 :green_heart:
golang
20m  6s

the patch passed

+1 :green_heart:
javac
20m  6s

the patch passed

+1 :green_heart:
compile
18m  4s

the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

-1 :x:
cc
18m  4s
/results-compile-cc-root-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt
root-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 46 new + 281 unchanged - 46 fixed = 327 total (was 327)

+1 :green_heart:
golang
18m  4s

the patch passed

+1 :green_heart:
javac
18m  4s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  5s

the patch passed

+1 :green_heart:
mvnsite
1m 31s

the patch passed

+1 :green_heart:
javadoc
1m  3s

the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 35s

the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

+1 :green_heart:
spotbugs
2m 31s

the patch passed

+1 :green_heart:
shadedclient
15m 55s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
17m  0s

hadoop-common in the patch passed.

+1 :green_heart:
asflicense
0m 54s

The patch does not generate ASF License warnings.

179m  5s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3019/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3019

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc golang

uname
Linux 03c5ca06e0ec 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / c1b7f59b91119e550cdf1a466ca53f75756d2095

Default Java
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3019/3/testReport/

Max. process+thread count
1653 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3019/3/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

The cc warnings are not related to the patch. I'm working on false positive cc warnings on YETUS-1107.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
35m 38s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 3 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
33m  7s

trunk passed

+1 :green_heart:
compile
19m 35s

trunk passed

+1 :green_heart:
checkstyle
1m  8s

trunk passed

+1 :green_heart:
mvnsite
1m 45s

trunk passed

+1 :green_heart:
javadoc
1m 52s

trunk passed

+1 :green_heart:
spotbugs
2m 33s

trunk passed

+1 :green_heart:
shadedclient
21m 40s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m  1s

the patch passed

+1 :green_heart:
compile
18m 54s

the patch passed

-1 :x:
cc
18m 54s
/results-compile-cc-root.txt
root generated 32 new + 173 unchanged - 32 fixed = 205 total (was 205)

+1 :green_heart:
golang
18m 54s

the patch passed

+1 :green_heart:
javac
18m 54s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  7s

the patch passed

+1 :green_heart:
mvnsite
1m 43s

the patch passed

+1 :green_heart:
javadoc
1m 49s

the patch passed

+1 :green_heart:
spotbugs
2m 38s

the patch passed

+1 :green_heart:
shadedclient
21m 51s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
17m 52s
/patch-unit-hadoop-common-project_hadoop-common.txt
hadoop-common in the patch passed.

+1 :green_heart:
asflicense
1m  1s

The patch does not generate ASF License warnings.

184m 16s

Reason
Tests

Failed junit tests
hadoop.service.launcher.TestServiceInterruptHandling

hadoop.ipc.TestIPC

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3019/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3019

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc golang

uname
Linux 0108ec7cb67a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 19552b41525949553d59fb84c93fb9b8c5b376db

Default Java
Red Hat, Inc.-1.8.0_312-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3019/4/testReport/

Max. process+thread count
1262 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3019/4/console

versions
git=2.9.5 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

The test failures are not related. HADOOP-18046 was filed for TestIPC. cc warnings will be addressed after bumping Yetus to 0.14.0 (containing https://github.com/apache/yetus/pull/227).
@jojochuang Are you ok to merge this?

I was able to test this locally and it worked as expected.

I'd suggest making it easy to control with a Maven system property like -Dopenssl.no.sm4.
OPENSSL_NO_SM4.patch

@snmvaughan Thanks for testing this.

I'd suggest making it easy to control with a Maven system property like -Dopenssl.no.sm4.

Please file another JIRA issue for you proposal. I'm not intending to disable SM4 even if the platform support it.

@jojochuang @iwasakims
Are you ok to merge this pr? It seems that changes to the common project's c code may trigger "Doesn't support SM4 CTR." For example: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6813/5/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt

@zhengchenyu I'm still willing to fix this and waiting for +1 from another committer.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m  0s

Docker mode activated.

-1 :x:
docker
22m 18s

Docker failed to build run-specific yetus/hadoop:tp-30537}.

Subsystem
Report/Notes

GITHUB PR
https://github.com/apache/hadoop/pull/3019

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3019/5/console

versions
git=2.34.1

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Related unit tests passed on my local environment. Manual testing looks fine as before.

for my reminder: In order to test SM4 codec, we need to put the jar of Bouncy Castle Provider on $JAVA_HOME/jre/lib/ext and add a line to $JAVA_HOME/jre/lib/security/java.security as described in the comment of HADOOP-15098.
security.provider.10=org.bouncycastle.jce.provider.BouncyCastleProvider

I merged this to trunk and branch-3.4. Thanks, @steveloughran.


Script to auto translate strings using Azure

Run all the language strings through Azure Translate API to generate translations.

Thank you for this Josh. As the translation service is neither public nor free it would not be very useful to include in the kit and also goes against the ethos of the project which is to use all open-source and free tools.