Deadlock in -[APMIdentity appInstanceID]

Description
We've been investigating some strange behaviors in our app that were are hard to reproduce, but manifested when foregrounding the app and as we fired off requests to refresh our content (and also send analytics events).
Eventually we figured out it seemed like some kind of contention / deadlock issue.
Eventually we were able to catch an instance and attached the debugger, where we were able to see the stack trace. The root cause seems to be a deadlock caused by contention on the dispatch_sync call inside -[APMIdentity appInstanceID] as per this screenshot:

In our case we were using Swift concurrency which has has a number of concurrent system threads matching the number of physical cores on the system (in my case this was 6 with an iPhone 14 Pro) servicing Swift Tasks. Each thread of these 6 was blocked on the same dispatch_sync call and no forward progress could be made.
It would stand to reason that this issue could also occur in other situations as well where other threads are bounded, not just when using Swift concurrency.
My understanding is that using dispatch_sync is tricky to get right, and may lead to deadlocks when accessed recursively, especially when the caller's context is unknown. According to man 3 dispatch_sync this is conceptually implemented using a dispatch_semaphore, which is described as unsafe to use with Swift concurrency in WWDC 2021 "Swift concurrency: Behind the scenes":

Unfortunately it's hard to introspect the GoogleAppMeasurement package to determine what could be the cause of the deadlock, but we also are using AppCheck, Analytics, Auth, Firestore, Functions, Dynamic Links, and RemoteConfig, among others. I could imagine this could be some kind of case where other mutual exclusion techniques elsewhere in Firebase could trigger this scenario.
Reproducing the issue
Reproducing this is tricky. I tried calling Analytics.appInstanceID in a tight loop concurrently (which was what was hanging) and wasn't able to reproduce. It seems likely that there's some other Firebase logic that has a dependency on GoogleAppMeasurement#APMIdentity that might be causing this issue. You can see that FIRAppAttestProvider.state is also blocked on something in that screenshot (unfortunately I don't have that stack available anymore), but they don't look related doing a basic inspection of FIRAppAppAttestProvider logic.

Sorry about the trouble and thanks for sharing the thorough analysis. This may be another reincarnation of an issue that has been fixed before. See #10080

Apropos this discussion, these Swift forum posts talking about deadlocks using DispatchQueue.sync + async barriers / reader-writer locks:

https://forums.swift.org/t/deadlock-when-using-dispatchqueue-from-swift-task/66058/25
https://forums.swift.org/t/deadlock-issue-when-accessing-thread-safe-objects-within-task-withtaskgroup/65322/4

Thanks for posting the details and links. We're looking into reproducing this and potential solutions.

Sorry about the trouble and thanks for sharing the thorough analysis. This may be another reincarnation of an issue that has been fixed before. See #10080

new crash is recently 16.6 iOS system

The root blocker is [ASIdentifierManager advertisingIdentifier]. Unfortunately, we haven't heard back from Apple. We added a timeout in dispatch_block_wait as a workaround. It seems that the timeout is not working properly, either.
APMIdentityWorkerQueue
0  libsystem_kernel.dylib         0x29e7c __ulock_wait + 8
1  libdispatch.dylib              0x51fc _dlock_wait + 56
2  libdispatch.dylib              0x4edc _dispatch_wait_on_address + 140
3  libdispatch.dylib              0x55a4 _dispatch_group_wait_slow + 60
4  libdispatch.dylib              0x8880 dispatch_block_wait + 308
5  NovalReader                     0xad9770 -[APMDefaultIdentitySupport resettableDeviceID] + 869904
6  NovalReader                     0xaeda14 __32-[APMIdentity updateIdentifiers]_block_invoke + 952500

If you are not using App Tracking Transparency or AdSupport, you may try using FirebaseAnalyticsWithoutAdIdSupport instead of FirebaseAnalytics.


[bug] cannot load from github url

With the latest mops,
base = "https://github.com/dfinity/motoko-base#moc-0.10.0"
leads to error:
Error: Call was rejected:
  Request ID: 9bef5779047d849a27829b9a44ff87156154451d1f417f3945d9c3eb47c4a760
  Reject code: 5
  Reject text: Canister oknww-riaaa-aaaam-qaf6a-cai trapped explicitly: Package 'base@https://github.com/dfinity/motoko-base#moc-0.10.0' not found

Thanks for bug report! Fixed in cli 0.28.1


chore(docs): offer an example for changing the default search bar text

Summary by CodeRabbit

New Features

Introduced a "Custom JavaScript" section for enhanced documentation customization.
Added support for multiple custom JavaScript files with various loading strategies.
Provided an example for modifying the search bar's placeholder text using custom JavaScript.
Expanded guidance on replacing default UI components with custom ones.

Documentation

Updated instructions for integrating custom CSS and JavaScript in the Fern documentation platform.

Closing as this is non-deterministic. When there's a better example, I can revisit.


Does not work over https

Hi scarytom,
I downloaded ci-eye v0.2.0 and when trying to access a Jenkins build server over https I get a stack trace like the one pasted below. Does seem to handle the error gracefully, as in, ci-eye remains active, presumably this means other views would still work - nice.
Admittedly https is unlikely to be required behind the firewall, but in my particular circumstances, I'm trying to access a hosted Jenkins cloud service. Is this likely to be something you'd consider including given such a minority use case? Happy to look into patching unless you have insider knowledge that makes it prohibitively difficult?
SEVERE: https://mutability-detector.ci.cloudbees.com/api/json
org.apache.http.client.ClientProtocolException
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:822)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:941)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:919)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:910)
        at org.netmelody.cieye.server.observation.protocol.RestRequester.makeRequest(RestRequester.java:65)
        at org.netmelody.cieye.server.observation.protocol.JsonRestRequester.makeJsonRestCall(JsonRestRequester.java:39)
        at org.netmelody.cieye.spies.jenkins.JenkinsCommunicator.makeJenkinsRestCall(JenkinsCommunicator.java:67)
        at org.netmelody.cieye.spies.jenkins.JenkinsCommunicator.makeJenkinsRestCallWithSuffix(JenkinsCommunicator.java:62)
        at org.netmelody.cieye.spies.jenkins.JenkinsCommunicator.views(JenkinsCommunicator.java:58)
        at org.netmelody.cieye.spies.jenkins.JenkinsSpy.jobsFor(JenkinsSpy.java:89)
        at org.netmelody.cieye.spies.jenkins.JenkinsSpy.targetsConstituting(JenkinsSpy.java:44)
        at org.netmelody.cieye.server.observation.PollingSpy.update(PollingSpy.java:94)
        at org.netmelody.cieye.server.observation.PollingSpy.access$100(PollingSpy.java:34)
        at org.netmelody.cieye.server.observation.PollingSpy$StatusUpdater.run(PollingSpy.java:153)

    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

Caused by: org.apache.http.HttpException: Scheme 'https' not registered.
at org.apache.http.impl.conn.DefaultHttpRoutePlanner.determineRoute(DefaultHttpRoutePlanner.java:115)
at org.apache.http.impl.client.DefaultRequestDirector.determineRoute(DefaultRequestDirector.java:721)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:358)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
... 21 more

give me 10 and I'll have a fix for this


Feature/fu 210715

Follow up 21.7.15 code in community
Also include:
Remove temporal code that adds posted metadata at create invitation

Related:
https://github.com/sktston/aries-cloudagent-python/issues/73


500 When Open PR

Gitea version (or commit ref):1.9.4
Git version:2.20.1 windows .1
Operating system:windows
Database (use [x]):

[ ] MySQL

Log gist:
2019/10/30 14:08:51 routers/repo/pull.go:303:PrepareMergedViewPullInfo() [E] GetCompareInfo: exit status 128 - fatal: ambiguous argument 'a5e691b37ba5324d20da34a2b56f291a14b3829c...refs/pull/164/head': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git  [...] -- [...]'

Description
URL:http://xxx.xxx.xxx.xxx/xxxx/xxxx/pulls/164

I'm fairly certain that this has been fixed. Therefore I'm closing this. @FalconWu2017 please reopen if the problem persists.


browser.js error

Describe the bug
getting this:
/.cpr/browser.js @947:7
TypeError: L is undefined
To Reproduce
no idea, just tried visiting the base url on localhost
Expected behavior
i dont get this error
Screenshots
see below
Server details
if the issue is possibly on the server-side, then mention some of the following:

server OS / version: 1.9.6

Client details
if the issue is possibly on the client-side, then mention some of the following:

the device type and model: windows
browser version: Firefox 117

Additional context

hey, thanks for the report and for giving copyparty a go 👍
think i know how to fix it, but want to make sure I've got it right --
while the error is visible (or while visiting any site on localhost with that browser), could you open the JavaScript console (hotkey Ctrl-Shift-K), run the following two commands and copy the output?

localStorage.lang
localStorage.lang===undefined

also, do you happen to be running any other web stuff on this machine / do you recall ever accessing any other service on localhost?

Command 1 outputted undefined
Command 2 outputted true
Also yeah im running a media server w/ sonarr/radarr on localhost

huh alright, that's not what i expected so good thing we checked -- could you open up the console in the tab that's showing the error, then refresh the tab, and run the following?
var r=['']; for (var a of 'acct dtheme lang u2sort'.split(' ')) r.push(window[a]); r.push(Object.keys(Ls)); r=r.join('], [')+']'; console.log(r);

there might be some relevant information being printed as the page is refreshed as well, so please include everything that appears starting from the refresh.
would also be great if you could hit Ctrl-U to show the html source of the page and copy/paste all of that into a textfile to attach here, but that would include the file listing from copyparty as well, so skip that if you've tried to view a folder that contains any filenames you wanna keep secret :>

I attached the source and only removed the name of my pc.
copyparty.txt
The complete console output is below.
Uncaught TypeError: L is undefined
    <anonymous> http://localhost:6969/.cpr/browser.js?_=F3Dq:947
browser.js:947:7
Uncaught TypeError: treectl is undefined
    <anonymous> http://localhost:6969/.cpr/up2k.js?_=F3Dq:40
up2k.js:40:1
InstallTrigger is deprecated and will be removed in the future. localhost:6969:80:238
sha-ok util.js:124:22
TypeError: result is undefined content_script.js:131:4
Uncaught (in promise) TypeError: L is undefined
    up2k_init http://localhost:6969/.cpr/up2k.js?_=F3Dq:800
    <anonymous> http://localhost:6969/.cpr/up2k.js?_=F3Dq:26
    promise callback* http://localhost:6969/.cpr/up2k.js?_=F3Dq:25
up2k.js:800:24
GET
http://localhost:6969/favicon.ico
[HTTP/1.1 404 Not Found 0ms]

var r=['']; for (var a of 'acct dtheme lang u2sort'.split(' ')) r.push(window[a]); r.push(Object.keys(Ls)); r=r.join('], [')+']'; console.log(r);
], [astolfo], [bz b z], [en], [s], [eng,nor] util.js:124:22
undefined

Thanks! Did you by any chance run copyparty with --lang en or --lang=en ? In that case, that would be why, since it only understands eng  currently.


Correct typos in notes for SPM

Describe the bug
The introductory notes have a few typos.
To Reproduce
Open SPM.ipynb
See spelling and grammatical typos in the introductory paragraphs.
Expected behaviour
No typos.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.

this was fixed in #877


add IDEs to gitignore

Description
Add IDEs (vscode and pycharm) files to gitignore file

This is considered as bad practice and should be done in your local git configuration.

Thanks for the pointer!


Change responsive to work better on Square devices

WHAT
Change responsive to work better on Square devices
WHY
No point having a larger top margin on full width square devices
HOW
Checklist :clipboard:

[ ] Add explicit visibility modifier and explicit return types for public declarations
[ ] Run spotless check
[ ] Run tests
[ ] Update metalava's signature text files

@luizgrp Can you post review?


it doesn't work with new cerebral-view-snabbdom

Hi,
cerebral-view-snabbdom does not export Component.DOM anymore. I tried to change .babelrc (I am new to babel ) but it didn't work.
Should I rewrite all App components with hyperscript? Or is there any solution to write it with JSX?
Thanks a lot.

Sounds like a bug in cerebral-view-snabbdom, maybe log an issue there?

OK, it seems that jsx is no longer included. you can add support by doing:
npm install snabbdom-jsx
// at the top of your component js file(s)
import { html } from 'snabbdom-jsx'

then in your .babelrc
{
  "plugins": [
    [ "transform-react-jsx", { "pragma": "html" } ]
  ]
}

I have also updated the cerebral docs with the above instructions http://www.cerebraljs.com/documentation/cerebral-view-snabbdom


Providing secrets as files in shared volume

Introduction
It has been shown by CyberArk Labs and other security experts that files are more secure than environment variables. Moreover, they are common to use when it comes to keeping secrets.
Providing the secrets as files means also keeping the secrets only in "CyberArk" hands and not in other places (like K8s secrets) enabling a more direct path from the secret source to the application making it more secure.
Saving secrets as files sets the ground to rotation in the future (although it is out of scope in this feature).
In this feature we will provide the ability for apps to consume Conjur/DAP secrets as files hence improve the delivery of secrets's security  while keeping the consumption of the secrets by the apps common and native.
The secrets will be saved as files in a memory based shared volume and will be provided before the application container is running as a one timer using init container.
Only init container is in scope.
The feature will be supported as OSS (like other phases) as community level.
Out of scope - There is no need to support multiple applications while delivery as files
Current mechanism
Current K8s secret provider saves the secrets in K8s secrets which is considered not as secure as CyberArk/DAP and is available to more than one pod. Once the secrets are in K8s secrets the RBAC is spread out to actually the operation team and not the security team.
Use cases
AS an application developer in K8s later on Liz the operator
I would like to control the secrets the applications is expecting & their location
So That the application can run
AS an application developer in K8s
I would like to place my secrets in files in memory based shared volume
So That no code change is needed for integration with Conjur\DAP
AS an application  in K8s
I would like to fetch my secrets from files in memory based shared volume
So That I can use them  in a common and a secure way (with no code change of the application before using secret provider)
The secrets are expected to be provided in advance before the app is initialized.
Process logic

Conjur admin, Martin, is creating Conjur policies with

Secrets needed to be consumed by the application.
An identity for the application's pod in which the secret provider relays as well- as the application pod is being identified using our K8s authentication the host name should follow the authentication identities guidelines and the application Identity guidelines.
Permissions to the application on the secrets
For example:

- !policy
  id: conjur/authn-k8s/my_cluster_auth
  body:
  - !webservice

  - !layer apps

  - !permit
    role: !layer apps
    privilege: [ read, authenticate ]
    resource: !webservice

- !policy
  id: allowed_apps
  annotations:
    description: Apps and services in cluster.
  body:

  - !layer k8sApps
  - &apps
    - !host
      id: namespace-based-app
      annotations:
        authn-k8s/namespace: my_namespace
        authn-k8s/authentication-container-name: cyberark-secrets-provider-for-k8s

  - !grant
    role: !layer k8sApps
    members: *apps

  - &variables
    - !variable password
    - !variable username

  - !permit
    role: !layer k8sApps
    privileges: [ read, execute ]
    resources: *variables

Liz/Martin set a config map
The config map will have groups of settings each group holds a map similar to conjur_map we have today.
2 alternatives are presented here one with folder path as group name and the other with a must have key called "folder-path". Other :  pairs will have the following structure : where the secret value of conjur variable will be saved in a file according to the folder-path.

In our flow:
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: my_config_map
  data:
    <folder-path>: |-
    username: db_credentials/username
    password: db_credentials/password

    <other-folder-path>: |-
    username1: other_credentials/username
    password1: other_credentials/password

  For example
  /secrets: |-
    username: db_credentials/username
    password: db_credentials/password

  Alternatively

  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: my_config_map
  data:
    group-name: |-
    username: db_credentials/username
    password: db_credentials/password
    folder-path: "/secrets"

    other-group-name: |-
    username1: other_credentials/username
    password1: other_credentials/password
    folder-path: "/mysecrets"

We need to understand if there is an issue here with special characters in the folder path that will require option 2 or that we like option 2 anyhow
Moti Cohen suggested to go with mountPath instead of folder-path if the path can be not relative than this is a great name change.

The operator, Liz, is creating a manifest in Openshift or K8s in which she defines a new image that can be an init container

For example:
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  labels:
    app: test-app
  name: test-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-app
  template:
    metadata:
      labels:
        app: test-app
    spec:
      serviceAccountName: < APP_SERVICE_ACCOUNT_NAME >
      containers:
      - image: < TEST_APP_DOCKER_IMAGE >
        imagePullPolicy: Always
        name: test-app
        ports:
        - containerPort: 8080
        volumeMounts:
          - mountPath: /cyberark <to be decided by dev - should match the path in secrets_folder_path>
            name: cyberark-secrets
            readOnly: true
    initContainers:
      - image: cyberark/cyberark-secret-provider-for-k8s
        imagePullPolicy: Always
        name: cyberark-secret-provider-for-k8s-file-flavor
        env:
          - name: CONTAINER_MODE
            value: init
          - name: MY_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: CONTAINER_MODE
              value: init

            - name: MY_POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name

            - name: MY_POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace

            - name: MY_POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP

            - name: CONJUR_APPLIANCE_URL
              value: < CONJUR_APPLIANCE_URL >

            - name: CONJUR_AUTHN_URL
              value: < CONJUR_AUTHN_URL >

            - name: CONJUR_ACCOUNT
              value: < CONJUR_ACCOUNT >

            - name: CONJUR_AUTHN_LOGIN
              value: host/conjur/authn-k8s/<authenticator-id>/apps/namespace-based-app

            - name: CONJUR_SSL_CERTIFICATE
              valueFrom:
                configMapKeyRef:
                  name: < CONFIG_MAP_NAME >
                  key: ssl-certificate

		  - name: SECRET_DESTINATION
            value: "files"

                -----NO KEY ------
        volumeMounts:
          - mountPath: /cyberark/secrets
            name: cyberark-secrets
         - mountPath: /cyberark/secrets-map
            name: secrets-map-configmap
      imagePullSecrets:
        - name: dockerpullsecret
      volumes:
        - name: cyberark-secrets
          emptyDir:
            medium: Memory
		configMap:
          name: secrets-map-configmap
          items:
            - key: my_config_map
              path: secrets-map

This new container name ,called in our example "secret provider for K8s - files flavour", will have the same env variables as the image in phase 1 had with few differences:

The value of SECRET_DESTINATION would be "files".
the memory volume should be of type memory
Subject to design - a new file called secrets-map will be given with a reference to the config map that was defined in previous step and stored in a mounted folder.

**Note the name of the image is a design decision

Liz, the operator  deploys the pod.

The pod initialize the during the initialization

5.1 The new container "secret provider for K8s - files flavor" is authenticating to Conjur using the pod's characteristics and host identity then fetches the needed secrets from Conjur and place them in the right path as files.
In our example
mountPath in the app container is /cyberark folder path of group 1 is /secrets
variable db_credentials/username is placed in volume mount cyberark-secrets of the app pod in a file path /cyberark/secrets/username
variable db_credentials/password is placed in volume mount cyberark-secrets of the app pod in a file path /cyberark/secrets/password
folder path of group 2 is /mysecrets hence
variable other_credentials/username is placed in volume mount cyberark-secrets of the app pod in a file path /cyberark/mysecrets/username1
variable other_credentials/password is placed in volume mount cyberark-secrets of the app pod in a file path /cyberark/mysecrets/password1
If the container is an init container it conclude his role here

5.2 The app container life begin - the application fetches the secrets from the files

/cyberark/secrets/username
/cyberark/secrets/password
Out of scope -If the new container is a sidecar it perform the above steps every 5 minutes, meaning it fetches the secrets from conjur and if changed place them on to the shared volume. That way we support rotation of secr
Requirements
The functionality that is given as today should not be erased or damaged as it provide ability to provide secrets as K8s secrets.
We will support only K8s applications that are running as containers in a pod.
DAP & OSS
The secret provider should work with DAP (Enterprise edition) and Conjur (OSS)
####Security or permissions
No special security or permissions configuration should be done in order to provide the application's container/pod its secrets by k8s secret provider.
The secret provider should serve only applications under at least the same namespace.
Should the volume be readOnly?
Configuration and Ease of usage
We should try and understand if it is possible to have the same secret provider image running in these different modes.
The value of the configuration called SECRET_DESTINATION should be "files" instead of "k8s_secrets".
Deployment
Deployment mechanism should be native and think of full container life cycle like upgrade.
we should think wat guidelines we need to provide to customers who want to move from init container with K8s secrets to init container with files
Performance
The application should start running with minimum delay in time.
Lets create an avg time test where an app consume 25 vars (5 PAS accounts)
Audits
We should investigate the need for audit when writing to files.
Failure Scenarios

Failure Scenario
Error message
Expected behavior/UX

Secret cant be fetchedConjur path is not correctIt can happen because no permissions where provided to the host ofConjur-authn-client or because the path is invalid.
key  can not be fetched from Conjur according to path . Error:
A log is written with the right error code and message. The log can be seen in regular cloud tools like openshift pod log, elastic search...Fail Pod

Failed to authenticate using Access token provided
Failed to authenticate using Access token provided here <>. Error: <>
proper log and fail POD

Failed to save secret to file file path can be incorrect permissions to write

proper log and fail POD

Monitoring
Monitoring Metrics Analysis
The container should be monitored using Openshift and K8s tools - probably as part of the POD.
We should take care that it is not stucked - we should do it by liveliness probe https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
Logging
They should have an error code and log level
The logs should appear in Cloud tools like elasticsearch**
Supportability
We should provide guidelines and troubleshooting info if the deployment of this image fails or it fails during runtime.
Conditions of satisfaction

The new process works as expected: including deployment of the init container, fetching of the Conjur secrets and writing them to files, initialization of the app container with all the right values
The old process works as before.
Error flows were discussed, defined and implemented
The Mapping mechanism of Conjur secrets works and all end cases are handled
The authentication mechanism is k8s authn
Secrets are being fetched according to the permissions of the container's host and are save as files in the shared volume
Logs are being written as these actions take place.
LATER STAGE - Out of scope for this phase - In case of a sidecar If a secret is rotated or changed the app would get a hold of the latest and greatest in a 5 min cycle.
Error handling
Retry once before failing
Test on a common env of a scaled pod X3 and few namespaces were on follower namespace serve all of them.

DoD

Test plan written and reviewed by PO & QAA
Automatic integration tests written according to a test plan and passed successfully
UT written for all classes\functions\major logic flows and passed successfully
Security review was done
Security action items were taken
Performance tests were done
Supportability tasks were written
Enhance logs and supportability
Logs were reviewed by TW and PO
Fill in the configurations in this feature doc
Configurations were reviewed by PO .
Documentation HO to TW and review docs_
Release as community level feature

References

Functional spec (private)
PRD (private)

Conjur server changes to consider:

cyberark/conjur#843 - batch retrieve secrets matching variable prefix
cyberark/conjur#2071 - batch retrieval endpoint doesn't fail when only  a subset of requested secrets can be retrieved

Related: cyberark/conjur#1132 - rethink batch secret retrieval audit

cyberark/conjur#1519 - Conjur has an /info endpoint
cyberark/conjur#1520 - Conjur has a /metrics endpoint

Closing in favor of #333.


Failed to execute linalg.fill + linalg.conv on vulkan-spirv

After we have support for tiling and fusing Linalg ops, I tried it on linalg.fill + linalg.conv. It does succeed in #1356 . However, It fails when execute in e2e test.
I think the reason might be the serialization in SPIR-V or the lowering from GPU.
I would suspect SPIR-V more because it can't pass mlir-translate -test-spirv-roundtrip z.mlir. It crashes in Deserializer::structurizeControlFlow().
We can serialize it to a binary, and run spirv-dis on it. It fails in spirv-val:
error: line 177: OpPhi's incoming basic block <id> 73[%73] is not a predecessor of <id> 123[%123].
  %125 = OpPhi %uint %194 %126 %47 %73

Log: fill_conv.txt
To repro:
Download the log, and take the last spv code.
compile mlir-translate, spirv-dis and spirv-val.
$ mlir-translate -split-input-file -serialize-spirv z.mlir > z.binary
$ mlir-translate -split-input-file -test-spirv-roundtrip z.mlir
$ spirv-dis z.binary
$ spirv-val z.binary

Oops. Let me take a look.

This should be fixed by https://reviews.llvm.org/D77543.

Fix landed in upstream as https://github.com/llvm/llvm-project/commit/47b234944dc063daf61d3e72e6b603ba9615baf0. I'll close this. Let me know if you still have problems after the fix.

Thanks for fixing it, I'll verify it.


It's a big error, i'm crazy

I follow your guideline from the readme.md but it's doesn' t work...it's has spend me a lot of time,
i am crazy now , i have no idea so that i beg for your help

Can you paste the error here?


⚠ Bofrost.de has degraded performance

In 31256cc, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1537 ms

Resolved: Bofrost.de performance has improved in 23ba7b9 after 6 minutes.


FPGA: Fix namespaces errors for math function calls

Existing Sample Changes
Description
Versions of host_ptr and device_ptr that are in the sycl:: namespace are deprecated and should be replaced by sycl::ext::intel namespace.
Explicitly specifying sycl:: is needed for some code samples.
Test Run: https://spetc.intel.com/testsummary?trview=0&testRunIds=10273947
Fixes Issue#
External Dependencies
List any external dependencies created as a result of this change.
Type of change
Please delete options that are not relevant. Add a 'X' to the one that is applicable.

[x] Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[ ] Command Line
[ ] oneapi-cli
[ ] Visual Studio
[ ] Eclipse IDE
[ ] VSCode
[ ] When compiling the compliler flag "-Wall -Wformat-security -Werror=format-security" was used

The change should target the development branch rather than the master branch


Need help with performance on writing custom caching module for our CDN

Hey Jim.  I've been working on a custom caching module for our CDN for the last several days.  I pretty much have it done but have ran into something I can't explain. When I use the DiskCach plugin that comes with ImageProcessor I'm getting response times in the 10 to 20ms range.  When I use my plugin I get response times in the 100ms range (or more).  I've done everything I know how to do to debug the code, going so far as to have the IsNewOrUpdatedAsync() method always return false (doing no checks) and the RewritePath method doing an immediate redirect also doing no checks  Same results.
There must be some magic sauce I'm missing.
I'm including my code below. Please note that my methods do not implement await since the SFTP plugin I'm using does not implement it.  Any ideas there would be helpful.
I've also removed my code for TrimCacheAsync as it was doubling the amount of time it took to load an image.
//-----------------------------------------------------------------------
// <copyright file="CacheFlyImageCache.cs" company="Schweitzer Engineering Laboratories Inc.">
//	Copyright (c) Schweitzer Engineering Laboratories Inc. All rights reserved.
// </copyright>
// <author>Dave Sargent</author>
// <summary>
//   Provides an <see cref="IImageCache" /> implementation that uses CacheFly storage.
// </summary>
//-----------------------------------------------------------------------

namespace Sel.Web.Plugins
{
	using System;
	using System.Collections.Generic;
	using System.Configuration;
	using System.IO;
	using System.Linq;
	using System.Net;
	using System.Net.Http;
	using System.Threading.Tasks;
	using System.Web;
	using ImageProcessor.Web.Caching;
	using ImageProcessor.Web.HttpModules;
	using Renci.SshNet;
	using Utils.Caching;

	/// <summary>
	/// Provides an <see cref="IImageCache"/> implementation that is file system based.
	/// </summary>
	public class CacheFlyImageCache : ImageCacheBase
	{
		/// <summary>
		/// The memory stream to store the private key in so it does not lock.
		/// </summary>
		private static Stream privateKeyFileStream;

		/// <summary>
		/// The CDN Base directory: https://cdn.selinc.com/
		/// </summary>
		private static string cdnBase;

		/// <summary>
		/// The ftp base which is added onto the cdn URL and used as the base for the directory in all FTP operations.
		/// Example: /imageprocessorcache/
		/// </summary>
		private static string ftpBase;

		/// <summary>
		/// The FTP Server name.
		/// </summary>
		private static string ftpServer;

		/// <summary>
		/// The FTP username
		/// </summary>
		private static string ftpUsername;

		/// <summary>
		/// The FTP Password.
		/// </summary>
		private static string ftpPassword;

		/// <summary>
		/// Private key path.
		/// </summary>
		private static string privateKeyFilePath;

		/// <summary>
		/// The pass phrase for the private key.
		/// </summary>
		private static string privateKeyPassPhrase;

		/// <summary>
		/// The encoded file name: 57e082a8c6e715f64e6a0de38acac9916a011303.jpg
		/// </summary>
		private string cachedFileName;

		/// <summary>
		/// The path derived from the encoded file name: 5/7/e/0/8/2
		/// Example: 5/7/e/0/8/2
		/// </summary>
		private string virtualPathFromKey;

		/// <summary>
		/// The relative file path to the image on the FTP server.
		/// Example: /static/imageprocessorcache/5/7/e/0/8/2/57e082a8c6e715f64e6a0de38acac9916a011303.jpg
		/// </summary>
		private string ftpRelPath;

		/// <summary>
		/// The original URL of the request.
		/// </summary>
		private string originalUrl;

		/// <summary>
		/// The cache key which is based on the origianl URL but encoded.
		/// </summary>
		private string cacheKey;

		/// <summary>
		/// An instance of SelMemCached for caching the image object data for speed increase.
		/// </summary>
		private SelMemcached selMemcached = new SelMemcached();

		/// <summary>
		/// Initializes a new instance of the <see cref="CacheFlyImageCache"/> class.
		/// </summary>
		/// <param name="requestPath">
		/// The request path for the image.
		/// </param>
		/// <param name="fullPath">
		/// The full path for the image.
		/// </param>
		/// <param name="querystring">
		/// The querystring containing instructions.
		/// </param>
		public CacheFlyImageCache(string requestPath, string fullPath, string querystring)
			: base(requestPath, fullPath, querystring)
		{
			// If not already loaded, load the settings from the settings file.
			if (string.IsNullOrWhiteSpace(cdnBase))
			{
				cdnBase = ConfigurationManager.AppSettings["CdnBase"];
				ftpBase = ConfigurationManager.AppSettings["FtpBase"];
				ftpServer = ConfigurationManager.AppSettings["FtpServer"];
				ftpUsername = ConfigurationManager.AppSettings["FtpUsername"];
				ftpPassword = ConfigurationManager.AppSettings["FtpPassword"];
				privateKeyPassPhrase = ConfigurationManager.AppSettings["PrivateKeyPassPhrase"];
				privateKeyFilePath = ConfigurationManager.AppSettings["PrivateKeyFilePath"];
			}

			HttpContext context = HttpContext.Current;
			this.originalUrl = context.Request.RawUrl;
			this.cacheKey = SelCacheKeyGenerator.NormalizeCacheKey(this.originalUrl);
		}

		/// <summary>
		/// Gets a value indicating whether the image is new or updated in an asynchronous manner.
		/// </summary>
		/// <returns>
		/// The <see cref="Task"/>.
		/// </returns>
		public override async Task<bool> IsNewOrUpdatedAsync()
		{
			// Gets the cached file name.
			// Example: 57e082a8c6e715f64e6a0de38acac9916a011303.jpg
			this.cachedFileName = await this.CreateCachedFileNameAsync();

			// Collision rate of about 1 in 10000 for the folder structure.
			// That gives us massive scope to store millions of files.
			// Gets the path from the key.
			// Example: 5\7\e\0\8\2
			string pathFromKey = string.Join("\\", this.cachedFileName.ToCharArray().Take(6));

			// Example: 5/7/e/0/8/2
			this.virtualPathFromKey = pathFromKey.Replace(@"\", "/");

			// Example: /static/imageprocessorcache/5/7/e/0/8/2/57e082a8c6e715f64e6a0de38acac9916a011303.jpg
			this.ftpRelPath = ftpBase + this.virtualPathFromKey + "/" + this.cachedFileName;

			// Example: https://cdn.selinc.com/imageprocessorcache/5/7/e/0/8/2/57e082a8c6e715f64e6a0de38acac9916a011303.jpg
			this.CachedPath = cdnBase + this.virtualPathFromKey + "/" + this.cachedFileName;

			bool isUpdated = false;

			// We cache a copy of the image data including a key, the path and the creation time in memcached to speed up loading.
			// status.  If the image has been cached and added to that list then this will return that data from memcached and we can skip FTP load.
			Sel.Models.Caching.CachedImage cachedImage = this.selMemcached.Get<Sel.Models.Caching.CachedImage>(this.cacheKey);

			// If there is no cachedImage in Memcached then we must load it from the FTP server.
			if (cachedImage == null)
			{
				// Create the SFTP connection and connect to the server.
				SftpClient sftp = this.ConnectToSftpServer();

				// If this file exists on the FTP server then it is already cached, created a Memcache copy of the file attributes we care about.
				if (sftp.Exists(this.ftpRelPath))
				{
					// Examples:
					// Key: 57e082a8c6e715f64e6a0de38acac9916a011303
					// Path: https://cdn.selinc.com/imageprocessorcache/5/7/e/0/8/2/57e082a8c6e715f64e6a0de38acac9916a011303.jpg
					cachedImage = new Sel.Models.Caching.CachedImage
					{
						Key = Path.GetFileNameWithoutExtension(this.cachedFileName), // Example: 57e082a8c6e715f64e6a0de38acac9916a011303
						Path = this.CachedPath, // Example:  https://cdn.selinc.com/imageprocessorcache/5/7/e/0/8/2/57e082a8c6e715f64e6a0de38acac9916a011303.jpg
						CreationTimeUtc = sftp.GetLastWriteTimeUtc(this.ftpRelPath)
					};

					// Add the file data to the in memory cache index for later comparison.
					// We use the original URL to better find them in Memcached.  They will all start with /uploadedimages
					this.selMemcached.Add(this.cacheKey, cachedImage, DateTime.Now.AddYears(1), null, null);
				}

				this.DisconnectSftpServer(ref sftp);
			}

			if (cachedImage == null)
			{
				// Nothing in the cache so we should return true.
				isUpdated = true;
			}
			else
			{
				// Check to see if the cached image is set to expire.
				if (this.IsExpired(cachedImage.CreationTimeUtc))
				{
					this.selMemcached.Remove(this.cacheKey);
					isUpdated = true;
				}
			}

			return isUpdated;
		}

		/// <summary>
		/// Adds the image to the cache in an asynchronous manner.
		/// </summary>
		/// <param name="stream">
		/// The stream containing the image data.
		/// </param>
		/// <param name="contentType">
		/// The content type of the image.
		/// </param>
		/// <returns>
		/// The <see cref="Task"/> representing an asynchronous operation.
		/// </returns>
		public override async Task AddImageToCacheAsync(Stream stream, string contentType)
		{
			var sw = System.Diagnostics.Stopwatch.StartNew();

			// Create the SFTP connection and connect to the server.
			SftpClient sftp = this.ConnectToSftpServer();

			// Attempt to created the destination directory
			this.ChangeDirectoryTo(ref sftp, ftpBase + this.virtualPathFromKey);

			// Upload the file to the server.
			sftp.UploadFile(stream, this.ftpRelPath, true);

			// Add the file data to Memcached.
			Sel.Models.Caching.CachedImage cachedImage = new Sel.Models.Caching.CachedImage
			{
				Key = Path.GetFileNameWithoutExtension(this.cachedFileName), // Example: 57e082a8c6e715f64e6a0de38acac9916a011303
				Path = this.CachedPath, // Example:  https://cdn.selinc.com/imageprocessorcache/5/7/e/0/8/2/57e082a8c6e715f64e6a0de38acac9916a011303.jpg
				CreationTimeUtc = sftp.GetLastWriteTimeUtc(this.ftpRelPath)
			};

			this.DisconnectSftpServer(ref sftp);
			this.selMemcached.Add(this.cacheKey, cachedImage, DateTime.Now.AddYears(1), null, sw.ElapsedMilliseconds);
		}

		/// <summary>
		/// Trims the cache of any expired items in an asynchronous manner.
		/// </summary>
		/// <returns>
		/// The asynchronous <see cref="Task"/> representing an asynchronous operation.
		/// </returns>
		public override async Task TrimCacheAsync()
		{
			// Do nothing here.  Too slow and didn't work well.
			// Our cache folder is going to eventually get messy, we will just clean it up manually.
		}

		/// <summary>
		/// Rewrites the path to point to the cached image.
		/// </summary>
		/// <param name="context">
		/// The <see cref="HttpContext"/> encapsulating all information about the request.
		/// </param>
		public override void RewritePath(HttpContext context)
		{
			// Using an HttpClient we check to make sure this image exists on the target CDN by doing a get head.
			// This makes sure the file is not missing for whatever reason.
			HttpClient httpClient = new HttpClient();
			TryFiveTimes(() =>
			{
				HttpResponseMessage responseMessage = httpClient.SendAsync(new HttpRequestMessage(HttpMethod.Head, this.CachedPath)).Result;
				HttpStatusCode responseCode = responseMessage.StatusCode;
				ImageProcessingModule.AddCorsRequestHeaders(context);
				if (responseCode == HttpStatusCode.NotFound)
				{
					// More than likely the file is missing from the destination caching server (ftp site)
					// but is still in Memcached, so let's remove it.
					selMemcached.Remove(this.cacheKey);

					// Need to redirect to original URL causing a re-cache of the image.
					context.Response.Redirect(this.originalUrl, true);
				}
				else
				{
					// Redirect to cached image on the CDN, we are done.
					context.Response.Redirect(this.CachedPath, false);
				}
			});
		}

		/// <summary>
		/// Tries to execute a delegate action five times.
		/// </summary>
		/// <param name="delegateAction">The delegate to be executed</param>
		private static void TryFiveTimes(Action delegateAction)
		{
			for (int retry = 0; retry < 5; retry++)
			{
				try
				{
					delegateAction();
					return;
				}
				catch (Exception ex)
				{
					if (retry >= 5)
					{
						Sel.Utils.Logging.ExceptionLogger.Log(ex);
					}
				}
			}
		}

		/// <summary>
		/// Change the directory on the SFTP server tot he target directory, creating sub directories on necessary.
		/// </summary>
		/// <param name="sftp">The currently open SFTP connection.</param>
		/// <param name="targetDirectory">Full target directory path.</param>
		private void ChangeDirectoryTo(ref SftpClient sftp, string targetDirectory)
		{
			// If we are not connected, can't change the directory so just bail.
			if (!sftp.IsConnected)
			{
				return;
			}

			sftp.ChangeDirectory("/");

			string[] dirParts = targetDirectory.Split('/');
			foreach (string dir in dirParts)
			{
				if (string.IsNullOrWhiteSpace(dir))
				{
					continue;
				}

				// Sometimes this fails, if it does, give it another try, usually works then.
				// Can't use TryFiveTimes due to the ref sftp
				try
				{
					if (!sftp.Exists(dir))
					{
						sftp.CreateDirectory(dir);
					}
				}
				catch (Exception)
				{
					if (!sftp.Exists(dir))
					{
						sftp.CreateDirectory(dir);
					}
				}

				try
				{
					sftp.ChangeDirectory(dir);
				}
				catch (Exception)
				{
					sftp.ChangeDirectory(dir);
				}
			}
		}

		/// <summary>
		/// Connect to the SFTP server and return the SFTPClient.
		/// </summary>
		/// <returns>Connected SFTP Client.</returns>
		private SftpClient ConnectToSftpServer()
		{
			SftpClient sftp = null;
			Stream lclPrivateKeyFileStream;

			// Sometimes parts of this fail, trying again tends to fix it.
			TryFiveTimes(() =>
			{
				// If the privateKeyFileStream that holds the private key data is null or we can't read it, then load it.
				if (privateKeyFileStream == null || !privateKeyFileStream.CanRead || privateKeyFileStream.Length == 0)
				{
					privateKeyFileStream = new MemoryStream();
					using (Stream input = File.OpenRead(privateKeyFilePath))
					{
						input.CopyTo(privateKeyFileStream);
					}
				}

				// Copy to the private key file stream into a local version as the PrivateKeyAuthenticationMethod is going to close it for us.
				lclPrivateKeyFileStream = new MemoryStream();
				privateKeyFileStream.Position = 0;
				privateKeyFileStream.CopyTo(lclPrivateKeyFileStream);
				lclPrivateKeyFileStream.Position = 0;

				// Create the PrivateKeyFile object used to authenticate tot he SFTP server.
				PrivateKeyFile privateKeyFile = new PrivateKeyFile(lclPrivateKeyFileStream, privateKeyPassPhrase);
				List<AuthenticationMethod> authMethods = new List<AuthenticationMethod>();
				authMethods.Add(new PasswordAuthenticationMethod(ftpUsername, ftpPassword));
				authMethods.Add(new PrivateKeyAuthenticationMethod(ftpServer, new PrivateKeyFile[] { privateKeyFile }));

				// Create new SSH connection information to connect to SFTP server.
				ConnectionInfo sshConnectionInfo = new ConnectionInfo(ftpServer, 22, ftpUsername, authMethods.ToArray());
				sshConnectionInfo.Timeout = new TimeSpan(0, 2, 0);

				sftp = new SftpClient(sshConnectionInfo);
				sftp.Connect();
			});

			return sftp;
		}

		/// <summary>
		/// Disconnect and dispose of the SFTP client.
		/// </summary>
		/// <param name="sftp">The SftpClient to disconnect..</param>
		private void DisconnectSftpServer(ref SftpClient sftp)
		{
			sftp.Disconnect();
			sftp.Dispose();
		}
	}
}

Update:  Redid the timing tests and something has changed.  The DiskCache plugin is coming back with times around 38 to 49 ms for the first few files then goes up to 100 thereafter.  My code seems pretty stable at 97 to 99 ms
So . . . Maybe it's OK?
Images attached.

BTW, I took a look at what CreateCachedFileNameAsync was doing and realized it was doing more than I thought. so I've added some code to match the key in the cached copy to the key from that routine to see if our image is different.
if (cachedImage == null)
{
	// Nothing in the cache so we should return true.
	isUpdated = true;
}
else
{
	// Check to see if the cached image is set to expire.
	if (this.IsExpired(cachedImage.CreationTimeUtc) || cachedImage.Key != Path.GetFileNameWithoutExtension(this.cachedFileName))
	{
		this.selMemcached.Remove(this.cacheKey);
		isUpdated = true;
	}
}

Sorry, one more update.  The above page test was just an HTML page with a bunch of images in it to show load.  When I test the actual site things change a ton (which is what I was originally seeing.
Below timings are for one of our product pages
https://selinc.com/products/849/
When using the disk cache, the main image comes back in 30ms, with my CDN cache (already cached, not doing FTP or any of that) it's a second!!!
Here's the part that is hurting my brain.  If I load just that image by going straight to its URL it only takes 18ms.  So server load time is greatly affecting things.  It's just affecting my code more than yours and I don't know why.  Maybe the lack of await code?  But I don't know how to fix that . . .

Well, I at least have something figured out.  It's the Response.Redirect.
I don't know WHY it's the Response.Redirect (same basic idea used in your S3 and Azure modules.
I can't think of a better way of doing this either.
Any thoughts?

I think I found the "issue"  Looks like it may just be some Chrome weirdness.  In the Firefox network console the times are extremely fast.  The majority of the time is spent in the "Content Download" section and I just don't think it's really waiting.  In Firefox I'm getting timings of 47 ms or so, which is what it should be.
Jim, I'm going to close this for now. If you have further input, please let me know.


Building bpfjit with sljit-gpu

Hi Alex,
Kudos for such great idea and implementation of sljit-gpu!
I want to use it and meantime completely stuck - would appreciate your help :)
I'm trying to build bpfjit on top sljit-gpu.
I have everything compiled and all tests passed, but bpfjit doesn't work yet.
There are couple of things I don't fully understand:

The original sljit_generate_code returns function pointer ready to be called. In sljit-gpu the target is to produce PTX if I understand correctly. What does sljit_generate_code return then? How do all tests run when the target is not executable?
There is generatePTX function in LLVMCWrappers that expects NNVM IR as an input. Does it mean that sljit_generate_code supposed to generate IR?
Part of the tests are commented. Does sljit-gpu supports full functionality of sljit? Can it be that bpfjit uses some unsupported functionality?

Best regards,
Leonid.

Hello,

Currently, sljit_generate_code generates native CPU code so that the LLVM backend can be easily validated. The idea is to validate the backend on the CPU first, then wire NVPTX up, in which case sljit_generate_code would generate Cubin. There are some relatively minor things left to do for running on GPUs, but NVVM IR is a big enough subset of LLVM IR that I don't expect this to be hard (and I have experience with this approach from my job).
See above.
It's possible to support everything, but enabling PCRE is my actual goal and not everything is needed for it. Priorities shifted and I had to put on hold this project; there's every chance I'll resume working on it later this year.

Extrapolating from my experience with PCRE, floating point support is probably not needed for bpfjit either. Certain things are hard to implement efficiently in a heterogeneous address space -- like overriding constants, jump targets and so on, but it looks feasible to change PCRE at least to avoid them.  BPFJIT might not need this functionality at all. Of course, you'll probably need to send a big batch of packets to keep 10k+ threads busy, therefore some adjustments will still be needed at the generated code call sites.

Thanks for the prompt answer!
Actually I have figured it out yesterday 😊
The last question is how to reformat the LLVM Module IR representation to the string so I can run generatePTX function?
Regards,
Leonid.
-----Original Message-----
From: "Alex Suhan"<EMAIL_ADDRESS>Sent: 03/06/2016 10:49
To: "asuhan/sljit-gpu"<EMAIL_ADDRESS>Cc: "Leonid Gorelik"<EMAIL_ADDRESS>"Author"<EMAIL_ADDRESS>Subject: Re: [asuhan/sljit-gpu] Building bpfjit with sljit-gpu (#1)
Hello,
Currently, sljit_generate_code generates native CPU code so that the LLVM backend can be easily validated. The idea is to validate the backend on the CPU first, then wire NVPTX up, in which case sljit_generate_code would generate Cubin. There are some relatively minor things left to do for running on GPUs, but NVVM IR is a big enough subset of LLVM IR that I don't expect this to be hard (and I have experience with this approach from my job).
See above.
It's possible to support everything, but enabling PCRE is my actual goal and not everything is needed for it. Priorities shifted and I had to put on hold this project; there's every chance I'll resume working on it later this year.
Extrapolating from my experience with PCRE, floating point support is probably not needed for bpfjit either. Certain things are hard to implement efficiently in a heterogeneous address space -- like overriding constants, jump targets and so on, but it looks feasible to change PCRE at least to avoid them. BPFJIT might not need this functionality at all. Of course, you'll probably need to send a big batch of packets to keep 10k+ threads busy, therefore some adjustments will still be needed at the generated code call sites.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

@leongor You can dump a module with module->dump() (C++ API, I don't know the C counterpart by heart). But Cubin can be generated directly from the in-memory representation, by doing something along the lines of:

Initialize all backends / asm printers:

llvm::InitializeAllTargets();
llvm::InitializeAllTargetMCs();
llvm::InitializeAllAsmPrinters();

Create a target machine:

std::string err;
auto target = llvm::TargetRegistry::lookupTarget("nvptx64", err);
target->createTargetMachine("nvptx64-nvidia-cuda",
                                                            "sm_30",
                                                            "",
                                                            llvm::TargetOptions(),
                                                            llvm::Reloc::Default,
                                                            llvm::CodeModel::Default,
                                                            llvm::CodeGenOpt::Aggressive)

Set the layout and target triple:

module->setDataLayout(
     "e-p:64:64:64-i1:8:8-i8:8:8-"
     "i16:16:16-i32:32:32-i64:64:64-"
     "f32:32:32-f64:64:64-v16:16:16-"
     "v32:32:32-v64:64:64-v128:128:128-n16:32:64");
module->setTargetTriple("nvptx64-nvidia-cuda");

Get the PTX:

std::stringstream ss;
llvm::raw_os_ostream raw_os(ss);
llvm::formatted_raw_ostream formatted_os(raw_os);
llvm::legacy::PassManager ptxgen_pm;
module->setDataLayout(nvptx_target_machine_->createDataLayout());
nvptx_target_machine_->addPassesToEmitFile(ptxgen_pm, formatted_os, llvm::TargetMachine::CGFT_AssemblyFile);
ptxgen_pm.run(*module);
formatted_os.flush();
/* ss.str() returns the PTX */

Use the CUDA API to get the Cubin (check cuLinkCreate, cuLinkAddData and cuLinkComplete).

The function to be called as a kernel will need some metadata as well, which can be added through the LLVM API. You can find information about this (and more) here: http://llvm.org/docs/NVPTXUsage.html. I might create a helper to do all this (with some example) over the next few days.

Thanks!
The helper will be very helpful.
This part can be pretty tricky for someone new to LLVM 😃
Regards,
Leonid.
-----Original Message-----
From: "Alex Suhan"<EMAIL_ADDRESS>Sent: 03/06/2016 11:40
To: "asuhan/sljit-gpu"<EMAIL_ADDRESS>Cc: "Leonid Gorelik"<EMAIL_ADDRESS>"Mention"<EMAIL_ADDRESS>Subject: Re: [asuhan/sljit-gpu] Building bpfjit with sljit-gpu (#1)
@leongor You can dump a module with module->dump() (C++ API, I don't know the C counterpart by heart). But Cubin can be generated directly from the in-memory representation, by doing something along the lines of:
Initialize all backends / asm printers:
llvm::InitializeAllTargets();
llvm::InitializeAllTargetMCs();
llvm::InitializeAllAsmPrinters();
Create a target machine:
std::string err;
auto target = llvm::TargetRegistry::lookupTarget("nvptx64", err);
target->createTargetMachine("nvptx64-nvidia-cuda",
"sm_30",
"",
llvm::TargetOptions(),
llvm::Reloc::Default,
llvm::CodeModel::Default,
llvm::CodeGenOpt::Aggressive)
Set the layout and target triple:
module->setDataLayout(
"e-p:64:64:64-i1:8:8-i8:8:8-"
"i16:16:16-i32:32:32-i64:64:64-"
"f32:32:32-f64:64:64-v16:16:16-"
"v32:32:32-v64:64:64-v128:128:128-n16:32:64");
module->setTargetTriple("nvptx64-nvidia-cuda");
Get the PTX:
std::stringstream ss;
llvm::raw_os_ostream raw_os(ss);
llvm::formatted_raw_ostream formatted_os(raw_os);
llvm::legacy::PassManager ptxgen_pm;
module->setDataLayout(nvptx_target_machine_->createDataLayout());
nvptx_target_machine_->addPassesToEmitFile(ptxgen_pm, formatted_os, llvm::TargetMachine::CGFT_AssemblyFile);
ptxgen_pm.run(module);
formatted_os.flush();
/ ss.str() returns the PTX */
Use the CUDA API to get the Cubin (check cuLinkCreate, cuLinkAddData and cuLinkComplete).
The function to be called as a kernel will need some metadata as well, which can be added through the LLVM API. You can find information about this (and more) here: http://llvm.org/docs/NVPTXUsage.html. I might create a helper to do all this (with some example) over the next few days.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Found it.
module->dump() counterpart in c is  LLVMDumpModule, but it dumps the IR to stderr.
Better to use module->print() ( LLVMPrintModuleToString in c) - creates it as a string (char*).

@leongor I'll resume working on this project today and figure out what's the most effective path to having bpfjit working on top of it. Thanks for your interest!

Thank you - appreciate it very much.

bpfjit uses a newer sljit API that the one I've implemented (0.9.2). The good news is it doesn't look too difficult to adjust and bpfjit might just work since it's only using a few sljit opcodes. I'm going to take a stab at it over the weekend.

It will be great to have it updated to the latest version.
I'm going to concentrate on the next step meantime - create offline PTX and load it to GPU.
Then I need to link the loaded function with my program (eventually I need an array of many functions on GPU to be linked).

@asuhan Another question regarding PTX. After loading PTX there is cuModuleGetFunction that requires a function name to be provided as a parameter.
Where do I find the target function name SLJIT is generating?

@leongor Should always be anon_func, you can also retrieve it with LLVMGetValueName(compiler->llvm_func) since the commit I've pushed today.

@asuhan Is there a way to set compiler->llvm_func to my own function name? I'm going to generate many PTX functions together and build one cubin from them.

@asuhan Got it!
LLVMSetValueName(LLVMGetNamedFunction(compiler->llvm_module, "anon_func"), "new_func") did the job.


Outlook mail tool added using Add tool is not recognized as tool available for chatting .

Electron built from commit id https://github.com/gptscript-ai/desktop/commit/59351d72bdcd4eab1b1e454c535ae0d965c37fa5
When I use "Add tool" option to add outlook mail and then ask any email question, I always get this response
To get the emails you received in your inbox today, I'll need to access your email account. This typically requires OAuth2 authentication.

Would you like to proceed with setting up the necessary permissions? If so, please provide the name of the email service you are using (e.g., Gmail, Outlook).

I had to respond with Use my existing OAuth2 authentication. which succeeds sometimes and gets email.
But also seen it fail with the "model is required" error.
I have seen this behavior only when adding outlook in chat using "Add tools".

This issue is not seen when chatting with a assistant that has outlook mail tool.

Related issue - https://github.com/gptscript-ai/desktop/issues/178

I believe this is now fixed. The confirmation prompts are still broken until I can merge #196, but as soon as I add the tool dynamically, I asked the LLM to list my inbox emails and it was able to do it successfully.

(This likely was fixed when I improved the titles of the Outlook Mail OpenAPI files in this PR.)

I dont see this issue anymore when testing with latest version outlook mail tool.

@g-linville I am seeing the same behavior again when testing with latest buids

^ That should be fixed now.

This issue is not seen anymore when testing with latest version of electron - 523bf344b9e and latest versions of the outlook mail tool.


Bogus duplicate dependency error

Description
dependency on 'https://github.com/apple/swift-docc-plugin.git' conflicts with dependency on 'https://github.com/apple/swift-docc-plugin.git' which has the same identity 'swift-docc-plugin'
https://github.com/hylo-lang/Lotsawa/actions/runs/7483973548/job/20370062536#step:7:54
Expected behavior
None of that.
Actual behavior
That.
Steps to reproduce

Check out https://github.com/hylo-lang/Lotsawa at the spm-dup-identity-bug tag.
cd into the resulting directory

export LOTSAWA_ENABLE_DOC_GENERATION=1
mkdir -p _site/docc
swift package --allow-writing-to-directory ./_site \
generate-documentation \
--product Lotsawa \
--output-path _site/docc \
--experimental-documentation-coverage --level detailed \
--enable-inherited-docs \
--transform-for-static-hosting \
--hosting-base-path "Lotsaws/docc/Lotsawa" \
--source-service github \
--source-service-base-url "https://github.com/hylo-lang/Lotsawa/blob/main" \
--checkout-path "$(pwd)"

Swift Package Manager version/commit hash
Swift 5.9.0
Swift & OS version (output of swift --version ; uname -a)
swift-driver version: 1.87.3 Apple Swift version 5.9.2 (swiftlang-<IP_ADDRESS>.56 clang-15<IP_ADDRESS>.5)
Target: arm64-apple-macosx14.0
Darwin DaveA-MBP16.local 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:53:18 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T6000 arm64

PEBKAC!!!

Out of curiosity, what was the user error?

Hmm yeah that is a nasty gotcha, it would be really nice/cool if the dependency declaration source location could be tracked and included in diagnostics. I think that would have helped debug your issue much quicker. e.g.
/foo/bar/Package.swift:30:10: error: dependency on 'https://github.com/apple/swift-docc-plugin.git' conflicts with dependency on 'https://github.com/apple/swift-docc-plugin.git' which has the same identity 'swift-docc-plugin'

/foo/bar/Package.swift:40:11: note: conflicting dependency on  'https://github.com/apple/swift-docc-plugin.git' here


Can't find Sign-in option

Issue Type: Feature Request
Hello Sir,
I have using visual studio code . And I unable to find sign in option so I can  login using my microsoft accout to use benifits. Please solve it.
VS Code version: Code 1.32.3 (a3db5be9b5c6ba46bb7555ec5d60178ecc2eaae4, 2019-03-14T23:38:49.842Z)
OS version: Linux x64 4.4.0-21-generic

There is no "sign in" option. vscode is free 😁


Add --group-by to shuffle groups instead of individual songs.

This PR addresses issues like #8 and #43, allowing users to shuffle by whatever subgroups they want. It also adds a handy --by-album flag that groups by album.

One of the new shuffle-by group tests is failing exclusively on travis, and I'm not getting any useful debug information for it. I've requested "debug build" access from travis to debug there. Until then, this PR will need to wait unfortunately.

Finally got access to travis's debug interface. Looks the integration tests were not waiting for MPD to fully update its database before starting ashuffle, so ashuffle was unable to find some songs in the huge database. Unfortunately, this means that the performance tests were not running against the entire "huge" DB 😦 . With the update-issue fixed, the tests regress a bit (down to ~750ms). That should be improved on in future versions.


get Parsing error: The keyword 'import' is reserved

module.exports = {
  parser: '@typescript-eslint/parser',
  plugins: ['@typescript-eslint/eslint-plugin'],
  extends: [
    'plugin:@typescript-eslint/recommended',
    'plugin:import/recommended',
    'plugin:import/typescript',
    'plugin:react/recommended',
    'plugin:react-hooks/recommended',
    'plugin:react/jsx-runtime',
    'plugin:prettier/recommended',
  ],
  settings: {
    'import/resolver': {
      typescript: {
        project: 'tsconfig.json',
      },
    },
    react: {
      createClass: 'createReactClass', // Regex for Component Factory to use,
      // default to "createReactClass"
      pragma: 'React', // Pragma to use, default to "React"
      fragment: 'Fragment', // Fragment to use (may be a property of <pragma>), default to "Fragment"
      version: 'detect', // React version. "detect" automatically picks the version you have installed.
      // You can also use `16.0`, `16.3`, etc, if you want to override the detected value.
      // default to latest and warns if missing
      // It will default to "detect" in the future
      flowVersion: '0.53', // Flow version
    },
  },
  env: {
    browser: true,
    node: true,
    jest: true,
  },
  overrides: [
    {
      files: ['*.ts', '*.tsx'],
      parserOptions: {
        project: 'tsconfig.json',
        sourceType: 'module',
        ecmaVersion: 2020,
      },
    },
  ],
}

this is my eslint.config.js, I had add .vscode in my project.
when use eslint V7 this is right.

@WebCaiQingChong  does the validation happen correctly when you use eslint in the terminal. If this is the case can you please provide me with a GitHub repository I can clone that demos what you are experiencing.

It isn't work in the termainal too. this is the git repo: https://github.com/WebCaiQingChong/useBox

my global eslint version is v8.2.0, my vscode-eslint version is latest v2.2.2

It also looks like you are using a custom eslint.config.js config file name. You need to let VS Code know about this as well using the eslint.options setting. Please note that this setting is ESLint version dependent and reflects the ESLint API documented here: https://eslint.org/docs/developer-guide/nodejs-api#-new-eslintoptions

Saw you are setting it in the settings.json however the format you use is for the new ESLint class API. You either need to use the CLIEngine options in eslint.options or set eslint.useESLintClass to true.

Works with the eslint.useESLintClass setting

thank you very mach,  i have a try.

it works for me,thank you very mach.


Add support for themes

Following the same principles as plugins :

[x]  _themes collection
[x] Github Pages are list in _data/github_pages.yml
[x] Theme screenshots
[x] 10 themes listed on the homepage with screenshot
[x] Theme listing with screenshot
[x] Theme detail with screenshot

@jekyll/plugin-core added only Github Pages for now, will add some gem-based themes later.

Preview on Github Pages on my repo: https://dirtyf.github.io/directory/


OpenAPI specification for submission and logging endpoints

We should have a dedicated OpenAPI specification file for just the API endpoints which are relevant to the systems submitting to the server in order to simplify client development.
-[ ] Add OpenAPI specs for logging and submission endpoints to README
-[ ] Add OpenAPI specs for logging and submission endpoints to wiki
-[ ] Test client generated code and add documentation on how to use it

They are already in the /doc folder in our repository.
I guess we could add swagger-editor badges and mention them more prominently in the readme / wiki.
Also, testing these would be a thing

We would probably also want a process to generate them (more or less) automatically

I recently implemented an approach to have those generated automatically and failed, since there aren't multiple OpenApiPlugins allowed in javalin.
However, I asked there what they think: https://github.com/tipsy/javalin/issues/1108

Ultimately, this is now coupled with tipsy/javalin#1117 and #206

Moved to milestone 0.6-alpha due to dependency to #206


Allow explicit server name in tls.Config

[ ] Link to issue, e.g. Resolves #NNN
[ ] Documentation added (if applicable)
[ ] Tests added
[ ] Branch rebased on top of current master (git pull --rebase origin master)
[ ] Changes squashed to a single commit (described here)
[ ] Build is green in Travis CI
[ ] You have certified that the contribution is your original work and that you license the work to the project under the Apache 2 license

Changes proposed in this pull request:

The ServerName member of the tls.Config struct will not be overridden when connecting to a gateway if the provided tls.Config struct already has a value set.

Rationale
(same basic deal as https://github.com/nats-io/nats-streaming-server/pull/767)
I am integrating NATS into Chef's Automate product. Automate internally has a Microservices architecture with connections between services secured by mutual TLS cert verification. Automate is usually deployed by our customers on their own infrastructure so we have elected not to rely on using hostnames in our certificates' CN fields; instead we use the internal service names (i.e., something like event-service instead of host.example.com).
This already works for NATS go clients using the option to set a custom tls.Config but when using the gateway feature, the ServerName is getting overwritten.
Happy to look into adding tests if this approach to solving the problem looks good.
/cc @nats-io/core

Coverage increased (+0.02%) to 91.534% when pulling 8ba3abef6ac918a00e320af223a034d3b1235e61 on danielsdeleo:dan/allow-explicit-server-name-in-gateway into 04deabf27d203a2b9ed2da4196cbd9f6d7de4d7b on nats-io:master.

Merging now.. will follow up right away with a test for that.


cache data before write

What problem does this PR solve?
cache data before write to TiDB in order to avoid recaculating
What is changed and how it works?
call DataSet.persist
Check List
Tests

No code

/run-all-tests

/run-all-tests


added transferring action repositories section

Why:
closes:  #7573

What's being changed (if available, include any code snippets, screenshots, or gifs):

Check off the following:

[x] I have reviewed my changes in staging, available via the View deployment link in this PR's timeline.

For content changes, you will also see an automatically generated comment with links directly to pages you've modified. The comment won't appear if your PR only edits files in the data directory.

[x] For content changes, I have completed the self-review checklist.

Thanks for the PR @SoundaryaKoutharapu, I'll get this triaged for review! ⚡

@sabrowning1
I wanted to thank you for your valuable input and the changes you proposed.
I've incorporated the changes you recommended.

Hey @sabrowning1
I made the changes you suggested
can you please review it

I am truly sorry I don’t want this and have no idea who asked you to do
all of this.
*Kevin Cox *
Sent from Gmail Mobile
On Fri, Oct 13, 2023 at 9:35 PM Soundarya Koutharapu <
@.***> wrote:

Hey @sabrowning1 https://github.com/sabrowning1
I made the changes you suggested
can you please review it
—
Reply to this email directly, view it on GitHub
https://github.com/github/docs/pull/28973#issuecomment-1762592428, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/BC4LJGKAR33ZFZXXNWXOTM3X7IJABANCNFSM6AAAAAA5WXS3XY
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Hey @sabrowning1,
would like to know whether this pr could be considered for Hactoberfest
can you please let me know?


could I use with .json, .md file?

could I use with .json, .md file?

This has been open for ages, sorry about that. This is possible by setting the extensions option: https://github.com/nathantreid/meteor-static-assets#options


Slice bounds out of range in decodeLineOfMediaPlaylist #2.

Hi! I found slice bounds out of range ( occurs at reader.go:796 ) during testing with go-fuzz.

Crash artifacts - crash.zip

Harness -  fuzz.zip

P. S:
Perhaps I described an incorrect wrapper and therefore am open to criticism.

Fix it


The image can't be use as a iscsi boot?

I use the image to install on the baremetal local disk . It works well, But if use it as a iscsi boot for ipxe, it throws the error that I have changed the hardware of software ,then it can't go on start

Hello,
Can you share a printscreen of the exact error you are getting?
Thank you,
Adrian

Closing issue, as no updates were received from the OP.


KeyError: 'device' when adding my new laptop

I'm trying to add my new laptop to HA by utilizing this integration.
Unfortunately, I'm getting the following error presented:
Logger: homeassistant.config_entries
Bron: config_entries.py:586
Eerst voorgekomen: 09:16:17 (5 gebeurtenissen)
Laatst gelogd: 09:51:15

Error setting up entry DESKTOP-2Z5DV73-HassAgent for hass_agent
Error setting up entry G10LaptopHassAgent for hass_agent
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 586, in async_setup
    result = await component.async_setup_entry(hass, self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/hass_agent/__init__.py", line 135, in async_setup_entry
    _logger.debug("setting up device from config entry: %s [%s]", entry.data["device"]["name"], entry.unique_id)
                                                                  ~~~~~~~~~~^^^^^^^^^^
KeyError: 'device'

The first setting up entry, starting with DESKTOP was the default value (added HassAgent for better visibility inside). The second entry starting with G10 is a new name I picked because of testing, but the error is the same.
The port is configured as 5115, but it happens with all configured ports.
HassAgent is beta2 of version 2.1.0. HA is 2024.7.2.

I just don't know where my error is at.
I opened this bug because I thought I was merely the integration which was acting up.
But now I'm starting to suspect the Hass.Agent itself. On my old laptop, called 'martijnlaptop' it immediately gets discovered by MQTT and thus the integration. When removing Hass.Agent device, and restart of HA, it comes back with the message new devices have been discovered.
With my new laptop this doesn't happen. When I rename DESKTOP-2Z5DV73 to any other device name, it just stops pushing changes to HA. The changes do get reflected with MQTT but it doesn't get detected by HA.
With extended logging enabled the only error is this:
2024-07-15 21:14:18.432 +02:00 [FTL] [PROGRAM] FirstChanceException: Unable to cast COM object of type 'System.__ComObject' to interface type 'NAudio.CoreAudioApi.Interfaces.IMMDevice'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{D666063F-1587-4E43-81F1-B948E807363F}' failed due to the following error: Interface wordt niet ondersteund (0x80004002 (E_NOINTERFACE)).
System.InvalidCastException: Unable to cast COM object of type 'System.__ComObject' to interface type 'NAudio.CoreAudioApi.Interfaces.IMMDevice'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{D666063F-1587-4E43-81F1-B948E807363F}' failed due to the following error: Interface wordt niet ondersteund (0x80004002 (E_NOINTERFACE)).
   at System.StubHelpers.StubHelpers.GetCOMIPFromRCW(Object objSrc, IntPtr pCPCMD, IntPtr& ppTarget, Boolean& pfNeedsRelease)
   at NAudio.CoreAudioApi.Interfaces.IMMDevice.GetState(DeviceState& state)
   at NAudio.CoreAudioApi.MMDevice.get_State()
   at HASS.Agent.Shared.Managers.Audio.AudioManager.<>c.<GetDevices>b__18_0(InternalAudioDevice d) in D:\a\HASS.Agent\HASS.Agent\src\HASS.Agent\HASS.Agent.Shared\Managers\Audio\AudioManager.cs:line 225
   at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
   at HASS.Agent.Shared.Managers.Audio.AudioManager.GetDevices() in D:\a\HASS.Agent\HASS.Agent\src\HASS.Agent\HASS.Agent.Shared\Managers\Audio\AudioManager.cs:line 225

[2024-07-15] HASS.Agent_20240715.log
Attached logfile, but maybe I should open this with the Hass.Agent github instead?

I'm having the same issue
No new devices being discovered (specifically media players)
Manual MQTT sensors are discovered
I tried two different PCs with the same results

Hello, apologies for the delay in response.
As an introduction statement: the integration requires a full rework - I'm working on this as the coming-going issues like "new device not being discovered", "device appearing as discovered despite already being configured" etc. are the result of the integration being written back in the day by original team and then "maintained" to the best of efforts.
The target is to rewrite it from scratch.
@mhoogenbosch I see that you've mentioned ports, are you using local API?
As a rule of thumb the better way is to go with the auto-discovery as the local API is in "life-support" stage (although looking at the issues I understand why people still use local API)
As a first step I'd like to ask both of you to:

make sure if you're using the forked version of integration - https://github.com/hass-agent/HASS.Agent-Integration/issues/2
remove all discovered/configured HASS.Agent devices (in the integration details page)
close all HASS.Agent instances that might be running (including satellite service)
in MQTT-Explorer, remove all topics/messages under "hass.agent/devices"
restart Home Assistant
start all HASS.Agent instances
restart Home Assistant
the devices should appear as discovered - configure them and optionally restart Home Assistant again

Just to be clear - the restarts are required due to integration issues and not Home Assistant issues.

for anyone interested, sequence mentionned above by @amadeo-alex works well for me  !
thanks for the time invested on this agent to the team


Code cleanup

@brandonpollett,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot


Webpack 5 support

Updated dependencies as old versions were vulnerable
Got rid of the (node:15752) [DEP_WEBPACK_COMPILATION_ASSETS] DeprecationWarning

Closes #35

I also want to note that the tests fileOptions, zipOptions, fileOptions and zipOptions need to be removed, since they are inconsistent

Merged and released through #37, thanks.
The test harness deletes dist/ when the tests pass; seeing dirty files when I commit reminds me to fix the tests and not commit broken code. (but I kept your change anyways cause it doesn't really matter)

Glad to see that this repository is still alive! :)

The test harness deletes dist/ when the tests pass; seeing dirty files when I commit reminds me to fix the tests and not commit broken code

Oh, got it. I was just confused that after running the tests, about two hundred files were expected to be committed


ModuleNotFoundError: No module named 'urllib2'

I am getting the error in the title (and below in the full traceback) when I attempt to run any az commands.
I run az login from powershell or cmd and I am prompted to open a web browser and enter the code. I do so and associate with my work account, and then from my local prompt I get a json response with the 2 VS subscriptions associated with my account, then I get this error. I started running other commands (disk list, consumption usage list, 'interactive')
When I entered interactive mode it prompted me to send telemetry and based on the error I said no. I kept getting the error inside interactive mode, but when I exited I do not get the error any more.
I don't mind sending telemetry, but I am going to keep it turned off as that seems to workaround getting this exception.
Environment summary
MSI installer
Windows 10
Powershell and cmd
CLI version output:
azure-cli (2.0.32)
acr (2.0.24)
acs (2.0.32)
advisor (0.5.1)
ams (0.1.0)
appservice (0.1.32)
backup (1.1.1)
batch (3.2.2)
batchai (0.2.2)
billing (0.1.8)
cdn (0.0.14)
cloud (2.0.13)
cognitiveservices (0.1.13)
command-modules-nspkg (2.0.1)
configure (2.0.15)
consumption (0.3.1)
container (0.1.23)
core (2.0.32)
cosmosdb (0.1.21)
dla (0.0.19)
dls (0.0.21)
dms (0.0.1)
eventgrid (0.1.12)
eventhubs (0.1.3)
extension (0.0.13)
feedback (2.1.1)
find (0.2.9)
interactive (0.3.20)
iot (0.1.20)
keyvault (2.0.21)
lab (0.0.22)
monitor (0.1.6)
network (2.1.0)
nspkg (3.0.2)
profile (2.0.24)
rdbms (0.2.2)
redis (0.2.13)
reservations (0.1.2)
resource (2.0.28)
role (2.0.23)
servicebus (0.1.2)
servicefabric (0.0.12)
sql (2.0.25)
storage (2.0.32)
vm (2.0.31)
Python location 'C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe'
Extensions directory 'C:\Users\brandon.azure\cliextensions'
Python (Windows) 3.6.1 (v3.6.1:69c0db5, Mar 21 2017, 17:54:52) [MSC v.1900 32 bit (Intel)]
Legal docs and information: aka.ms/AzureCliLegal
C:\Users\brandon>Traceback (most recent call last):
File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\telemetry_upload.py", line 8, in
import urllib2 as HTTPClient
ModuleNotFoundError: No module named 'urllib2'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\telemetry_upload.py", line 12, in
import urllib.request as HTTPClient
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\urllib\request.py", line 88, in
import http.client
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\http\client.py", line 71, in
import email.parser
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\email\parser.py", line 12, in
from email.feedparser import FeedParser, BytesFeedParser
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\email\feedparser.py", line 27, in
from email._policybase import compat32
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\email_policybase.py", line 9, in
from email.utils import _has_surrogates
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\email\utils.py", line 29, in
import socket
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\socket.py", line 49, in
import _socket
ImportError: DLL load failed: %1 is not a valid Win32 application.

Closing due to inactivity. Please open if this is still an issue.


[FEATURE] Add self-examination and calibration routines to RDK

Is your feature request related to a problem? Please describe.
It would be helpful to have the ability to trigger self-examination and calibration routines remotely, without using the UI.
Describe the solution you'd like
Add the self-examination and calibration routines to the RDK.
Describe alternatives you've considered
I tried to run the joint performance test by remotely executing a plan.
The plan execution runs fine but there is no option to get/save/examine data nor apply it to the robot.
Adding a PASS/FAIL test result would be helpful (similarly to what done when executing from the Elements UI).

@yellownebula10 This feature will be added in RDK v1.5.

That's great news, thank you @pzhu-flexiv! Do you know when RDK v1.5 will be released? Will this feature allow to get the pass/fail values from audit and to apply calibration values from the RDK?

@yellownebula10 Due to limited bandwidth, this feature will have to be postponed to v1.6. Thanks for your understanding.

Hi @pzhu-flexiv,
what is the timeline for RDK v1.6? Will it be compatible with SW v3.7?

RDK v1.6 will be released with SW v3.8 by the end of Dec. RDK v1.5 is for SW v3.7


Bump iot to 2.0.11

update confirmed by @GermanBluefox at 6.11.2023 by telegram

2.0.11 created 20.06.2023
2200 user
OK


tools/jpackage/windows/Win8282351Test.java in jdk_tools_0 fails on most machines, passing on test-azure-win2012r2-x64-3

tools/jpackage/windows/Win8282351Test.java (in jdk_tools target), fails in error on build-alibaba-win2012r2-x64-2, test-ibmcloud-win2012r2-x64-1, but passes on test-azure-win2012r2-x64-3:
ERROR: Expected [0]. Actual [1]: Check command [C:\Users\Jenkins\workspace\Test_openjdk17_hs_extended.openjdk_x86-64_windows_testList_1\openjdkbinary\j2sdk-image\bin\jpackage.exe --dest .\test\output --name Win8282351Test$-$$-$$$ --type msi --app-image .\test\appimage\Win8282351Test --win-menu --win-shortcut --verbose](12) exited with 0 code
[10:20:12.777] [  FAILED  ] Win8282351Test.test; checks=4

test suite/name (e.g, BUILD_LIST, TARGET, CUSTOM_TARGET)?
a link into recent Test_ job on https://ci.adoptopenjdk.net which showed the failure
Hyperlink to re-run in Grinder:
Is there an existing issue elsewhere covering this?
Which machine(s) does it work on? test-azure-win2012r2-x64-3 (see Grinder 6319 for success run), not sure what version of WiX toolkit is present on this machine.
Which machine(s) does it fail on? Most.  deep history shows failing on build-alibaba-win2012r2-x64-2, test-ibmcloud-win2012r2-x64-1, build-ibmcloud-win2012r2-x64-2, build-azure-win2012r2-x64-2 WiX <IP_ADDRESS>10 detected
mentioned in https://github.com/adoptium/aqa-tests/issues/4240#issuecomment-1382741799

Test Info
Test Name: jdk_tools_0
Test Duration: 17 min 38 sec
Machine: build-alibaba-win2012r2-x64-2
TRSS link for the test output: https://trss.adoptium.net/output/test?id=63c219e9ed0b30773ba354a9
Build Info
Build Name: Test_openjdk17_hs_extended.openjdk_x86-64_windows_testList_1
Jenkins Build start time: Jan 13 2023, 06:26 pm
Jenkins Build URL: https://ci.adoptopenjdk.net/job/Test_openjdk17_hs_extended.openjdk_x86-64_windows_testList_1/76/
TRSS link for the build: https://trss.adoptium.net/allTestsInfo?buildId=63c218eaed0b30773ba34d89
Java Version
openjdk version "17.0.6" 2023-01-17
OpenJDK Runtime Environment Temurin-17.0.6+8 (build 17.0.6+8)
OpenJDK 64-Bit Server VM Temurin-17.0.6+8 (build 17.0.6+8, mixed mode, sharing)
This test has been failed 2 times since Jan 12 2023, 06:16 am
Java Version when the issue first seen
openjdk version "17.0.6" 2023-01-17
OpenJDK Runtime Environment Temurin-17.0.6+8 (build 17.0.6+8)
OpenJDK 64-Bit Server VM Temurin-17.0.6+8 (build 17.0.6+8, mixed mode, sharing)
Jenkins Build URL: https://ci.adoptopenjdk.net/job/Test_openjdk17_hs_extended.openjdk_x86-64_windows_testList_1/75/
The test failed on machine build-alibaba-win2012r2-x64-2 1 times
The test failed on machine test-ibmcloud-win2012r2-x64-1 1 times
Rerun in Grinder

Rerunning
build-alibaba-win2012r2-x64-2 https://ci.adoptopenjdk.net/job/Grinder/6358/console
test-ibmcloud-win2012r2-x64-1 https://ci.adoptopenjdk.net/job/Grinder/6359/console
test-azure-win2012r2-x64-3 https://ci.adoptopenjdk.net/job/Grinder/6360/console

Passes on only. I'll take test-ibmcloud-win2012r2-x64-1 offline, but keep build-alibaba-win2012r2-x64-2 online as its a build machine with no test label

a few other tests got the same issue only pass on test-azure-win2012r2-x64-3
javax/net/ssl/SSLSocket/Tls13PacketSize.java.Tls13PacketSize
security/infra/java/security/cert/CertPathValidator/certification/GoogleCA.java.GoogleCA
tools/jpackage/share/PerUserCfgTest.java.PerUserCfgTest
tools/jpackage/share/ServiceTest.java.ServiceTest
tools/jpackage/windows/Win8282351Test.java.Win8282351Test
tools/jpackage/windows/WinLongVersionTest.java#id1.WinLongVersionTest_id1

Since test-azure-win2012r2-x64-* machines have been replaced with test-azure-win2022-x64-* machines, we do not have any machines where this jdk_tools testcase can pass.

@steelhead31 - could this be a similar issue to what you have seen where 'stuff' as a service on Windows machines has weird characters in the path?  I am noting the error msg (see https://github.com/adoptium/infrastructure/issues/2885#issuecomment-1890709946) that reads: The following string format was not expected by the external UI message logger: "The Windows Installer Service could not be accessed. This can occur if the Windows Installer is not correctly installed. Contact your support personnel for assistance.".

@steelhead31 - could this be a similar issue to what you have seen where 'stuff' as a service on Windows machines has weird characters in the path? I am noting the error msg (see #2885 (comment)) that reads: The following string format was not expected by the external UI message logger: "The Windows Installer Service could not be accessed. This can occur if the Windows Installer is not correctly installed. Contact your support personnel for assistance.".

@smlambert its possible, but I think there are a couple of more likely culprits related to windows configuration / group policy settings, the later versions of windows lock down the windows installer service quite a lot, and some of the azure windows images have quite locked down permissions, if you'd like I can break this out into a separate issue, and I can investigate further?... It'd be handy if somebody could give me some pointers on the aqa targets required for me to be able to test this on a command line though!

re: https://github.com/adoptium/infrastructure/issues/2885#issuecomment-1891644837
in Jenkins, TARGET=jdk_tools
Steps to run on outside of Jenkins / on commandline (presumes you have a Temurin JDK downloaded onto the machine into /jdkDir replace with actual location of your JDK that you want to test):
git clone https://github.com/adoptium/aqa-tests.git
cd aqa-tests
export TEST_JDK_HOME=/jdkDir
export BUILD_LIST=openjdk
export CUSTOM_TARGET=tools/jpackage/windows/Win8282351Test.java
./get.sh
cd TKG
make compile
make _jdk_custom

For a more complete run of all testcases in the jdk_tools target, not just Win8282351Test, you could change the last line to:
make _jdk_tools

Test passed on win2022 machine 3 .
# AQACert.log content:
#
# Hostname: test-azure-win-2
#SHA.txt content:
#
# Timestamp: Tue Feb 13 11:42:48 2024 UTC
# CUSTOM_TARGET: tools/jpackage/windows/Win8282351Test.java
# RESULTS_SUMMARY: TOTAL: 1 EXECUTED: 1 PASSED: 1 FAILED: 0 DISABLED: 0 SKIPPED: 0
1..1
ok 1 - jdk_custom_0
 ---
   duration_ms: 171050
 ...

Specific test, does pass as both adoptopenjdk and jenkins user on all 3 test-azure-win2022-x64-x machines, testing all tools tests... testing with jdk17.0.9+9.1

I've managed to replicate the error via grinder on one machine https://ci.adoptium.net/job/Grinder/8812/console , despite it passing locally.. so it definitely seems machine specific

Works ok via cli & jenkins on test-azure-win2022-x64-3 , works via cli on test-azure-win2022-x64-1 & test-azure-win2022-x64-2, however fails via grinder on both test-azure-win2022-x64-1 & test-azure-win2022-x64-2.. investigating further, suspect its a windows oddity.... as all 3 machines were built from the same playbooks.

This is now working on all 3 machines..  ( machine 1: https://ci.adoptium.net/job/Grinder/8826/console , machine 2 : https://ci.adoptium.net/job/Grinder/8827/console & machine 3 : https://ci.adoptium.net/job/Grinder/8825/console ).. .Thanks muchly!

I believe this to be the same root cause as the WSL detection issue, in that the PATH environment variable was being corrupted by the windows service wrapper used to run jenkins as a service, these machines were switched to using a CLI startup, and I believe this is the reason the tests now pass.

@steelhead31 So we can close this?

@karianna yes, I was just waiting on a final confirmation, but it all looks good now.


Fix not showing team allocations when worker allocation is closed

Describe this PR
What is the problem we're trying to solve
if i try to deallocate just the worker, it removes the team allocation too.
What changes have we introduced
Add descriptions of what changes were made to address the problem and provide a solution.
Code and test
Checklist

[ ] Added end-to-end (i.e. integration) tests where necessary e.g to test complete functionality of a newly added feature
[x] Added tests to cover all new production code
[ ] Added comments to the README or updated relevant documentation (add link to documentation), where necessary.
[ ] Checked all code for possible refactoring
[x] Code pipeline builds correctly

This was branched off from here, hence you see extra code and test.
Please merge branch above first.


Minor wording update to README.md for "Extension Pack"

"Extension Pack" not "Expansion Pack"

Haha yes, +1. I was gonna make the same PR after running into #42. I had seen the expansion pack thing on the readme but when searching for "expansion" on the downloads page found nothing and ignored it.

Whoops, that's silly. Thanks!


gitignore fixes for monorepo

The leading slashes made it only work at the root level, rather than flow down into multiple projects

I've confirmed there's no notable changes with this update applied so I'll go ahead and merge.


Test Web: Bad rendering of buttons in custom dialog

Testing #83296:
Version: 1.40.0-insider
Commit: b7b21844e760e632f39bedf651da2bc93f54b0cc
Date: 2019-10-29T06:38:51.389Z
Browser: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0
Run the command Help -> About:

@sbatten while looking into https://github.com/microsoft/vscode/issues/83573 I noticed how we use max-width: fit-content for buttons in dialogs and notifications. I think there is a bug in FF that results in what we see here, but I also really do not remember why we used max-width in the first place. We probably want to overflow: ellipsis when there are many buttons, but I can still see that when switching to width: fit-content, which I think is exactly the intent (give buttons enough width to show their text).
I hope this is not causing regressions, let's see how it goes. It is changed for electron+web via https://github.com/microsoft/vscode/commit/557cf012ca0d1e25366af58185de5221d47b39a3

Verified out of sources.

@bpasero I tested with lots of buttons on a smaller window width and the effect is good. LGTM.


Get URL Calling by sending Encoded XML format in URL

Describe the bug
I am using org.springframework.cloud:spring-cloud-starter-openfeign:3.0.3 version for Feign client. trying to create GetMapping and send XML as encoded format in QueryParameter. but the server does not accept this encoded XML Format
Sample
My Code is
//@Headers({"Content-Type: application/xml"})
    //@Headers("Content-Type: application/x-www-form-urlencoded")
    @Headers({"Content-Type: application/json"})
    @GetMapping(value = "/abc.dll")
    ResponseEntity<String> serviceCall(@RequestParam(name = "API") String api, @RequestParam(name = "XML") String XML);.

Ideally it is constructing some what like below
https://secure.abc.com/abc.dll?API=Verify&XML=<AddressValidateRequest USERID%3D"023MBUSA5454"><Revision>1</Revision><Address><Address1>1 textile BENZ DR</Address1><Address2>1 textile BENZ DR</Address2><City>SANDY SPRINGS</City><State>GA</State><Zip5>30328 - 4312</Zip5><Zip4/></Address></AddressValidateRequest>
infact Feign client constructing similar URL but not giving proper response.
However the same URL tried from Browser and it is giving me actual required Output.

@dnyaneshwarjadhav thanks for submitting the issue.
Please learn how to properly format code and logs.
Please provide a minimal, complete, verifiable example that reproduces the issue.

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.


Can't seek back on encrypted hls VOD playlist

Question
i have a simple hls encrypted m3u8 generated by FFmpeg and served through python simple http server:
#EXTM3U
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:16
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-KEY:METHOD=AES-128,URI="key.bin"
#EXTINF:16.433333,
output0.ts
#EXTINF:14.433333,
output1.ts
#EXTINF:14.166667,
output2.ts
#EXTINF:14.966667,
output3.ts
#EXT-X-ENDLIST

it does play fine on react-native-video and i can seek forward.
however i can't seek backwards and the whole playback stops how can i fix this ? or is it a bug in the library ?

Is key.bin in the same path as the manifest?

I also see first segment exceeds the #EXT-X-TARGETDURATION

Is key.bin in the same path as the manifest? I guess it is as it plays at first, I don't think it is related, but as you are not setting attributes, can you remove the last commas on EXTINF?

Yes it is in the same path as the manifest, and the manifest is auto generated so i see no reason to remove the commas although i will try to.

I also see first segment exceeds the #EXT-X-TARGETDURATION, spec says it should be equal to nearest integer (which is true in your manifest, but I will try playing with it and setting it to 17)

after hours of searching it turns out that i needed to add #EXT-X-ALLOW-CACHE:YES to the manifest and it worked !

great news @3bodyZZ thanks for reporting it back.
What it is buggy now?

great news @3bodyZZ thanks for reporting it back.
What it is buggy now?

yeah and i think its because of the server i saw the logs it is very slow at handling get requests, i will try with another server and see.

Are you setting progressive support on the server for the ts files?
Related Headers:
"Accept-Ranges": "bytes",
 "Cache-Control": "public, max-age=3600",
  "Content-Length": file.size,

Server should handle Range header on requests


Added Hootsuite as twemproxy users

Hootsuite uses twemproxy in front of both Memcached and Redis services. Our blog post on our migration from MySQL and Memcached to Redis that uses Twemproxy is shown here: http://code.hootsuite.com/mysql-to-redis/

Sweet. I read through the migration blog post. Congrats on a successful migration :) Seems like a non trivial effort.
My hunch regarding timeouts with logging "turned on" might have to do with logging to EBS volume on AWS boxes instead of a Instance store. AWS is notorious about throttling I/O to EBS.


Improve UG features section

Addresses issues such as #137 and makes the UG features section more user-friendly

Codecov Report

Merging #156 (e042716) into master (7435896) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #156   +/-   ##
=========================================
  Coverage     54.20%   54.20%
  Complexity      429      429
=========================================
  Files            92       92
  Lines          1760     1760
  Branches        210      210
=========================================
  Hits            954      954
  Misses          756      756
  Partials         50       50

Impacted Files
Coverage Δ

.../eatwhere/foodguide/logic/commands/AddCommand.java
100.00% <ø> (ø)

...where/foodguide/logic/commands/FindTagCommand.java
22.22% <ø> (ø)

.../eatwhere/foodguide/logic/commands/TagCommand.java
0.00% <ø> (ø)

...atwhere/foodguide/logic/commands/UntagCommand.java
0.00% <ø> (ø)

...rc/main/java/eatwhere/foodguide/ui/HelpWindow.java
0.00% <ø> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


The serializable class Product does not declare a static final serialVersionUID field of type long

/Users/pierreslam/Desktop/cloneandroid/openfoodfacts-androidapp/app/src/main/java/openfoodfacts/github/scrachx/openfood/models/Product.java
Warning:(24, 14) The serializable class Product does not declare a static final serialVersionUID field of type long

@teolemon I have added the serialVersionUID and from what I have researched, this should work as expected. Please correct me if I am wrong.


Check for user preferences during install or uninstall.

Branch Develop
Target 0.14.x
Expected Behavior
During install, check for old user preferences and remove them.
During uninstall, if possible, remove the old user preferences.

Updates @cpauya?

Hi @MCGallaspy - sorry this is still outstanding as of today.
The approach here is to modify the ka-lite-remover.sh - this same script is copied as KA_Lite_Uninstall.tool after installation.
Will work on this this week.

Fixed in #357