https://services.nvd.nist.gov/ endopint is giving 503 Service Unavailable

OWASP Client version: 9.0.8
We are seeing the services endpoint timeout from last couple of days
RUN /tmp/dependency-check/bin/dependency-check.sh --updateonly  --nvdApiKey ${NVD_API_KEY} --nvdApiDelay 3000
Picked up JAVA_TOOL_OPTIONS: -Dhttp.proxyHost=*** -Dhttp.proxyPort=80 -Dhttps.proxyHost=*** -Dhttps.proxyPort=80
[INFO] Checking for updates
[WARN] NVD API request failures are occurring; retrying request for the 5 time
[WARN] NVD API request failures are occurring; retrying request for the 6 time
[WARN] NVD API request failures are occurring; retrying request for the 7 time
[WARN] NVD API request failures are occurring; retrying request for the 5 time
[WARN] NVD API request failures are occurring; retrying request for the 6 time
[WARN] NVD API request failures are occurring; retrying request for the 7 time

We have verified the nvdApiKey is valid.
The curl response to the endpoint gives the response below
curl -H "Accept: application/json" -H "apiKey:******" -v https://services.nvd.nist.gov/rest/json/cves/2.0\?cpeName\=cpe:2.3:o:microsoft:windows_10:1607:\*:\*:\*:\*:\*:\*:\*
* Uses proxy env variable https_proxy ***
*   Trying <IP_ADDRESS>...
* TCP_NODELAY set
* Connected to <proxy> (<IP_ADDRESS>) port 80 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to services.nvd.nist.gov:443
> CONNECT services.nvd.nist.gov:443 HTTP/1.1
> Host: services.nvd.nist.gov:443
> User-Agent: curl/7.61.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.0 200 Connection established
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=*.nvd.nist.gov
*  start date: May 31 01:13:03 2024 GMT
*  expire date: Aug 29 01:13:02 2024 GMT
*  subjectAltName: host "services.nvd.nist.gov" matched cert's "*.nvd.nist.gov"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET /rest/json/cves/2.0?cpeName=cpe:2.3:o:microsoft:windows_10:1607:*:*:*:*:*:*:* HTTP/1.1
> Host: services.nvd.nist.gov
> User-Agent: curl/7.61.1
> Accept: application/json
> apiKey:****
>
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
< HTTP/1.1 503 Service Unavailable
< content-length: 107
< cache-control: no-cache
< content-type: text/html
<

503 Service Unavailable

No server is available to handle this request.

* Connection #0 to host *** left intact

Is there a change in Service endpoint? Can we get some assistance with this.

From the Web;
The endpoint intermittently shows

And also TBD with initial public draft

We also have myriads of jobs/pipelines failing due to this issue. Though this should teach us a lesson not to rely on its availability.

@jeremylong Can you please help us with this issue ?

Same here 9.2.0

The NVD API (https://nvd.nist.gov/developers/vulnerabilities) seems on and off at the moment
[root@blabla ~]# curl -H "Accept: application/json" -v https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2019-1010218
* About to connect() to services.nvd.nist.gov port 443 (#0)
*   Trying <IP_ADDRESS>...
* Connected to services.nvd.nist.gov (<IP_ADDRESS>) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=*.nvd.nist.gov
*       start date: May 31 01:13:03 2024 GMT
*       expire date: Aug 29 01:13:02 2024 GMT
*       common name: *.nvd.nist.gov
*       issuer: CN=R3,O=Let's Encrypt,C=US
> GET /rest/json/cves/2.0?cveId=CVE-2019-1010218 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: services.nvd.nist.gov
> Accept: application/json
>
< HTTP/1.1 200 OK
< content-type: application/json
< x-frame-options: SAMEORIGIN
< access-control-allow-origin: *
< access-control-allow-headers: accept, apiKey, content-type, origin, x-requested-with
< access-control-allow-methods: GET, HEAD, OPTIONS
< access-control-allow-credentials: false
< date: Mon, 01 Jul 2024 09:13:29 GMT
< content-length: 2639
< apikey: No
< strict-transport-security: max-age=31536000
<
{"resultsPerPage":1,"startIndex":0,"totalResults":1,"format":"NVD_CVE","version":"2.0","timestamp":"2024-07-01T09:13:30.227","vulnerabilities":[{"cve":{"id":"CVE-2019-1010218","sourceIdentifier":"josh@bress.net","published":"2019-07-22T18:15:10.917","lastModified":"2020-09-30T13:40:18.163","vulnStatus":"Analyzed","cveTags":[],"descriptions":[{"lang":"en","value":"Cherokee Webserver Latest Cherokee Web server Upto Version 1.2.103 (Current stable) is affected by: Buffer Overflow - CWE-120. The impact is: Crash. The component is: Main cherokee command. The attack vector is: Overwrite argv[0] to an insane length with execl. The fixed version is: There's no fix yet."},{"lang":"es","value":"El servidor web de Cherokee más reciente de Cherokee Webserver Hasta Versión 1.2.103 (estable actual) está afectado por: Desbordamiento de Búfer - CWE-120. El impacto es: Bloqueo. El componente es: Comando cherokee principal. El vector de ataque es: Sobrescribir argv[0] en una longitud no sana con execl. La versión corregida es: no hay ninguna solución aún."}],"metrics":{"cvssMetricV31":[{"source":"nvd@nist.gov","type":"Primary","cvssData":{"version":"3.1","vectorString":"CVSS:3.1\/AV:N\/AC:L\/PR:N\/UI:N\/S:U\/C:N\/I:N\/A:H","attackVector":"NETWORK","attackComplexity":"LOW","privilegesRequired":"NONE","userInteraction":"NONE","scope":"UNCHANGED","confidentialityImpact":"NONE","integrityImpact":"NONE","availabilityImpact":"HIGH","baseScore":7.5,"baseSeverity":"HIGH"},"exploitabilityScore":3.9,"impactScore":3.6}],"cvssMetricV2":[{"source":"nvd@nist.gov","type":"Primary","cvssData":{"version":"2.0","vectorString":"AV:N\/AC:L\/Au:N\/C:N\/I:N\/A:P","accessVector":"NETWORK","accessComplexity":"LOW","authentication":"NONE","confidentialityImpact":"NONE","integrityImpact":"NONE","availabilityImpact":"PARTIAL","baseScore":5.0},"baseSeverity":"MEDIUM","exploitabilityScore":10.0,"impactScore":2.9,"acInsufInfo":false,"obtainAllPrivilege":false,"obtainUserPrivilege":false,"obtainOtherPrivilege":false,"userInteractionRequired":false}]},"we* Connection #0 to host services.nvd.nist.gov left intact
aknesses":[{"source":"nvd@nist.gov","type":"Primary","description":[{"lang":"en","value":"CWE-787"}]},{"source":"josh@bress.net","type":"Secondary","description":[{"lang":"en","value":"CWE-120"}]}],"configurations":[{"nodes":[{"operator":"OR","negate":false,"cpeMatch":[{"vulnerable":true,"criteria":"cpe:2.3:a:cherokee-project:cherokee_web_server:*:*:*:*:*:*:*:*","versionEndIncluding":"1.2.103","matchCriteriaId":"DCE1E311-F9E5-4752-9F51-D5DA78B7BBFA"}]}]}],"references":[{"url":"https:\/\/i.imgur.com\/PWCCyir.png","source":"josh@bress.net","tags":["Exploit","Third Party Advisory"]}]}}]}

On another attempt right after:
[root@blabla ~]# curl -H "Accept: application/json" -v https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2019-1010218
* About to connect() to services.nvd.nist.gov port 443 (#0)
*   Trying <IP_ADDRESS>...
* Connected to services.nvd.nist.gov (<IP_ADDRESS>) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=*.nvd.nist.gov
*       start date: May 31 01:13:03 2024 GMT
*       expire date: Aug 29 01:13:02 2024 GMT
*       common name: *.nvd.nist.gov
*       issuer: CN=R3,O=Let's Encrypt,C=US
> GET /rest/json/cves/2.0?cveId=CVE-2019-1010218 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: services.nvd.nist.gov
> Accept: application/json
>
< HTTP/1.1 503 Service Unavailable
< content-length: 107
< cache-control: no-cache
< content-type: text/html
<

503 Service Unavailable

No server is available to handle this request.

* Connection #0 to host services.nvd.nist.gov left intact

No luck on the builds though, all of them are failing.

Also affected with Dependency-Check Core version 9.0.9.
Is there perhaps an official page by nvd that indicates availability? I couldn't find anything.

Workaround is false

another workaround:

download meta and json.gz files from https://nvd.nist.gov/vuln/data-feeds#JSON_FEED
put them on a local http server
configure this location as data feed:
http://internal-mirror.mycorp.com/nvdcve-{0}.json.gz

Workaround in our case: We have private build servers on which we have OWASP DC installed. Each team running their build runs with --noupdate, and every 4 hours we run a separate job to update all OWASP DC databases. In this case at least the builds from the teams keep running, and they never have to wait for updates. In case updating fails, and it has failed before, only we, the system team, are impacted.

probably not much this project can do anything about, altho it would be nice if it would fail fast.  rather than hanging for hours.

IMHO, it's probably at root due to https://github.com/jeremylong/DependencyCheck/issues/6746 and the NVD website getting a DDoS from many thousands of ODC clients constantly retrying requests that will never succeed (until fixed within ODC).

Any updates on this issue ? looks like 10.0.0 version has some bug while updating CVEs.

Try with setting api key --nvdApiKey that resolved issues for us with fetching data.
https://nvd.nist.gov/general/news/API-Key-Announcement
https://nvd.nist.gov/developers/request-an-api-key

Needing to use an API key to avoid certain errors/rate limiting has been true since v9 of ODC (especially if starting without an existing cache/database), but if you are recently upgrading from v8 -> v9 or v8 -> v10 you may have this as a root cause for your issue.
But there also appear to be some other load problems, at least yesterday.
As for bugs and compatibility problems with v10 and certain databases/setups, you probably need to follow https://github.com/jeremylong/DependencyCheck/issues/6746 . Right now I don't think there is any working version of ODC v9 or v10 that is fine across all setups. Might be OK with an H2 Database but definitely some issues with Postgres and MySQL? Not sure about v8.

Yes, the NVD API is having capacity issues right now. I have seen successful calls into the API - generally using an API key. There is not much the project (ODC) can do about this.
I am releasing 10.0.1 - which contains a few minor fixes if you are using postgres or mssql as a centralized database. The release also removes a bit of debug logging that makes it look like errors occurred when they did not.

Hi @jeremylong
Where can you follow status of that capacity issue? Do they have a page with a news feed?
I've also had issues for 2 days now and being not able to update our CVE database is becoming critical.

@zg2pro No, they don't.
They have widespread and well documented funding and governance problems which you can google and read about. They have had persistent periodic technical problems since the launch of the API especially.
In any case, if you are relying upon OSS software and the US government's NVD for your critical workflows rather than a commercial vendor, it's all best efforts.
Personally (just a hunch), I have a feeling that the NVD capacity issues will abate as people update to ODC 10.0.1+ and gradually get their caches back in sync - or another batch get forced to relook at their caching strategy as a result of the issues.
In any case, my H2-database ODC cache updated slowly but successfully  on 10.0.1 after 3 days of nothing flowing through, so it does seem generally to be working, especially if you are not trying to populate a database from scratch.

In any case, my H2-database ODC cache updated slowly but successfully on 10.0.1 after 3 days of nothing flowing through, so it does seem generally to be working, especially if you are not trying to populate a database from scratch.

Sounds like I just picked a bad time to blow away my local database (it seemed to have gotten corrupted) but this was also a good opportunity to update to 10.0.1 from 9.x!

Hi @jeremylong,
I'm currently using Dependency Check v9.0.9 with a Postgres database and have been encountering 503 errors for the NVD services since last Friday. I noticed that there are several related issues reported.
My doubt is, given my setup (v9.0.9 of Dependency Check with a Postgres database), is upgrading to v10.0.1 the only solution, or are there alternative fixes available?
Thanks!

Version 9 will not be able to parse the current results from the NVD. 10+ is the only currently supported version.

Does the 10+ version also support Oracle databases? Our NVD mirroring is hosted on Oracle DB.

yes

@jeremylong On a fresh build (without local database available), we still get the following warnings:
[INFO] Checking for updates

[INFO] NVD API has 255,810 records in this update

[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=2000 : 3 time

[WARNING] NVD API request failures are occurring; retrying request for the 5 time

[WARNING] NVD API request failures are occurring; retrying request for the 6 time

[WARNING] NVD API request failures are occurring; retrying request for the 5 time

[WARNING] NVD API request failures are occurring; retrying request for the 5 time

[WARNING] NVD API request failures are occurring; retrying request for the 5 time

[WARNING] NVD API request failures are occurring; retrying request for the 7 time

[WARNING] NVD API request failures are occurring; retrying request for the 6 time

[WARNING] NVD API request failures are occurring; retrying request for the 6 time

[WARNING] NVD API request failures are occurring; retrying request for the 6 time

[WARNING] NVD API request failures are occurring; retrying request for the 7 time

[WARNING] NVD API request failures are occurring; retrying request for the 7 time

[WARNING] NVD API request failures are occurring; retrying request for the 7 time

[WARNING] NVD API request failures are occurring; retrying request for the 8 time

[WARNING] NVD API request failures are occurring; retrying request for the 8 time

[WARNING] NVD API request failures are occurring; retrying request for the 8 time

[WARNING] NVD API request failures are occurring; retrying request for the 8 time

[WARNING] NVD API request failures are occurring; retrying request for the 9 time

[WARNING] NVD API request failures are occurring; retrying request for the 9 time

[WARNING] NVD API request failures are occurring; retrying request for the 9 time

[WARNING] NVD API request failures are occurring; retrying request for the 9 time

[WARNING] NVD API request failures are occurring; retrying request for the 10 time

[WARNING] NVD API request failures are occurring; retrying request for the 10 time

[WARNING] NVD API request failures are occurring; retrying request for the 10 time

[WARNING] NVD API request failures are occurring; retrying request for the 10 time

[INFO] Downloaded 10,000/255,810 (4%)

[WARNING] NVD API request failures are occurring; retrying request for the 11 time

[WARNING] NVD API request failures are occurring; retrying request for the 5 time

[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=20000 : 3 time

[WARNING] NVD API request failures are occurring; retrying request for the 5 time

[WARNING] NVD API request failures are occurring; retrying request for the 6 time

[WARNING] NVD API request failures are occurring; retrying request for the 6 time

[WARNING] NVD API request failures are occurring; retrying request for the 5 time

[WARNING] NVD API request failures are occurring; retrying request for the 7 time

[WARNING] NVD API request failures are occurring; retrying request for the 7 time

[WARNING] NVD API request failures are occurring; retrying request for the 6 time

[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=22000 : 3 time

[WARNING] NVD API request failures are occurring; retrying request for the 8 time

[INFO] Downloaded 20,000/255,810 (8%)

[WARNING] NVD API request failures are occurring; retrying request for the 8 time

[WARNING] NVD API request failures are occurring; retrying request for the 7 time

[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=28000 : 3 time

[WARNING] NVD API request failures are occurring; retrying request for the 5 time

[WARNING] NVD API request failures are occurring; retrying request for the 5 time

[INFO] Downloaded 30,000/255,810 (12%)

[WARNING] NVD API request failures are occurring; retrying request for the 6 time

It does seem that it is downloading something? But, also, a lot of warnings. Not sure if we will end up with a complete database. It is taking a very long time, also.

@henrykuijpers after update to 10.0.0 I deleted my DB and the initial download needen some time and I had some WARNINGs. It's normal. Cause NVD API is the problem.

But, it is still going wrong, when using an empty repo (without any DB files locally cached yet):
[ERROR] Error updating the NVD Data
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data
    at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi (NvdApiDataSource.java:389)
    at org.owasp.dependencycheck.data.update.NvdApiDataSource.update (NvdApiDataSource.java:116)
    at org.owasp.dependencycheck.Engine.doUpdates (Engine.java:906)
    at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase (Engine.java:711)
    at org.owasp.dependencycheck.Engine.analyzeDependencies (Engine.java:637)
    at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.runCheck (BaseDependencyCheckMojo.java:1960)
    at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.execute (BaseDependencyCheckMojo.java:1143)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:904)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:281)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:204)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:255)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:201)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:361)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:314)
Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 503
    at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:360)
    at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi (NvdApiDataSource.java:349)
    at org.owasp.dependencycheck.data.update.NvdApiDataSource.update (NvdApiDataSource.java:116)
    at org.owasp.dependencycheck.Engine.doUpdates (Engine.java:906)
    at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase (Engine.java:711)
    at org.owasp.dependencycheck.Engine.analyzeDependencies (Engine.java:637)
    at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.runCheck (BaseDependencyCheckMojo.java:1960)
    at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.execute (BaseDependencyCheckMojo.java:1143)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:904)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:281)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:204)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:255)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:201)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:361)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:314)
[INFO] Updating CISA Known Exploited Vulnerability list: https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json
[INFO] Begin database defrag
[INFO] End database defrag (11566 ms)
[WARNING] Unable to update 1 or more Cached Web DataSource, using local data instead. Results may not include recent vulnerabilities.

NVD returns 503 statuses etc.

Here (in NL) I got it working again on my laptop, using the 10.1 update from ant, I am surprised to see the data folder is still named data\9.0 ?
I want to put the data directory here on the network so all our teamcity agents can share this update rather than each of the agents downloading their own, is that a good idea or not ? No time and resources to setup a central database for this now...

Here (in NL) I got it working again on my laptop, using the 10.1 update from ant, I am surprised to see the data folder is still named data\9.0 ?

Yes it’s expected. The data directory is versioned independently, it’s just a coincidence that they matched in the last version, and there is much more than H2 DBs potentially in there.

I want to put the data directory here on the network so all our teamcity agents can share this update rather than each of the agents downloading their own, is that a good idea or not ? No time and resources to setup a central database for this now...

Yes, you can cache and share the whole data directory, but IMHO might not be a good idea to have multiple agents potentially writing to the same network folder. I’ve not seen documented a locking mechanism that would make that safe across all files in the dir. read the docs in any case: https://jeremylong.github.io/DependencyCheck/data/cacheh2.html

Thanks Chad, there is a lock file appearing there during the update so for now I'll just see what happens on the next update, I suppose I can disable the auto-update on all dependency check builds except for the one in trunk running most frequently...

Version 9 will not be able to parse the current results from the NVD. 10+ is the only currently supported version.

Thanks for the response. Regarding v9, are there any plans to release an update that, while it won't support CVSS4.0, ensures compatibility without breaking?

@juanmanuelromeraferrio you can read some discussion about this at https://github.com/jeremylong/DependencyCheck/pull/6756 but it seems unlikely at this stage. However there shouldn't be any remaining bugs preventing you from using v10 to my knowledge and it's not really a breaking change except for folks with certain DB configurations. It will also make no difference to whether the NVD API responds with timeout related errors to have a new V9.
As to whether this needed to be done with a v10, I think it would have been possible to use a different approach and still deal with Jeremy's concerns (release v10 with a database epoch ratchet that forces re-retrieval of all CVEs since the v4 go-live date, overriding local last updated records), but it's all work and like all OSS needs time and effort, sometimes more than is available for free 😅

<OT>

IMHO might not be a good idea to have multiple agents potentially writing to the same network folder.

there is a lock file appearing there during the update

We've had this setup for years with potentially ~10 agents going against the same data directory simultaneously. Haven't had any issues so far. If an agent sees there's a lock file in place (i.e. another agent running an update currently), it waits until the lock file disappears. It prints a message to that effect to the console. Of course, should the agent running the update die, the lock file won't be removed automatically thereby causing issues for the waiting agents.
</OT>

We've had this setup for years with potentially ~10 agents going against the same data directory simultaneously. Haven't had any issues so far. If an agent sees there's a lock file in place (i.e. another agent running an update currently), it waits until the lock file disappears. It prints a message to that effect to the console. Of course, should the agent running the update die, the lock file won't be removed automatically thereby causing issues for the waiting agents.

Thanks @marcelstoer ! TIL!
Might be a good idea for us to update the documentation to note this as a valid configuration for the multiple-writers option. Maybe I'll have a go at that (and try and fix the formatting at the same time!)

@chadlwilson I can also vouch for the setup that @marcelstoer is having because we are also using a similar config for more than 2 years and likewise, we also get the warning if an update is already in place and another agent tries to update the same data.

@ftiercelin I missed your comment about the data feed option - it is used by many organizations that create a datafeed using the vulnz-cli.

@marcelstoer @ankurga If you'd like to review my suggested update to the docs at https://github.com/jeremylong/DependencyCheck/pull/6804/files?short_path=2fc6754#diff-2fc67544c882e9fcef577dbcac2c1d6285dbb9bc8d890a9637fa35d09f88e72c it may be useful :-)

@chadlwilson Looks good. Thank you !! :)

Right, thanks again Chad, this setup has been running fine now with dozens of teamcity agents running on the same data directory, significantly reducing our load on the NVD backend,

Still same issue after upgrading to 10.0.1.
`
========================== Starting Command Output ===========================
"C:\Program Files\PowerShell\7\pwsh.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'C:\agent_work_temp\e3476919-ec8c-430a-ac54-f3bd7695d183.ps1'"
09:19:52,511 |-INFO in ch.qos.logback.classic.LoggerContext[dependency-check] - Could NOT find resource [logback-test.xml]
09:19:52,511 |-INFO in ch.qos.logback.classic.LoggerContext[dependency-check] - Found resource [logback.xml] at [jar:file:/C:/dependency-check/dependency-check/lib/dependency-check-cli-10.0.1.jar!/logback.xml]
09:19:52,511 |-WARN in ch.qos.logback.classic.LoggerContext[dependency-check] - Resource [logback.xml] occurs multiple times on the classpath.
09:19:52,511 |-WARN in ch.qos.logback.classic.LoggerContext[dependency-check] - Resource [logback.xml] occurs at [jar:file:/C:/dependency-check/dependency-check/lib/dependency-check-cli-10.0.1.jar!/logback.xml]
09:19:52,511 |-WARN in ch.qos.logback.classic.LoggerContext[dependency-check] - Resource [logback.xml] occurs at [jar:file:/C:/dependency-check/dependency-check/lib/dependency-check-cli-9.0.9.jar!/logback.xml]
09:19:52,527 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@4d3167f4 - URL [jar:file:/C:/dependency-check/dependency-check/lib/dependency-check-cli-10.0.1.jar!/logback.xml] is not of type file
09:19:52,667 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
09:19:52,667 |-INFO in ch.qos.logback.classic.joran.action.ContextNameAction - Setting logger context name as [dependency-check]
09:19:52,667 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
09:19:52,683 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [console]
09:19:52,683 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
09:19:52,699 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.commons.jcs] to ERROR
09:19:52,699 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.hc] to ERROR
09:19:52,699 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
09:19:52,699 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [console] to Logger[ROOT]
09:19:52,699 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
09:19:52,699 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@ed9d034 - Registering current configuration as safe fallback point
[INFO] Checking for updates
[WARN] NVD API request failures are occurring; retrying request for the 1 time
[WARN] NVD API request failures are occurring; retrying request for the 1 time
[WARN] NVD API request failures are occurring; retrying request for the 2 time
[WARN] NVD API request failures are occurring; retrying request for the 3 time
[WARN] NVD API request failures are occurring; retrying request for the 1 time
[WARN] NVD API request failures are occurring; retrying request for the 1 time
[WARN] NVD API request failures are occurring; retrying request for the 2 time
[WARN] NVD API request failures are occurring; retrying request for the 3 time
[WARN] NVD API request failures are occurring; retrying request for the 4 time
[ERROR] Error updating the NVD Data
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data
at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:389)
at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:349)
... 7 common frames omitted
[INFO] Skipping Known Exploited Vulnerabilities update check since last check was within 24 hours.
[WARN] Unable to update 1 or more Cached Web DataSource, using local data instead. Results may not include recent vulnerabilities.
`

Workaround in our case: We have private build servers on which we have OWASP DC installed. Each team running their build runs with --noupdate, and every 4 hours we run a separate job to update all OWASP DC databases. In this case at least the builds from the teams keep running, and they never have to wait for updates. In case updating fails, and it has failed before, only we, the system team, are impacted.

can you show that about how do you run a separate job to update all OWASP DC databases?

@AmyChihHi see the documentation here; it sounds like they are using option 3 and caching the data directory.

To add on further, probably with the single node database updater option if --noupdate is involved as described by @richardzaat earlier. To summarise what is documented, you typically either have shared (readonly) storage for the database, or you have all the readers download-unpack or rsync a copy of the data dir from somewhere central prior to running.
The specific methodology for doing so is highly dependent on your environment and what you have available to you.

@AmyChihHi see the documentation here; it sounds like they are using option 3 and caching the data directory.

I tried it but got the following error message:
[INFO] Checking for updates
[ERROR] Error updating the NVD Data; the NVD returned a 403 or 404 error
Please ensure your API Key is valid; see https://github.com/jeremylong/Open-Vulnerability-Project/tree/main/vulnz#api-key-is-used-and-a-403-or-404-error-occurs
If your NVD API Key is valid try increasing the NVD API Delay.
If this is ocurring in a CI environment
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data; the NVD returned a 403 or 404 error
Please ensure your API Key is valid; see https://github.com/jeremylong/Open-Vulnerability-Project/tree/main/vulnz#api-key-is-used-and-a-403-or-404-error-occurs
If your NVD API Key is valid try increasing the NVD API Delay.
If this is ocurring in a CI environment
at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:387)
at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:116)
at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906)
at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:878)
at org.owasp.dependencycheck.App.runUpdateOnly(App.java:427)
at org.owasp.dependencycheck.App.run(App.java:172)
at org.owasp.dependencycheck.App.main(App.java:89)

@AmyChihHi see https://github.com/jeremylong/DependencyCheck/issues/6816 and #6817 - unless you are getting 503s, you have a different problem.

@AmyChihHi in my case the error
... was actually a SSL handshake error with services.nvd.nist.gov
when debugging, the error was printed, but otherwise, I was just getting
[ERROR] Error updating the NVD Data; the NVD returned a 403 or 404 error
I used https://github.com/jeremylong/InstallCert to install the new cert for services.nvd.nist.gov.
In my case I am behind a corporate proxy (Zscaler) which probably explains why I had to add the certs for this new url
@jeremylong : it seems the "Error updating the NVD Data; the NVD returned a 403 or 404 error" error message is not always accurate. It seems to be also triggered by
caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed:sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target '

@ftiercelin it'd probably be helpful if you filed any misleading error messages as a separate issue with full stack trace, as opportunities for improvement are otherwise going to get lost here (just as they did with 9.x when the switch to this domain was first done - the use of this NVD domain is only new for folks who haven't been using 9.x earlier than this change).
Also, we are now conflating 503 errors with 403 errors in one ticket which is even more confusing for folks trying to figure out whether their problem is transient or permanent (without user action).

@chadlwilson I'll also be improving the error reporting around the NVD API Key (see ovc/55678e58084496f39ca6a92a9fc7dc90b5e82231). I've seen at least one CI with an incorrectly set NVD API Key provide an empty string - which can really cause some confusion and problems.

Yeah, I've seen issues before with Gradle system properties and project properties being read incorrectly (or at the wrong part of Gradle lifecycle) so it'll definitely help to have a bit more "it looks like you're doing something wrong..." here.

Good morning all, I have a big issues related to this topic the API 2.0 https://services.nvd.nist.gov/rest/json/cves/2.0 during the last 3/4 days has been returning HTTP code 503 most interrogation from plugin and the process of dependency-check needs up to 1h for finishing....this is unacceptable.
I'm using the latest version of the plugin 11.1.0 with NVD API key.
I have no others particular configuration on it, some one has some suggestion?
Thank you in advance.

Good morning all, I have a big performance issue related to this topic the API 2.0

All I can say: same here ...

Is it possible to use https://docs.vulncheck.com/community/nist-nvd/nvd-2 ?
Instead of https://services.nvd.nist.gov/rest/json/cves/2.0 in order to circumvent this issue?

@dspaeth-breuni I suppose so, there's a configuration item called nvdApiEndpoint, see https://jeremylong.github.io/DependencyCheck/dependency-check-maven/check-mojo.html

@ftiercelin thanks for your reply. The vulncheck API needs "Authorization: Bearer "
the nist api uses the header "apiKey: ", so can I change this too?

The vulncheck API needs "Authorization: Bearer "

If this is not feasible via the client, you would have to set up some outbound reverse proxy (i.e. with nginx) that enriches your request accordingly.

We accept PRs...

Facing the same issues, 503 errors on all requests

We accept PRs...

This is not as simple as I thought. It's not just about shifting from header 'apiKey:{key}' to header 'Authorization: Bearer {key}'
The Vulncheck API isn't working exactly as the NVD one.
I think we need a new type of client in package io.github.jeremylong.openvulnerability.client to handle this new API.
Then we could work on using this new client type in this project.
I have created an enhancement request: https://github.com/jeremylong/Open-Vulnerability-Project/issues/231
cc @dspaeth-breuni

The incident causing most of these issues has been resolved. hopefully, the NVD will not mass update all of the vulnerabilities again.


how to using split method?

Hi,
Is there any way to use matcher.split() but for groups instead match,
Thanks you.

Grouping support in the Matcher class is limited to outer closing (...). You want to use the PerlMatcher to perform grouping. Or what is your question, exactly?

Hi,
can I make matcher.split() work in this case? The code I'm using is as follows:
	reflex::PCRE2Matcher matcher("brown(\\d+)", "How now45 brown cow now brown123 cow now brown378.");
	while (matcher.split() != 0)
	std::cout << "Found " << matcher.text() << std::endl;

my expected result is:
"How now45 brown cow now brown"
" cow now"

You will get:
Found How now45 brown cow now
Found  cow now
Found .

because both brown123 and brown378 are the "split points" that match the pattern. Split returns text between the pattern matches.

hi @genivia-inc genivia,
my omission, so my expected result is:
How now45 brown cow now brown
 cow now brown
I mean there is one more option separated by group in match which is (\d+)

That's not how splitting works. Remember that split returns the text between the pattern matches. The pattern is brown(\\d+) which matches brown123 and brown378. To split on numbers only, use the pattern \\d+. To split on numbers only after the word brown, use perl matching with a so-called lookbehind pattern to also match brown but not consume it as part of the pattern.

Hi @genivia,
please!!! that will make the split more flexible.


[5.7] Add dynamic SELECT clauses for aggregate functions

This PR proposes dynamic SELECT clauses for aggregate functions to reduce the usage of raw expressions:
DB::table('purchases')->select('user_id')->selectCount('*')->groupBy('user_id');
// select "user_id", count(*) from "purchases" group by "user_id"

DB::table('purchases')->select('user_id')->selectSum('amount', 'total')->groupBy('user_id');
// select "user_id", sum("amount") as "total" from "purchases" group by "user_id"

As the second parameter, you can optionally specify a column alias.
This feature is similar to the dynamic WHERE clauses: ->whereMyColumn('value')
The discoverability is not ideal. We could add @method annotations for the most common functions:
/**
 * @method $this selectAvg(string $column, string|null $as = null)
 * @method $this selectCount(string $column, string|null $as = null)
 * @method $this selectMax(string $column, string|null $as = null)
 * @method $this selectMin(string $column, string|null $as = null)
 * @method $this selectSum(string $column, string|null $as = null)
 */
class Builder
{

What is the point of making these dynamic. Why wouldn't you just add or macro in actual methods?

We could handle a dozen aggregate functions with one method.


Create file with list of contributors to this repository

Create file with list of contributors to this repository

Added via https://github.com/RedHatInsights/insights-results-aggregator-utils/pull/173


(scribus) Migrated to AU

I'm updating this with the softwareName tomorrow, I'm more or less about to sign of for today.

@gep13 just added the missing softwareName.
Just waiting for the appveyor build to run, then I'll merge this one.

Damn, didn't notice that the archive became too large.
I'll need to rework it by downloading the files (or perhaps just to download the music backgrounds).
The archive became around 19 MB too large XD


School: Fixed the name of the text file

Please add this school to the registry
https://www.sunnfolk.no

@GameDevTeacher Pull request merged. Thank you!


add hard coded limits

-added usgs limits not in wqx domain in the get ref function (Historical Lower Reporting Limit, Elevated Detection Limit).
-updated ref csv as well.

Looks like we need to update the internal copy of the reference file (or may need to adjust this test since we are no longer using the domain table directly)?
── Failure ('test-GenerateRefTables.R:20:3'): Is GetDetLimitRef up to date? ────
old_latedate == new_latedate is not TRUE

@cristinamullin Whoops! Let me take a look.


Compound type support

Hello @aldanor, currently compound types are not supported (H5T_COMPOUND).
Do you currently have any plans for supporting them?

Yes, I have big plans :) See this branch: https://github.com/aldanor/hdf5-rs/tree/feature/types
There's a little bit left to finish it (+ndarray integration, +proper unicode handling) which I plan to finish before New Year, but probably not much sooner.

Usage examples: kind of like this - https://github.com/aldanor/hdf5-rs/blob/feature/types/src/types/h5def.rs#L135


Compute force vector for torsion problem with polygonal domains

Following the docs, the force vector F must be computed for torsion problem
$$
F_i = \int_{t_{min}}^{t_{max}} \langle \mathbf{p}, \ \mathbf{p}'\rangle \cdot \ln |\mathbf{r}_i| \ dt
$$

$\mathbf{r}_i$ is the distance of the point $\mathbf{p}(t)$ to the source point $\mathbf{s}_i$


This is Sam's edits to Lisa's game

These are the the changes I made to Lisa's game and now I'm sending her a pull request.

Thanks, dude!


cgit: support cross-compilation

asciidoc was in the incorrect location, and CC and AR must be set explictly.
Motivation for this change
Support cross-compilation for more packages.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS (on staging)
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Fits CONTRIBUTING.md.

Building on master is broken due to zstd being broken, but it should work on staging or staging-next (tested on staging). Changes are simple so I don't know if this should target staging anyway or what is best practice.

If you care enough about history we can wait until staging is merged to master. But I'm fine with merging directly to master.

Unfortunately this broke the cgit package, because the cgit.cgi script is no longer installed in $out. (I bisected it with nix-build -A cgit && test -f ./result/cgit/cgit.cgi).
Looking at the build log, it seems the new way of setting CGIT_SCRIPT_PATH does not work:
installing
install flags: SHELL=/nix/store/b45zavallnsvqwjs9wg9xw167jcs0935-bash-4.4-p23/bin/bash prefix=\$\(out\) CGIT_SCRIPT_PATH=\$out/cgit/ CC=cc AR=ar install
    SUBDIR git
install -m 0755 -d ut/cgit/
install -m 0755 cgit ut/cgit//cgit.cgi
install -m 0755 -d ut/cgit/
install -m 0644 cgit.css ut/cgit//cgit.css
install -m 0644 cgit.png ut/cgit//cgit.png
install -m 0644 favicon.ico ut/cgit//favicon.ico
install -m 0644 robots.txt ut/cgit//robots.txt

Uh oh, I thought I checked that that gave a reasonable result in the script. I guess it needs some escaping then?

Maybe give it the same treatment as prefix? ("prefix=$(out)")

That seems to work, yes :+1:

See #131842


failure TypeError: facility.links is not iterable

Hi,
Error Code:
failure TypeError: facility.links is not iterable
at main (file:///C:/Users/Adokshaj/OneDrive%20-%20Sclera/Documents/Folder%20to%20test%20out%20Tandem%20JS%20REST%20API%20Code/assets/read-asset-properties.js:28:33)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
IDE Used:
WebStorm
Version: 2024.2.3
Build: 242.23339.15
26 September 2024
Steps to reproduce:

Replace 'APS_CLIENT_ID', 'APC_CLIENT_SECRET', 'FACILITY_URN' is the code
Run read-asset-properties.js file

Hello, is there any model in your facility, pls? There is missing check if facility.links is not null - I'll fix this.

Thank you @liskaj. The issue is solved.
Adding APS Client ID to our facility using your method solved this issue.


Add Cards In Header Section

Description:
I think so in replacement of slider we can use the cards it enhances the look of website, Let me show you some degines..
Technologies used:
HTML
CSS
JavaScript
Hope you liked it, if you need any change please suggest
ScreenShort:

Demo Vedio:
https://github.com/akshitagupta15june/PetMe/assets/98798977/6271b16a-ca34-4398-8e3e-de7fa462cb56

@Susmita-Dey @akshitagupta15june @muditchoudhary @akshaaatt  Please Suggest me can i raise a PR or need any changes

I am from GSSoC'23.
I would like to be assigned this task.

Hey there! I'm Ashutosh Kumar, a full-stack MERN web developer. With experience in numerous successful MERN stack  projects, I excel in  HTML, CSS,JavaScript,C, C++,jQuery, Node.js, React.js, Bootstrap, Tailwind, Express, MongoDB, API,GraphQL,WebRTC.
You can find me on LinkedIn : https://www.linkedin.com/in/ashutosh-kumar-60b262251/
GitHub : https://github.com/Ashutosh-0506.
Looking forward to contributing to GSSOC 2024!


Embedded programming languages

I need a very interesting answer that torments me, I will say that the documentation is not so well done. It looks like you did it for yourself.
How do I make an embedded programming language, for example, in Rust, C++? That is, something is needed to make an embedded programming language work after the main programming language if it is an interpreter or compiler.

The language server protocol doesn't provide any means right now for embedded languages. We do provided a special guide how to achieve this if you write a language server for VS Code (https://code.visualstudio.com/api/language-extensions/embedded-languages). The main reason is that language embedding depends on how clients manage language support.
Hope that helps.

This problem is not solved, because since it does not describe to me how to do. You just described the details in two words.
For example, I still don't understand how to make my LSP (https://code.visualstudio.com/api/language-extensions/language-server-extension-guide)
For example, how to make variables that the developer himself created and functions, constants and much more displayed. How to display hints after the instructions, for example, what you can write, make documentation for the instructions.
I've been dealing with this issue for a very long time that it's not even a month.
A lot of questions and zero answers.

Examples both for LSP servers and embedded languages can be found here: https://github.com/microsoft/vscode-extension-samples

@dbaeumer I think really that LSP should provide the capability to support embedded language. The HTML language server sample works because it embedds CSS, JavaScript language service inside the language server. This strategy have some limitation:

when your language server is written with another language than TypeScript, you cannot manage this strategy. For instance vscode-xml uses LemMinx, an XML Language server written in Java, we cannot use CSS, JavaScript language service inside the XML Language service.
if you want to manage in HTML Language server another language with <script language="java"> it will very hard to implement this feature since Java Language server is written in Java.

In other words I think LSP should provide the capability to delegate to an another language server the LSP operations. The idea is that the language server should provide an embedded range with a given language. The LSP client could consumes textDocument/embeddedRanges which return a list of {Range, embeddedLangeId} and after that LSP client delegate the LSP operation according the embedded range/languageId.

Unfortunately this is not so easy since the embedded part can usually not be send to the another language server 1:1 since they are almost always fragments which might not be syntactically correct. So some rewriting and position mapping needs to be done and that needs to be implemented by the server.
What we did discuss is to offer a way that a server A can request a LSP operation in server B via the client to make this from an implementation point of view easier. But in principal this is already possible via custom messages.


fix: pin whisperx before breaking change

Per the convo here: https://github.com/ahmetoner/whisper-asr-webservice/pull/125#issuecomment-1861725446
I've modified both Dockerfiles to pin these. Whisperx hasn't been integrated into the upstream pyproject; I don't see anywhere else that it's provided as a dependency.
Also, I did not look into the root cause of the breaking change to the whisperx package, but I'm pasting the error message here:
AttributeError: 'WhisperModel' object has no attribute 'feat_kwargs'
[2024-01-27 13:37:17 +0000] [28] [ERROR] Exception in ASGI application
Traceback (most recent call last):
  File "/app/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 404, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/app/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/app/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 276, in __call__
    await super().__call__(scope, receive, send)
  File "/app/.venv/lib/python3.10/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/app/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/app/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/app/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/app/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/app/.venv/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/app/.venv/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/app/.venv/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/app/.venv/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/app/.venv/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/app/.venv/lib/python3.10/site-packages/fastapi/routing.py", line 237, in app
    raw_response = await run_endpoint_function(
  File "/app/.venv/lib/python3.10/site-packages/fastapi/routing.py", line 165, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
  File "/app/.venv/lib/python3.10/site-packages/starlette/concurrency.py", line 41, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
  File "/app/.venv/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/app/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/app/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/app/app/webservice.py", line 91, in asr
    result = transcribe(
  File "/app/app/mbain_whisperx/core.py", line 41, in transcribe
    result = model.transcribe(audio, **options_dict)
  File "/app/whisperX/whisperx/asr.py", line 194, in transcribe
    language = language or self.detect_language(audio)
  File "/app/whisperX/whisperx/asr.py", line 248, in detect_language
    model_n_mels = self.model.feat_kwargs.get("feature_size")
AttributeError: 'WhisperModel' object has no attribute 'feat_kwargs'

Don't merge, I forgot to remote the git depth parameter, and need to make the changes in the non-gpu docker file as well.


How is a reference supplied in modena v2?

Hi, I'd really like to try out modena on some data of heavily modified RNA. These are of different references. I noticed that (with v2) there is no clear way to specify a reference (in v1 this was supplied in the tombo resquiggling step, but as tombo is deprecated and the conversion into single fast5 files is painful, I'd really like to not use it). How is a reference supplied for v2?

Hi Patrick. With v2 using f5c instead of Tombo as the preprocessing step, would some of the various f5c commands be of help to you? A reference fasta can be supplied with some other f5c subcommands.

Hi Patrick, we haven't really tested Modena with f5c resquiggling. The problem is that f5c uses a different resquiggling algorithm so this might affect results. So I would suggest either using Tombo or alternatively to try out the basecaller based algorithm as described in our paper.. Unfortunately this is still not implemented in the current version of Modena.  If you need more detail about this basecaller-error based algorithm, let me now. We are interested in further improving Modena, so any feedback is great.

Thank you so much for the swift responses!
I agree that this is likely something that would be updated in the f5c resquiggle algorithm (right now it attempts to resquiggle based on the read basecalls). However, for me personally, I figured out how to run remora resquiggling, which looks okay for my purposes. Remora can do both resquiggling against the basecalled sequence, and against an aligned reference. The latter is important for me because the modifications often introduce basecalling errors (even outside the modified side), and given this is RNA data, there is also a significant amount of random basecalling errors. The remora API is not that straightforward imho (especially as it requires soft-clipped alignments with the move table in bam tags), but maybe it is worth to use that for v2?
On a tangent, I personally am wary of basecall error-based algorithms because they rely on an artifact of the ML model, which is going to be sensitive to sequence context and will likely change with updates, especially as ONT tries to make models more robust when encountering modifications. Also, once relying on a specific model version, basecalling things twice carries some issues: On the infrastructure site converting pod5 to (single) fast5 can become non-trivial as file types are updated, and there are issues when combining basecall output from a newer model with mod calls from an older, because read splitting in newer dorado versions alters read_ids for ~30% of reads. That said, they are clearly empirically very performant.
Anyways, thanks for exploring signal-based de novo mod detection, good luck with the paper!

Sorry for the late response. I took a few days off for Christmas.
The main thrust of our paper was to show an algorithmic improvement. So if other algorithms we benchmarked Modena against, used Guppy/Tombo, we had to use those too. Additionally, we got initial results back in the summer of 2022, and back then Guppy and Tombo were very much used. Unfortunately, benchmarking against other algorithms (which were published before or in 2022), retesting all algorithms on many benchmark datasets, writing the paper and doing a major revision (which included many new datasets ad analysis) all took quite a lot of time. So that's the reason why Modena as it was published (and extensively tested) is based on the Guppy/Tombo combination.
The way I see it (I'm expressing only my own opinion to be clear), is that you can either focus to show that your new algorithm is better than other algorithms of the same type (which was our primary goal), or alternatively, switch to a more biology oriented focus, to show that your new algorithm is successful with some "very cool" new biological dataset(s) and very latest technology.
The first approach will almost inevitably necessitate the use of somewhat older (third party) softwares or technology, because you have to benchmark against competitor algorithms, which were developed some time ago. Otherwise, using for example, different resquiggling algorithms than competitor algorithms, defeats the whole purpose of comparing algorithms against each other, since they would have (somewhat) different inputs. So any difference in performance could be either due to one algorithm being better/worse or simply due to different inputs.
Also, I agree 100% with what you said about the basecaller-error based algorithms. Their weakness is that they greatly depend on the the underlying basecaller, which will and do change. But the same is true for resquiggling algorithms too. That is why one of our current focuses is on trying to make Modena independent on such "third party" (resquiggling) algorithms.
Modena v2 is very much a "work in progress" right now. V1 on the other hand is robust, since it was tested on many different datasets.

No worries, I totally understand it!
And I fully agree regarding the resquiggling algorithms - from my own observations, those have a major impact on the accuracy of detection, and can unfortunately also introduce biases in different sequence contexts, types of modifications and at increased modification densities. Personally, I have found tombo resquiggling to perform the best by far, but as mentioned its infrastructural requirements aren't scalable. I am intrigued what you are working on there!
On that note, what is also often overlooked is the choice of aligner: minmap2 in particular often doesn't align sequences with a substantial number of mismatches (which is more common in modified RNA), and I personally also had some trouble getting the HMM in eventalign working for RNA004.
Anyways, I will close this for now, just with a small reiteration that I believe reference-based resquiggling/modification calling is imho important, because modifications may introduce basecalling errors which if not corrected/overwritten before resquiggling would result in resquiggling artifacts and/or underestimation of modifications.
Thank you so much again for this discussion!

I'm actually not personally involved in the "new resquiggling" algorithm, so I can't really tell you much about it at this stage. As for everything else, you can always send me an email to<EMAIL_ADDRESS>(I currently have some technical issues with my work email ivan.vujaklija@fer.hr). The conversation over the mail is more private, so I could go into more details (when I know more about our next steps). And in principle, if we find some common interest at some point, we might also collaborate on some project.


Deprecate unsafe pin projection macros

Use of these macros in futures-rs has been removed, so we can probably go ahead this.
Closes #29
r? @cramertj @Nemo157

Hm, I think I forgot to release this deprecation.

Hm, I think I forgot to release this deprecation.


Correct logging spelling

just quick change on some spelling and grammar i picked up when running earlier

The PR looks great. But it has conflicts with the main branch. Could you please rebase against the latest origin/main?


Better documentation for expected CORS server configuration

When using shaka with CORS, the expected server configuration is not immediately obvious. Could a suggested configuration be added to the README?
For example:
Access-Control-Allow-Origin: https://example.com   # or * for all sites
Access-Control-Allow-Headers: Range                # is IMS or any other header necessary?
Access-Control-Max-Age: 600                        # cache preflight requests for 10 mins
Access-Control-Allow-Credentials: true             # iff using credentials

I'm not confident of the best configuration or I would send a pull request.

I am hesitant to add this to the documentation since this is a common server-side configuration.  CORS requirements are imposed by the browser, not by Shaka, so most servers will have to deal with this whether they use Shaka or not.  Also, different server software will require different configurations to allow CORS.
There is already an entry in the FAQ when dealing with HTTP_ERROR.  We may also want to mention CORS setup in the tutorials.  But since this doesn't apply to Shaka specifically, it doesn't belong in the README.

Thanks @TheModMaker. I appreciate the point. That said, at least the Access-Control-Allow-Headers: Range does depend specifically on the headers sent by shaka. I'm fine with leaving out the documentation for the other headers (as you say, they are not dependent on shaka). However, without diving into the shaka source, how can one be sure of which non-simple headers shaka adds to the requests?
To me, anything that requires reading the source for correct server configuration should be documented. What do you think?

I'll be sure to mention the Range header in the FAQ and in a new section in the tutorials.

The new docs have been cherry-picked to v2.2.x and will be released in v2.2.2.


first attempt to rearrange files

This is my first pass at getting the files in a more pythonic arrangement.
Fixed #61

Tests pass with this new arrangement (there was an issue with relative importing in the test scripts for the previous arrangement). A minor tweak is needed in test_parastell.py to use the proper coil file name (coils.example instead of coils.txt).
In the Examples directory, we'll want to rename the example coils file as before and modify the relevant lines in the example scripts. We'll also want to change how ParaStell is imported in the example scripts as mentioned above (import parastell.parastell instead of import parastell).

Thanks @gonuke, this looks good to me. Do we want to open this for review?


Please add a screen shot

I'd like to add this to awesome-zsh-plugins, but I prefer to add themes with screen shots so that users can see what they look like without having to install them.
Also, please consider renaming it to something less generic than theme.
Thanks!

Cool, update this issue when you do (or just PR it into https://github.com/unixorn/awesome-zsh-plugins/ if you want) and I'll get it added.


Switching between translations

I may have forgot something, but it doesn't seem that it is possible to switch from 1 translation to another using the API.
Am I wrong ?
I think to achieve this, there should be a pickadate.defaults.langs[localeCode] allowing to override some pickadate.defaults when changing a lang
Something a bit like what momentjs is doing with its defineLocale() call
WDYT ?

+1 for this
Would be a great feature, as I am building an app with a language switching button and have no option to change between translations...


Update/mobile site map

#15 Completed

Does this work duplicate what was already done in PR #42 in regards to the sitemap?

Does this work duplicate what was already done in PR #42 in regards to the mobile sitemap?

Yes, good catch, I messed up the commits.


invoke: 1.1.1 -> 1.2.0

Motivation for this change
Update invoke to its latest version.
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[ ] NixOS
[x] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Fits CONTRIBUTING.md.

@GrahamcOfBorg build python27Packages.invoke python36Packages.invoke pipenv

Success on x86_64-darwin (full log)
Attempted: python27Packages.invoke, python36Packages.invoke, pipenv
Partial log (click to expand)
  The scripts pipenv and pipenv-resolver are installed in '/nix/store/3zj29k6bk9xg1x5kjm6v1dxlz3ir5vqx-pipenv-2018.10.9/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pipenv-2018.10.9
/private/tmp/nix-build-pipenv-2018.10.9.drv-0/pipenv-2018.10.9
post-installation fixup
strip is /nix/store/53nysl8bqwxihwzs1pgwka20nf8mbvlp-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/3zj29k6bk9xg1x5kjm6v1dxlz3ir5vqx-pipenv-2018.10.9/lib  /nix/store/3zj29k6bk9xg1x5kjm6v1dxlz3ir5vqx-pipenv-2018.10.9/bin
patching script interpreter paths in /nix/store/3zj29k6bk9xg1x5kjm6v1dxlz3ir5vqx-pipenv-2018.10.9
wrapping `/nix/store/3zj29k6bk9xg1x5kjm6v1dxlz3ir5vqx-pipenv-2018.10.9/bin/pipenv'...
wrapping `/nix/store/3zj29k6bk9xg1x5kjm6v1dxlz3ir5vqx-pipenv-2018.10.9/bin/pipenv-resolver'...

Success on aarch64-linux (full log)
Attempted: python27Packages.invoke, python36Packages.invoke, pipenv
Partial log (click to expand)
Successfully installed pipenv-2018.10.9
/build/pipenv-2018.10.9
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/9xiw9ch9ra5panq2012ysx1km4cp7jdr-pipenv-2018.10.9
strip is /nix/store/p9akxn2sfy4wkhqdqa3li97pc6jaz3r1-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/9xiw9ch9ra5panq2012ysx1km4cp7jdr-pipenv-2018.10.9/lib  /nix/store/9xiw9ch9ra5panq2012ysx1km4cp7jdr-pipenv-2018.10.9/bin
patching script interpreter paths in /nix/store/9xiw9ch9ra5panq2012ysx1km4cp7jdr-pipenv-2018.10.9
checking for references to /build in /nix/store/9xiw9ch9ra5panq2012ysx1km4cp7jdr-pipenv-2018.10.9...
wrapping `/nix/store/9xiw9ch9ra5panq2012ysx1km4cp7jdr-pipenv-2018.10.9/bin/pipenv-resolver'...
wrapping `/nix/store/9xiw9ch9ra5panq2012ysx1km4cp7jdr-pipenv-2018.10.9/bin/pipenv'...

Success on x86_64-linux (full log)
Attempted: python27Packages.invoke, python36Packages.invoke, pipenv
Partial log (click to expand)
Successfully installed pipenv-2018.10.9
/build/pipenv-2018.10.9
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/kc88wvh084ky0wbc2zzj4xj0ac8s5i74-pipenv-2018.10.9
strip is /nix/store/vcc4svb8gy29g4pam2zja6llkbcwsyiq-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/kc88wvh084ky0wbc2zzj4xj0ac8s5i74-pipenv-2018.10.9/lib  /nix/store/kc88wvh084ky0wbc2zzj4xj0ac8s5i74-pipenv-2018.10.9/bin
patching script interpreter paths in /nix/store/kc88wvh084ky0wbc2zzj4xj0ac8s5i74-pipenv-2018.10.9
checking for references to /build in /nix/store/kc88wvh084ky0wbc2zzj4xj0ac8s5i74-pipenv-2018.10.9...
wrapping `/nix/store/kc88wvh084ky0wbc2zzj4xj0ac8s5i74-pipenv-2018.10.9/bin/pipenv-resolver'...
wrapping `/nix/store/kc88wvh084ky0wbc2zzj4xj0ac8s5i74-pipenv-2018.10.9/bin/pipenv'...


Lodash security vulnerability

A security vulnerability was found in lodash.
I saw it's a dependency in a few of the botium projects. Would be great if you could release updates.

Will take care at the next build. Noticed it as well in our continuous builds.

Some more vulnerabilities. Are you gonna release updates to fix them?

of course.


Add component id for Micronaut Http plugin

[x] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #.
[x] Update the CHANGES log.

Related issue : https://github.com/apache/skywalking/issues/9429
Need attach UI  submodule information. Right?

Finally , I choose a ID : 131 .

What are those services' question marks?

Finally , I choose a ID : 131

Then, please remember to update Java agent plugin codes by using one and the new component ID only.

What are those services with question marks?

The one is invoke self by localhost, other one is call a common domain


device parameters

Hello.
Congratulations on your work.
when installing the plugin, I get the following message:
AC@@Target Heating Cooling State] Characteristic not in required or optional characteristic section for service HeaterCooler. Adding anyway.
Observing the plugin instructions, I found in item 4 the following information:
4.Add device parameters
Would it be possible to show examples of this configuration? I don't know where and how I add the parameters of my equipment.
After the last update my equipment doesn't work. it turns on but does not start the engine.

Can you please share detailed log? Not able to understand the problem.


Fix Coverity-1604641

Coverity flagged an overflow warning here that can occur if BIO_write returns an error.
The overflow itself is a bit of a non-issue, but if BIO_write returns < 0, then the return from i2a_ASN1_OBJECT will be some odd value representing whatever the offset from the error code to the number of bytes the dump may or may not have written (or some larger negative error code if both fail.
So lets fix it.  Only do the dump if the BIO_write call returned 0 or greaater.

This pull request is ready to merge

merged, thank you!


Rewrite auto-bootstrap.sh to pipeline actions?

I like the idea of the auto-bootstrap.sh script from the example repo (https://github.com/aws-samples/aws-bootstrap-kit-examples/blob/main/source/1-SDLC-organization/lib/auto-bootstrap.sh)
But I am wondering if the functionality can be brought into this project and perhaps would generate pipeline actions programmatically in TS?
I think a lot of functionality there is redundant, like getting org accounts and so on. All of that info is already known via nestedOU.

Templates can be rendered from CDK:
   // in TemplateStack
    new CfnInclude(this, 'Template', {
      templateFile: 'node_modules/aws-cdk/lib/api/bootstrap/bootstrap-template.yaml',
      parameters: {
        TrustedAccounts: [...],
        TrustedAccountsForLookup: [...],
        CloudFormationExecutionPolicies: ['arn:aws:iam::aws:policy/AdministratorAccess'],
      },
    })

  // later
  new TemplateStack(app, 'TemplateStack')

  // CloudFormation template of the bootstrap stack, with all of the params filled in
  const { template } = app.synth().stacks[0]

Which now allows us to just use aws CLI:
aws cloudformation \
  --profile YOUR_PROFILE \
  --region YOUR_REGION \
    create-stack \
      --stack-name CDKToolkit \
      --capabilities CAPABILITY_NAMED_IAM \
      --template-url rendered-bootstrap-template.json

But I am also wondering if it is possible to have the above TemplateStack as part of the aws-bootstrap-kit code and then extract it from the cdk.out dir and manually call aws cloudformation ... on it.

Another option is to release an NPM package that can then be used as:
npx --yes @aws-bootstrap-kit/org-bootstrap --output=bootstrap.sh
sh bootstrap.sh

or
npx --yes --quiet @aws-bootstrap-kit/org-bootstrap | sh

Which under the hood uses @aws-sdk/* packages to figure out what needs to be bootstrapped and how.
Potentially even just using CDK itself to render a CF template.
This approach would be easier, reusable, and better testable.

Indeed we could do it through npx executable to replace the script. Happy to take a PR for that (keeping in mind that it needs to work in codebuild).

What is missing though is that CDK Bootstrap template is not in TS code yet. It's YAML. And the one that is in the demo repo is also stale (still on v8).
It'd be great to define the official CDK Bootstrap template as code (from which YAML/JSON can be rendered). See issue https://github.com/aws/aws-cdk/issues/16685
This would allow extension of the Bootstrap template.
In the meantime, it can be imported thru CfnInclude as noted above.


[WIP] Feature/install module from git

So far only working for modules that don't require any further configuration

Would propose a new variable for external modules with configuration.
icingaweb2_custom_modules:
  grafana:
    enabled: true
    source: git,url,tag
    files:
      - config
    config:
      graphite_host: url
      graphite_port: foobar

hi, can you rebase to the current state and then we will together review the PR?


Fix errors, make Bash compatible, and improve output format

Clean up shellcheck warnings and add shebang
Add bolding and enforce line width limit
Move vars that can change to the front
Tell user that script is not frozen
Make the arg checking bash compatible
Use default SAT unit spelling
Show bullet points in output
Show --help if an erroneous parameter is provided
Fix calculation error in decision logic

For context, I tried executing the script in Bash but that failed ... are you using some other shell?
Then I tried to make it Bash compatible, but got a bit extra ambitious and improved a few other things as well.

By the way, I identified many of the improvements via ShellCheck ... would definitely recommend that, I have a vim plugin that automatically highlights all improvement ideas and errors.
https://github.com/koalaman/shellcheck

Just realized that the script always rounded down, so I fixed that ... now it shows 1 decimal and rounds correctly.

Nice!, thanks for putting in the effort to improve the code and squash some bugs in the process.
I am running Raspberry Pi OS Lite. Didn't think the shell would differ that much compared to other linux distributions. Might also explain why bc was not installed. I tried to avoid dependencies users have to install because we better not bloat the system on a node.
If you think bc is a very common calculator tool on linux, I am going ahead and merge it.

Thank you. Do you call it via bash overview.sh?
There was one bug in particular that should cause all recent versions of bash to fail:
https://github.com/Czino/lnstats/blob/0a764752880ba741abba55485dacde2e0eae8a34/overview.sh#L87-L89
There's a missing space before ]. Here's my version (I also have Raspberry Pi):
$ bash --version
GNU bash, version 5.0.3(1)-release (arm-unknown-linux-gnueabihf)

Besides, another bash non-compliance I found was this:
https://github.com/Czino/lnstats/blob/0a764752880ba741abba55485dacde2e0eae8a34/overview.sh#L105-L109
if [ false ]; then echo HELLO; fi prints "HELLO" in bash, because the statement would always return true. It doesn't evaluate the false statement otherwise, it just sees that there's a non-empty string (=true). You would have to remove the square brackets (which I did).

FYI, I have shellcheck integrated into my vim, so the first thing I saw after opening the file was this warning:

After I fixed that, it showed me ~15 other errors, which were mostly caused by the fact that the file didn't have a Shebang (the first line that I added) ... thus it was not even clear if it's a Bash script or something lese:

After I added the Shebang, it knew that this was a Bash script, so it showed me the real errors etc. etc. etc. It's really useful.

If you think bc is a very common calculator tool on linux, I am going ahead and merge it.

Missed the question .... yes, I was surprised that bc was not part of standard Raspbian ... it's super common, I thought it was standard.

Awesome, will most certainly start using the shellcheck. I recently started learning shell when getting involved into lightning, so naturally I am still learning all these things.
Been running the scripts both ways: . overview.sh and bash overview.sh
Have the same version tho:
bash --version
GNU bash, version 5.0.3(1)-release (arm-unknown-linux-gnueabihf)

Merged your code now 🎉

Cool man. It's really useful to me so thank you for researching all those lncli commands for me :)


Make BiggerDecimal extend Serializable

This PR makes BiggerDecimal extend Serializable. This should ensure that all ADT elements are fully Serializable, which is necessary for Spark friendliness (among other things).

Thanks @jeremyrsmith! Mind adding a test for BiggerDecimal here?

@jeremyrsmith I'm going to go ahead and merge this—we can add the test later.


Add a wiki: Dragon Age Origins Wiki

An anonymous user has submitted a new wiki via our submission form.
Link to origin wiki: https://dragonage.fandom.com/wiki/Dragon_Age_Wiki
Link to destination (independent) wiki: https://dragonage.neoseeker.com/wiki/Main_Page

There is a huge content gap between the origin and destination wiki (13,646 vs. 199 content pages). I don't see the benefit of a redirect here.


GeoJSON should be syntax highlighted as JSON instead of plain text out of the box without manual configuration

Does this issue occur when all extensions are disabled?: Yes

VS Code Version: 1.58.2
OS Version: Lubuntu 20.04

Steps to Reproduce:

Create example.geojson with following contents:

{
  "type": "FeatureCollection",
  "features": []
}

Open it
Note lack of highlighting

Not a duplicate of  #89734 #82749 as it was about ability to config it (what is nice that it exists!). And the whole point of VS Code (at least to me) is that it works well out of the box, if I wanted configuring I would use Emacs/Vim

Seems fixable by editing https://github.com/microsoft/vscode/blob/747d667946a59a71cddd8957677b92b7dc6ef3e3/extensions/json/package.json#L22


inPageLinks not working

Hello,
I am testing this library for a production feature. I'm needing the inPageLinks option to work as it's mentioned here.
However whenever I set it true I still receive this in the console:
[iFrameSizer][iFrameID] In page linking not enabled
Any ideas on why it's still false even when I set it with $('iframe').iFrameResize({inPageLinks:true,log:true});
Thank you for your time!

You can see it work in the demo here:
http://davidjbradshaw.com/iframe-resizer/example/
I expect something else in your page is cause the issue, but you have not given me any info to go on.
If you create a simple test case and it is still an issue than please add a link to it here.

Hey @davidjbradshaw Thanks for responding so quickly.
I've setup a test case located at https://hungry-hippo.netlify.com/
As you can see the option is set to true although the console states otherwise.

You need to use a more upto date version of the script in the parent. Currently you are using this, which predates the inPageLinks options.
iFrame Resizer (iframeSizer.min.js ) - v2.8.3 - 2015-01-29


Have myenv default to local when building

Description
Currently, myenv defaults to cc. This is a very specific case and most people are building via local developments. When building this way, one often types commands by hand.
In contrast, when working with a build environment when scripting, e.g. Travis, or CC, you're writing scripts where it is straight forward to add the -Dmyenv=local to the script parameters.
Suggested fix
Change (_template.xml):
<property name="myenv" value="cc" />

to:
<property name="myenv" value="local" />

Known issues
This may require changes to the CC scripts. I do not have access to them, thus I have no idea about the scope of this change.
Opening this as a discussion.

I wrote a script to automatically find the environment:
<property environment="env"/>

  <!-- Try to detect the environment: -->
  <condition property="auto-myenv" value="travis">
    <and>
      <isset property="env.CI" />
      <isset property="env.TRAVIS" />
    </and>
  </condition>

  <condition property="auto-myenv" value="appveyor">
    <and>
      <isset property="env.CI" />
      <isset property="env.APPVEYOR" />
    </and>
  </condition>

  <local name="computer.hostname" />
  <exec executable="hostname" outputproperty="computer.hostname"/>

  <condition property="auto-myenv" value="cc">
    <and> <!-- Check for CC on cruise -->
      <os family="unix" />
      <equals arg1="${computer.hostname}" arg2="cruise.eecs.uottawa.ca" />
    </and>
  </condition>

  <condition property="auto-myenv" value="local">
    <os family="unix" />
  </condition>

  <condition property="auto-myenv" value="wlocal">
    <os family="windows" />
  </condition>

  <echo message="auto-myenv=${auto-myenv}" />

However, I do not want to consider swapping out the setting of myenv with it until I can get a confirmation of a CC variable. @TimLethbridge what is the output of hostname on cruise control?
Here is a checklist of targets:

[x] Windows -> wlocal
[x] Unix (osx and Linux) -> local
[x] Travis -> travis
[x] Appveyor -> appveyor
[ ] CC -> cc

on cc, the output of hostname is
cruise

Thanks, @TimLethbridge!


[Chore] license

Hi :wave:
Would you mind to add a license to the repo? It's problematic not to have license for packaging the tool in Linux distros.
Thanks
See

https://choosealicense.com/
https://tldrlegal.com/

Done !!

Thank you.


Fix for [yourporn]

Please follow the guide below

You will be asked some questions, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your pull request (like that [x])
Use Preview tab to see how your pull request will actually look like

Before submitting a pull request make sure you have:

[x] At least skimmed through adding new extractor tutorial and youtube-dl coding conventions sections
[x] Searched the bugtracker for similar pull requests
[x] Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

[ ] I am the original author of this code and I am willing to release it under Unlicense
[x] I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

[x] Bug fix
[ ] Improvement
[ ] New extractor
[ ] New feature

sxyprn seems to be using "cdn6" instead of "cdn5" now.

I don't see any point in this since they will change it right after. Correct fix would be to use phantomjs.


🎁: Provide AKER with Access to the JC Fusion Portal SharePoint - Link SharePoint Support Page to New Handover App

Expected outcome

Please can the AKER email group for JC Stord have access to the JC Fusion Portal SharePoint as visitors / view only: https://statoilsrm.sharepoint.com/sites/Portal
Edward Dutton is sourcing an Aker shared Email address - unless one is already known?
Please can the new Handover app have the Equinor SharePoint linked to the "App Info" button Please?

Business Value / Developer Experience

Allows AKER access the acronyms
Acceptance Criteria

[ ] persons with an @akersolutions.com and are on the JC project can access the portal
[ ] the New Handover app in the JC portal has a link to the Handover Support Page

@atlewee is the owner of this sharepoint, you need to talk to him for granting access.
The link to the handover support page button is going out in production any minute


Issue upgrading com.netflix.dgs.codegen beyond 5.1.2

Hello.
com.netflix.dgs.codegen works well in 5.1.2
When I try to upgrade to 5.1.3 and then run generateJava I get:
FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':apps:financial-analytics-kimera:generateJava'.
> 'java.lang.String kotlin.text.CharsKt.titlecase(char, java.util.Locale)'

When I go for a more recent version like 5.1.17 and then run build + generateJava I get:
> Invalid Syntax : There are more tokens in the query that have not been consumed offending token 'Node' at line 267 column 85

Not sure why this is happening, tried a bunch of versions after 5.1.2 and can't update to any of them.
Any ideas why and what I can do?
Thanks

Closing issue. There was a problem with my query.


Stop media streaming block not working

I have a problem while trying to stop the media streaming because it seams that the streaming never ends until the call is finished.
To put in context, I am developing a call center answered by a bot (using connect, lex, lambda, S3, kinesis, dynamo and transcribe) and I have different interactions with the user. For example, I start asking the user his name, then his phone number, then his address, etc.. This audios are stored in S3 using the lambda function for this project, that transcribes the content, stores the transcription in Dynamo and stores a .wav in S3.
For all those interactions, I start streaming before the user talks and I end it after he finish. Then I start again the streaming, ask the user whatever, wait for his response and end the streaming again.
The problem is that in this example, the first audio never stops till I finish the call and, as it is still running, contains as well the answer for the second, third and all successive questions, making a huge file that only ends when the call is finished. The second audio contains the second answer as well as the third, fourth, etc.. The third audio will contain the third response, fourth, etc...
It seems that the problem is that the "Stop media streaming" block is not working and the streaming never ends.

Same problem here. The "Stop media streaming" block do not trigger the eof, so the MKVParser wait up to 30s to this event to end the process. If the call ends before, the eof arrive and the parsers end earlier.

@vgonisanz I opened a support case in AWS like a couple of years ago and they answered me that there was an issue with the stop media streaming function and that they would solve it as soon as they could. I am not aware whether they have solve it or not, or even if they have started... I suppouse they haven't solve it because you are facing the same problem, but I've never received again any more messages from them.
Sorry for not being able to help you more.

StopStreaming is an async function, and takes some time from the moment it has been invoked until the streaming stops. You could do a simple test by:

StartStreaming
Provide some speech input
StopStreaming, but continue providing speech input for another minute
Disconnect the call
Playback the file
You would hear the speech input, but it should be shorter than the actual duration of the call
That would prove that Streaming stopped before the end of the call

In case you need a lot of 'short' streaming segments, the better approach is to StartStreaming once, and then only store Timestamps when you'd Start and Stop (using either Contact Attributes or a DynamoDB table). At the end of the call, you would implement a post-processing function, that would read from KVS, based on Timestamps you stored in Contact Attributes / DynamoDB.
When you want to configure ‘Live media streaming’ on Amazon Connect, you’d actually find that option under ‘Data storage’ section, and you can set ‘Data retention period’.
This means that Amazon Connect streams audio to KVS, but KVS keeps (‘stores’) the audio for the time specified in ‘Data retention period’.
For example, if your retention period was set to 24 hours, KVS would keep the streamed audio for 24 hours.
You can read KVS in real-time, if that was needed, for instance, if you need to transcribe the call in real-time, provide the customer sentiment in real-time etc.
But, reading KVS in real-time is not mandatory. You can read it any time, over and over again, as long as you read it within the ‘Data retention period’.
For instance, if I wanted to record customer’s interaction with ‘IVR’ (Contact Flow, Lex), I could place Start Streaming block in the Contact Flow, let customer interact with Contact Flow/Lex, and at some point Stop Streaming, and then invoke a Lambda function passing Start Timestamp and Stop Timestamp, to read the stream.
Even without invoking Stop Streaming, I could wait for CTR to be produced, and only then invoke a Lambda function to go and read from the KVS stream.

But then additionally get SpeakNameStart and SpeakNameStop markers (timestamps) from the attributes, and invoke your Lambda function again to read KVS between these 2 markers.
As a result, you’d have 2 wav files, one with the entire recording, and the other one with the audio only between SpeakNameStart and SpeakNameStop markers (timestamps).
When reading from KVS, you’d first get fragments between the two timestamps, and then read the media from all the fragments. For that purpose, you'd use AmazonKinesisVideoArchivedMedia and:
...
final ListFragmentsResult fragmentResult = amazonKinesisVideoArchivedFragment.listFragments(
                new ListFragmentsRequest()
                        .withFragmentSelector(new FragmentSelector().withFragmentSelectorType(FragmentSelectorType.SERVER_TIMESTAMP).withTimestampRange(new TimestampRange().withStartTimestamp(new Date(Long.parseLong(this.StartTimestamp))).withEndTimestamp(new Date(Long.parseLong(this.StopTimestamp)))))
                        .withStreamName(streamName)
        );
...

I hope this helps, on a high-level, but please let us know if you need any additional details.
Thanks,
Milos

Closing due to no response, please feel free to re-open should you need any additional help.


Which version of Menhir is required for experimental-refactor branch?

Is there some way that this should be specified as a dependency in the opam file? I tried a few different versions of the OPAM Menhir library and neither of them would compile successfully with that branch.

Some specific errors I'm trying to resolve:
make complains about an unbound module: MenhirLib.IncrementalEngine.INSPECT_AND_DEBUG.
I'm having trouble finding the version of Menhir that has this signature.

Dependencies inlined in experimental branch.


Proposal: add selectionHeightStyle and selectionWidthStyle to TextSelectionThemeData

Use case
In order to get consistent text selection behavior with regards to the heights and widths of the selection boxes, it seems like one has to override the selectionHeightStyle and selectionWidthStyle properties of TextField and SelectableText widgets manually for every such widget. TextFormField is actually missing these properties, which I guess is a bug.
Proposal
I suggest to add these properties to the TextSelectionThemeData class (or a seperate ThemeData class if this one is unsuitable for some reason), and to have TextField, SelectableText and TextFormField inherit the values from such a theme.

CC @chunhtai

This sounds good to me, but ideally this should work nicely with global selection.  @chunhtai how is the selection height and width set for global selection?
Other than global selection it sounds like the approach would be:

Deprecate selectionHeightStyle and selectionWidthStyle in TextField, SelectableText, and TextFormField.  Keep them in CupertinoTextField and EditableText.
Add selectionHeightStyle and selectionWidthStyle to TextSelectionThemeData.
In TextField and SelectableText, pass the new values from the theme into the selectionHeightStyle and selectionWidthStyle parameters of EditableText.

Is this a material specific thing? if not, it should also be added to the DefaultSelectionStyle so that RichText can also use it


feat: Add Lighthouse PeerDAS Dashboard

Grafana dashboard for monitoring Lighthouse PeerDAS metrics.

This is awesome, thanks @KatyaRyazantseva ❤


24h Volume

As shown in the screenshot below, in the All Markets screen there's a filter that sorts by 24h volume, that is not currently available.

In order to offer this data, we need first to update the TDEX trade protocol to expose a Volumes RPC
https://github.com/tdex-network/tdex-specs/issues/43
and then implement in the tdex-daemon RPCs
This will take some time, will suggest to not show the 24h volume feature as for now.


Sacred, SLURM, MPI and fork(): a tale of nested bugs

I've recently started using sacred together with a library that relies on mpi to parallelise, on a slum cluster at my institution.
That library uses mpi4py.
Since I can't run MongoDB on the cluster, and so I can't use the nice IDSIA package, I resorted to run my experiments with mpirun, and FileStorageObserver, taking care of only logging data from the master node. However, all nodes setup and execute the experiment setup, which is somewhat redundant but should in theory work.
However, it seems that during experiment setup, there are calls to cpuinfo, which essentially spawn a subprocess (through os.popen(cpuinfo_cmd)), which causes a fork. However some CRAY MPI libraries do not properly support fork, and the output is the following:
HOME: /mnt/home/fvicentini
ROOT_DIR: /mnt/home/fvicentini/1-Carleo/FFNN
DATA_STORE_DIR: /mnt/home/fvicentini/1-Carleo/FFNN/datastore_1
DATA_TREE_DIR: /mnt/home/fvicentini/1-Carleo/FFNN/data_tree_1
SLURM_JOB_ID: 602839
--------------------------------------------------------------------------
A process has executed an operation involving a call to the
"fork()" system call to create a child process.  Open MPI is currently
operating in a condition that could result in memory corruption or
other system errors; your job may hang, crash, or produce silent
data corruption.  The use of fork() (or system() or other calls that
create child processes) is strongly discouraged.

The process that invoked fork was:

  Local host:          [[2983,1],2] (PID 241632)

If you are *absolutely sure* that your application will successfully
and correctly survive a call to fork(), you may disable this warning
by setting the mpi_warn_on_fork MCA parameter to 0.
--------------------------------------------------------------------------
Traceback (most recent calls WITHOUT Sacred internals):
  File "/cm/shared/sw/pkg/devel/python3/3.7.3/lib/python3.7/subprocess.py", line 395, in check_output
    **kwargs).stdout
  File "/cm/shared/sw/pkg/devel/python3/3.7.3/lib/python3.7/subprocess.py", line 472, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/cm/shared/sw/pkg/devel/python3/3.7.3/lib/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/cm/shared/sw/pkg/devel/python3/3.7.3/lib/python3.7/subprocess.py", line 1522, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 14] Bad address: 'cat'

During handling of the above exception, another exception occurred:

Traceback (most recent calls WITHOUT Sacred internals):
  File "/mnt/home/fvicentini/.direnv/1_Carleo_FFNN/lib/python3.7/site-packages/cpuinfo/cpuinfo.py", line 2193, in get_cpu_info
    output = get_cpu_info_json()
  File "/mnt/home/fvicentini/.direnv/1_Carleo_FFNN/lib/python3.7/site-packages/cpuinfo/cpuinfo.py", line 2174, in get_cpu_info_json
    p1 = Popen(command, stdout=PIPE, stderr=PIPE, stdin=PIPE)
  File "/cm/shared/sw/pkg/devel/python3/3.7.3/lib/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/cm/shared/sw/pkg/devel/python3/3.7.3/lib/python3.7/subprocess.py", line 1522, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 14] Bad address: '/mnt/home/fvicentini/.direnv/1_Carleo_FFNN/bin/python'

This issue is not easy to reproduce and for some reason does not always show up. I have the impression it mainly shows up when different mpi processes are run on different machines, but that is not always the case.
Monkey patching sacred to avoid logging cpuinfo informations solves the issue.
It would be nice if Sacred allowed an option to prevent him from calling cpuinfo.

@PhilipVinc which Cray machine were you using at the time?

However some CRAY MPI libraries do not properly support fork, ...

Curious if there is documentation of this limitation somewhere you could point me to?