ld: library not found for -liomp5 when installing

I try to install Corrfunc on macos Big sur.
With all dependencies installed, after git clone and make install, I get the following error:
MacBook-Pro:Corrfunc $ make install
mkdir -p lib
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C theory install
Enabling OpenMP with clang.

-------COMPILE SETTINGS------------
	 MAKE		 = ["/Applications/Xcode.app/Contents/Developer/usr/bin/make"]
	 CC		 = ["clang-omp"]
	 OPT		 = ["-DPERIODIC -DENABLE_MIN_SEP_OPT  -DCOPY_PARTICLES  -DUSE_OMP"]
	 CFLAGS		 = [" -DVERSION=\"2.4.0\" -DUSE_UNICODE -std=c99 -m64 -g -Wsign-compare -Wall -Wextra -Wshadow -Wunused -fPIC -D_POSIX_SOURCE=200809L -D_GNU_SOURCE -D_DARWIN_C_SOURCE -O3  -fopenmp -funroll-loops -march=native -fno-strict-aliasing -Wformat=2  -Wpacked  -Wnested-externs -Wpointer-arith  -Wredundant-decls  -Wfloat-equal -Wcast-qual -Wcast-align -Wmissing-declarations -Wmissing-prototypes  -Wnested-externs -Wstrict-prototypes   -Wno-unused-local-typedefs "]
	 CLINK		 = [" -liomp5 -lm"]
	 PYTHON		 = ["python"]
	 GSL_CFLAGS	 = ["-I/usr/local/Cellar/gsl/2.7.1/include"]
	 GSL_LINK	 = ["-L/usr/local/Cellar/gsl/2.7.1/lib -lgsl -lgslcblas -Xlinker -rpath -Xlinker /usr/local/Cellar/gsl/2.7.1/lib"]
	 PYTHON_CFLAGS	 = ["-isystem/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -isystem /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/core/include/numpy/"]
-------END OF COMPILE SETTINGS------------

clang-omp  -DVERSION=\"2.4.0\" -DUSE_UNICODE -std=c99 -m64 -g -Wsign-compare -Wall -Wextra -Wshadow -Wunused -fPIC -D_POSIX_SOURCE=200809L -D_GNU_SOURCE -D_DARWIN_C_SOURCE -O3  -fopenmp -funroll-loops -march=native -fno-strict-aliasing -Wformat=2  -Wpacked  -Wnested-externs -Wpointer-arith  -Wredundant-decls  -Wfloat-equal -Wcast-qual -Wcast-align -Wmissing-declarations -Wmissing-prototypes  -Wnested-externs -Wstrict-prototypes   -Wno-unused-local-typedefs  logbins.c  -liomp5 -lm -o logbins
ld: library not found for -liomp5
clang-12: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [logbins] Error 1
make: *** [install] Error 2

I guess the problem is the library should be libomp5, instead of liomp5?
But I couldn't find where to change it so couldn't confirm this.
Any comment would be very helpful.

@manodeep I'll leave this to you as I'm pretty useless at Mac!
@ziarriall I've lightly edited your comment for formatting

Ohh crap - totally missed this one!
@ziarriall Did you manage to find a work-around?

I have not encountered clang-omp in quite a few years now - but these are the two relevant lines that would need to be commented out/changed

Sorry for the late reply. I changed the line 267 to -Lmypathtolib -lomp, now it works. Thanks a lot.

Looks like this was resolved


How can i load  antMatcher's  urlPattern+httpMethod  from mysql?

I want dynamic update  user->role->permissiones
user->R_ADMIN-> GET /books/** ,hi,you are in get books permissiones.

This is not really a question for this sample. Maybe ask in Stack Overflow in the spring-security tag?


Fix checksum aggregate for DECIMAL input

Resolves #6642
CC: @mbasmanova @majetideepak

HI @majetideepak deepak Thank you for your reply.
The implementation of Presto LongDecimal hash can be found in LongDecimalType and UnscaledDecimal128Arithmetic

Hi @majetideepak deepak, I have modified the code according to the comments. Could you please help to review agai?Thanks!

@majetideepak I'll add it after this merge, Thank you.

HI, @mbasmanova Could also help me review this? thank you.

@mbasmanova https://github.com/facebookincubator/velox/issues/6715 This should not be caused by processing NULL, but should be caused by data out of bounds when processing timestamp.
presto:tpch_velox> desc checksum_test;
 Column |                         Type                         | Extra | Comment
--------+------------------------------------------------------+-------+---------
 id     | row("id" integer, "name" varchar(10), "t" timestamp) |       |
(1 row)

presto:tpch_velox> select id from checksum_test;
  id
------
 NULL
(1 row)

Query 20230926_133223_00008_8atwv, FINISHED, 1 node
Splits: 2 total, 2 done (100.00%)
0:03 [1 rows, 381B] [0 rows/s, 131B/s]

presto:tpch_velox> select to_base64(checksum(id)) from checksum_test;
    _col0
--------------
 h8rrhbF5N54=
(1 row)

Query 20230926_133232_00009_8atwv, FINISHED, 1 node
Splits: 2 total, 2 done (100.00%)
0:04 [1 rows, 381B] [0 rows/s, 105B/s]

It is most likely caused by Timestamp::toMillis processing data out of bounds.

Hi @mbasmanova Do you have any more comments?

Hi @kgpai could you help merge this PR? Thank you.

Hi @kgpai could you help merge this PR? Thank you.

hi @mbasmanohi @kgpai can one of you please help with merging this change? Thanks.

Hi @mbasmanova @kgpai can one of you please help with merging this change? Thanks.

There is overflow in shift:
[2023-10-24T02:26:42.245-07:00] buck-out/v2/gen/fbcode/da4de3c780a17bfa/velox/type/__velox_type__/buck-headers/velox/type/DecimalUtil.h:82:67: error: signed shift result (0x8000000000000000) sets the sign bit of the shift expression's type ('int64_t' (aka 'long')) and becomes negative [-Werror,-Wshift-sign-overflow]
[CONTEXT] [2023-10-24T02:26:42.245-07:00]   static constexpr int64_t kInt64Mask = ~(static_cast<int64_t>(1) << 63);
[CONTEXT] [2023-10-24T02:26:42.245-07:00]                                           ~~~~~~~~~~~~~~~~~~~~~~~ ^  ~~

@Yuhta Fixed, Thank you.
By the way, was the -Wshift-sign-overflow check added recently? The code above was actually moved from here to DecimalUtil.h


why use pdm-backend over setuptools?

Honest curiosity question: why use pdm-backend rather than setuptools? What are the benefit?
If I get an answer, I will make a PR to include it in the doc because I think that's a question many people have.

I built this backend initially to replace the many mysterious configurations of setuptools, think of data_files, package_data, include_package_data and MANIFEST.in, few people can tell the differences and know how to use them. And what's more, it used to be the only companion backend of PDM(it's no longer the case today).
pdm-backend provides sane defaults to support common package layouts, so that users don't need to configure anything or only need a few to make the whole thing work.


⚠ Spryr has degraded performance

In 0ebbcc0, Spryr (https://spryr.com) experienced degraded performance:

HTTP code: 200
Response time: 7737 ms

Resolved: Spryr performance has improved in f93fd87 after 4 minutes.


Improve sections (part 2)

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (a9a68ae) to head (3f48caa).

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #415   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           94        94
  Lines         2393      2418   +25
  Branches       211       211
=========================================
+ Hits          2393      2418   +25

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Custom items in TUP

It would be nice to have user-selected items in the Torn User Peek

Won't fix. Torn has implemented a near identical feature.


[ru] Localize Horizontal scaling

Describe your changes
Localize Horizontal scaling
:heavy_exclamation_mark:   Please review my translation carefully, as there are (I believe) some math and semantic errors in the original. Should you confirm my opinion, then we can open the issue to the original term.
:heavy_exclamation_mark:  Namely, adding four 4GB RAM modules (as the article suggests) will bring the total system memory up to 20GB. On top of that, such an action is exactly what vertical scaling is about and thus misleading.
Related issue number or link (ex: resolves #issue-number)
resolves #3116
Checklist before opening this PR (put x in the checkboxes)

[x] This PR does not contain plagiarism

don’t copy other people’s work unless you are quoting and contributing it to them.

[x] I have signed off on all commits

signing off (ex: git commit -s) is to affirm that commits comply DCO. If you are working locally, you could add an alias to your gitconfig by running git config --global alias.ci "commit -s".

/label tide/merge-method-squash


[Fork-Join] Need improvement to the error messages

In the below code segment, the if condition (test["HotelWorker"] != null) is incorrect where HotelWorker is not defined. But it throws the below error which is not meaningful to the user.

ballerina.lang.errors:Error, message: unknown error occurred

In the same code segment if the reply statement not available for a resource, the same error will be thrown, which is again not clear to the user

Same error when the join statement have invalid workers
eg: join (some 1 Tour1, XXX) (map test)

fork {
            worker Tour1 {
                system:println("Inside Tour1 Worker");
                system:println("constant value is in tour 1 is " + index);
                message tour1 = {};
                messages:setStringPayload(tour1, "This is tour 1");
                tour1 -> fork;
            }
        } join (some 1 Tour1) (map test) {

            system:println("Inside JOIN 2");
            if (test["HotelWorker"] != null) {
                any[] abc;
                abc, _ = (any[])test["HotelWorker"];
                results[0] = (message)abc[0];
                system:println(results[0]);
                //reply results;
                messages:setStringPayload(result, "ssssss");
                reply result;
            }
        }

This is happening as there were errors before executing the reply statement. So Ballerina sends this error message to the client to ack the client. This behaviour is wrong as it confuses the user

This is fixed in v0.94, where meaningful errors are logged in invalid conditions.


Cherry-pick Bulk polling of volumes for vSphere implementation to 1.7 release

Cherry-pick bulk polling of volumes for vSphere implementation https://github.com/kubernetes/kubernetes/pull/41113 to 1.7 release. Since 1.7 release branch doesn't have vsphere code refactoring changes, we have to implement this fix from scratch in 1.7 release.

We need an approval from 1.7 branch manager before we can start working. It's because when we recently cherry-picked this request to 1.8 branch, there was a denial from the 1.8 branch manager as the code change is large. We have made a special request to 1.8 branch manager to approve it as it has an performance impact.
So before we go ahead with working on this fix, it is better we can get an approval from 1.7 branch manager.

cc @BaluDontu confirmed that branch manager for 1.7 denied request to add this big change as cherry pick.


CPU usage on dockerd 100%

If I start logspout-logstash using this docker-compose the cpu usage spikes to 100% and stays there...
  logspout:
    image: bekt/logspout-logstash
    mem_limit: 256m
    restart: on-failure
    depends_on:
      - logstash
    environment:
      ROUTE_URIS: logstash://logstash:5000
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

before starting logspout
 7015 root      20   0 1423204  80956  29440 S 106.3  1.3   1:25.72 dockerd

after starting logspout
 7015 root      20   0 1423204  80972  29440 S   0.3  1.3   1:58.57 dockerd

Experiencing the same issue here:
logspout --version
v3.2-dev-custom
docker version
Client:
Version:      1.12.3
API version:  1.24
Go version:   go1.6.3
Git commit:   6b644ec
Built:        Wed Oct 26 23:26:11 2016
OS/Arch:      linux/amd64
Server:
Version:      1.12.3
API version:  1.24
Go version:   go1.6.3
Git commit:   6b644ec
Built:        Wed Oct 26 23:26:11 2016
OS/Arch:      linux/amd64
Logspout container logs with DEBUG enabled show a constant stream of:
1/26/2017 12:37:44 PM2017/01/26 17:37:44 pump.pumpLogs(): 495069bafc93 stopped
1/26/2017 12:37:45 PM2017/01/26 17:37:45 pump.pumpLogs(): 495069bafc93 started
1/26/2017 12:37:45 PM2017/01/26 17:37:45 pump.pumpLogs(): 25635c95f804 stopped
1/26/2017 12:37:45 PM2017/01/26 17:37:45 pump.pumpLogs(): 25635c95f804 started
1/26/2017 12:37:46 PM2017/01/26 17:37:46 pump.pumpLogs(): 495069bafc93 stopped
1/26/2017 12:37:46 PM2017/01/26 17:37:46 pump.pumpLogs(): 495069bafc93 started
1/26/2017 12:37:46 PM2017/01/26 17:37:46 pump.pumpLogs(): a27d4bfb3a41 stopped
1/26/2017 12:37:46 PM2017/01/26 17:37:46 pump.pumpLogs(): a27d4bfb3a41 started
1/26/2017 12:37:46 PM2017/01/26 17:37:46 pump.pumpLogs(): 32e734dcf238 stopped
1/26/2017 12:37:46 PM2017/01/26 17:37:46 pump.pumpLogs(): 32e734dcf238 started
1/26/2017 12:37:47 PM2017/01/26 17:37:47 pump.pumpLogs(): 495069bafc93 stopped
1/26/2017 12:37:47 PM2017/01/26 17:37:47 pump.pumpLogs(): 495069bafc93 started
1/26/2017 12:37:48 PM2017/01/26 17:37:48 pump.pumpLogs(): 495069bafc93 stopped
1/26/2017 12:37:48 PM2017/01/26 17:37:48 pump.pumpLogs(): a27d4bfb3a41 stopped
1/26/2017 12:37:48 PM2017/01/26 17:37:48 pump.pumpLogs(): 495069bafc93 started
1/26/2017 12:37:49 PM2017/01/26 17:37:48 pump.pumpLogs(): a27d4bfb3a41 started

@pythianarceri Are you able to provide a docker-compose.yml I can use to reproduce the issue?

We use a custom built image compiled with the logstash output module,
logspout:
  image: pythiant9shared/logspout-logstash
  environment:
    DEBUG: '1'
    ROUTE_URIS: logstash://logstash:5000
  stdin_open: true
  external_links:
  - logstash/logstash-collector:logstash
  volumes:
  - /var/run/docker.sock:/var/run/docker.sock
  tty: true
  labels:
    io.rancher.scheduler.global: 'true'
    io.rancher.container.hostname_override: container_name

Any luck getting this fixed or at least the issue identified, is it normal for the pumps to be started/stopped all the time ?
We are running under RancherOS 1.3.3 just in case

Same thing here. I end up seeing a lot of these messages in /var/log/docker.log
time="2017-03-13T21:36:26.491710944Z" level=error msg="Error streaming logs: invalid character '\\x00' looking for beginning of value"
time="2017-03-13T21:36:26.514911587Z" level=error msg="Error streaming logs: invalid character '\\x00' looking for beginning of value"
time="2017-03-13T21:36:26.547530212Z" level=error msg="Error streaming logs: invalid character '\\x00' looking for beginning of value"

It's happened a couple times. Sometimes that "invalid character" is different but the gist of the log message is the same. We see dockerd up around 100% CPU and a constant stream of those errors.

Hi all,
We've managed to reproduce the same issue as @pythianarceri - on Jan 26. To add some detail; running a docker logs -f {ContainerID} does not actually follow the logs and exits straight away. This would make sense with the pumpLogs starting and stopping as the HTTP connection being closed for some reason.
This is reproducible on the latest master commit (7c22df4e9d780959b266d7a088ff8012171edbfb) as well as 513b74c70a31c20294507f3fb21c34e52b2df48e and 8b98313954b3283d7203d1c3984b5b69ae307e0b.
Our container is running a few processed inside, all of which have PYTHONUNBUFFERED=1.
Whilst I consider this to be a bug within our container not outputting to stdout and stderr correctly it also makes our docker nodes perform really badly when the containes are scheduled.
Can we add some kind of counter when the pump starts and if it restarts {n} times within a second make it backoff?.
# docker version
Client:
 Version:      17.03.0-ce
 API version:  1.26
 Go version:   go1.7.4
 Git commit:   60ccb22
 Built:        Mon Mar 13 18:07:51 UTC 2017
 OS/Arch:      linux/arm

Server:
 Version:      17.03.0-ce
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.4
 Git commit:   60ccb22
 Built:        Mon Mar 13 18:07:51 UTC 2017
 OS/Arch:      linux/arm
 Experimental: false

Same issue here. Reverted back to docker image v3.1 and logging and CPU usage is back to normal.

Same issue, we also reverted to docker image v3.1.
We are running Docker version 17.04.0-ce, build 4845c56, and deploying our logspout container with the new swarm mode ('docker stack deploy'). We're not using logstash, but forward logs to a remote syslog directly.
Here is the content of our docker-compose.yml file:
`version: '3'
services:
logspout:
image: gliderlabs/logspout
environment:
SYSLOG_STRUCTURED_DATA: "- $SYSLOG_CONF"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
ports:
- 10888:80
command: syslog+tls://$LOGS_ENDPOINT
deploy:
mode: global`

I have the same problem, using the most recent master image (ef55cfcceda7) and version v3.3-dev on both docker 17.04.0-ce and 1.13.1.
It's fairly difficult to reproduce, and seems to occur only after a period of time - I'm not sure whether its a result of complex log messages (or potentially unexpected charsets), or just the volume of messages shipped off.
During the time that it's at high cpu usage, I also have seen a number of weird messages such as what mattdodge described
@woutor @sebgl after you reverted back to version 3.1 - did you see the problem pop up again?

@gaieges: Nope, with 3.1 the CPU usage remains normal.

Same issue there with Docker 17.05.0-ce and logspout 3.3. We can't go back to 3.1 cause we need label filtering feature.

Same issue here. Reverted back to docker image v3.1 and logging and CPU usage is back to normal.

I'm using v3.3 in production with papertrail. Is there anybody experiencing this issue that is NOT using logstash? I'm thinking the problem might be in that module...

@michaelshobbs I don't use the logstash module, I just use the tcp+syslog output.

Interesting....anybody have any thoughts on this one?

Same issue here, but i cannot ack the high cpu load. Only a lot loglines in /var/log/syslog and the pump.pumpLogs() when debugging enabled.
'''
.....
165+02:00" level=error msg="Error streaming logs: invalid character 'l' after object key:value pair"
Jun 16 11:55:18 xyz docker[11404]: time="2017-06-16T11:55:18.307515574+02:00" level=error msg="Error streaming logs: invalid character 'l' after object key:value pair"
Jun 16 11:55:18 xyz docker[11404]: time="2017-06-16T11:55:18.432245150+02:00" level=error msg="Error streaming logs: invalid character 'l' after object key:value pair"
....
'''

We encountered the same issue today randomly on one of hosts. I unfortunately do not have anymore info than what others have reported.
Docker 17.03.1-ce
RancherOS v1.0.2
logspout configured to send to syslog+tcp

Hello folks,
I was having the same problem:
Jun 28 14:07:20 xxx docker[7686]: time="2017-06-28T14:07:20.892703572Z" level=error msg="Error streaming logs: invalid character 'l' after object key:value pair"
Jun 28 14:07:20 xxx docker[7686]: time="2017-06-28T14:07:20.894972987Z" level=error msg="Error streaming logs: invalid character 'l' after object key:value pair"

and it turns out to be corrupt json-file.log, it happened when I got a disk full so json-file.log was truncated.
I had some thing like this at my json-file.log after disk space got fixed and docker running again.
{"log":"    2017/06/28 07:46:00 [INFO] consul: adding WAN ser{"log":"    2017/06/28 07:46:00 [INFO] consul: adding WAN server xxx.dc1 (Addr: <IP_ADDRESS>:8300) (DC: dc1)\n","stream":"stdout","time":"2017-06-28T07:46:00.5738544Z"}

So it isn't a valid json and it breaks docker logs, somehow it make logspout enter a loop and make docker print those lines.
If you try to run, docker logs $container  on the container with that corrupt json-file.log, you will get one line of the error.
So look for corrupted/bad formatted json-files.log
I just removed that line and everything was working normally again.
Hope that helps.

On @sfrique 's note - I wonder if using journald log driver (if possible) instead of json-file would help this?

@gaieges I tried with journald, same problem

We're seeing the same thing - 100% CPU usage when running logspout (pumping to logstash).  Enabling logspout's debug logs showed rapid repetition of log entries such as:
21/07/2017 11:57:202017/07/20 23:57:20 pump.pumpLogs(): fb6033455fd0 stopped
21/07/2017 11:57:202017/07/20 23:57:20 pump.pumpLogs(): fb6033455fd0 started

Initially, we were seeing this on most of our hosts, with two container IDs showing up on each.  The containers in question were Rancher's metadata and DNS containers.  We upgraded Rancher from 1.6.2 to 1.6.5 and upgraded all the system services, and logspout now seems to run happily on most of our hosts.  However, we're still getting the stopped/started log spam (and corresponding CPU load) on our DB hosts - with only a single container ID flopping: mongodb.

The scenario described by @pauln is exactly what we are experiencing, (we haven't had chance to upgrade to 1.6.5 yet)

@garceri, we're currently running Docker 17.03 CE, though we're looking at upgrading to 17.06 CE in the near future.

Experiencing exactly this issue quite often, my thoughts on that:

according to reaction of other people does not affect older logspout versions -> seems could be prevented on the logspout level. What is the difference between versions?
malformed logs seem to be the trouble. How they emerge? Could it be prevented on the docker daemon level?
can I help gathering any data while this happens? Running with DEBUG: true shows just the constant stream of pump start/stop

2017/07/26 07:09:48 pump.pumpLogs(): 21c90bb7ac5c stopped
2017/07/26 07:09:48 pump.pumpLogs(): 21c90bb7ac5c started
2017/07/26 07:09:48 pump.pumpLogs(): 21c90bb7ac5c stopped
2017/07/26 07:09:48 pump.pumpLogs(): 21c90bb7ac5c started

@robcza thanks for the additional details. can you provide a full debug log from start to stop of the logspout container that has entered into this state? looking for any other clues here.
To your question about difference between versions, we tag each release and thus you can browse the history by comparing tags. Additionally, we have a CHANGELOG.md with convenient links to compare each version.

Same for me on docker 17.03.1-ce using this image bekt/logspout-logstash:latest.
I have managed to get a full debug log since logspout container creation.
There is a constant start / stop stream for c21bde2e91e7.
I had this issue on other machines as well with various services.
# docker logs elk-logspout
2017/07/28 10:50:57 setting allowTTY to: false
# logspout v3.3-dev-custom by gliderlabs
# adapters: raw udp tcp logstash
# options : debug:True persist:/mnt/routes
# jobs    : http[]:80 pump routes
# routes  :
#   ADAPTER	ADDRESS		CONTAINERS	SOURCES	OPTIONS
#   logstash+tcpprod00:1111				map[]
2017/07/28 10:50:57 pump.Run(): using inactivity timeout:  0s
2017/07/28 10:50:57 pump.pumpLogs(): 4dd58c94144a ignored: environ ignore
2017/07/28 10:50:57 pump.pumpLogs(): 741c807703d4 started
2017/07/28 10:50:57 pump.pumpLogs(): 03c007defa61 started
2017/07/28 10:50:57 pump.pumpLogs(): 0398b9b5542e started
2017/07/28 10:50:57 pump.pumpLogs(): 0b1b04fd17a9 started
2017/07/28 10:50:57 pump.pumpLogs(): 18b4f33a6b1d started
2017/07/28 10:50:57 pump.pumpLogs(): 18cd5d3b88cb started
2017/07/28 10:50:57 pump.pumpLogs(): dfebc3118ce3 started
2017/07/28 10:50:57 pump.pumpLogs(): e12708a548c4 started
2017/07/28 10:50:57 pump.pumpLogs(): 78423dfc9805 started
2017/07/28 10:50:57 pump.pumpLogs(): 872c498a0c2f started
2017/07/28 10:50:57 pump.pumpLogs(): 2f78048ee356 started
2017/07/28 10:50:57 pump.pumpLogs(): bcedcc0cd5d2 started
2017/07/28 10:50:57 pump.pumpLogs(): 0b8e0146d61c started
2017/07/28 10:50:57 pump.pumpLogs(): a1c7966d4ad3 started
2017/07/28 10:50:57 pump.pumpLogs(): d49788286a32 started
2017/07/28 10:50:57 pump.pumpLogs(): 41c5e05d0019 started
2017/07/28 10:50:57 pump.pumpLogs(): ce7030af1846 started
2017/07/28 10:50:57 pump.pumpLogs(): 78cff52b2cc8 started
2017/07/28 10:50:57 pump.pumpLogs(): c5b69f63a3e8 started
2017/07/28 10:50:57 pump.pumpLogs(): 52ab8eb7e542 started
2017/07/28 10:50:57 pump.pumpLogs(): f26b009798f4 started
2017/07/28 10:50:57 pump.pumpLogs(): 7b1f7d90859a started
2017/07/28 10:50:57 pump.pumpLogs(): a5b864e479b7 started
2017/07/28 10:50:57 pump.pumpLogs(): 08581c7c3829 started
2017/07/28 10:50:57 pump.pumpLogs(): 9e4497a0dcaf started
2017/07/28 10:50:57 pump.pumpLogs(): 227ca66d7f83 started
2017/07/28 10:50:57 pump.pumpLogs(): 6f6b327b7b9e started
2017/07/28 10:50:57 pump.pumpLogs(): 846ba10f0ee7 started
2017/07/28 10:50:57 pump.pumpLogs(): e672bc2d0fe2 started
2017/07/28 10:50:57 pump.pumpLogs(): f1d48ee0211f started
2017/07/28 10:50:57 pump.pumpLogs(): f8fdca2aff57 started
2017/07/28 10:50:57 pump.pumpLogs(): 5856ed01b7ff started
2017/07/28 10:50:57 pump.pumpLogs(): 57e77510b9de started
2017/07/28 10:50:57 pump.pumpLogs(): a05333be7c0f started
2017/07/28 10:50:57 pump.pumpLogs(): dcab95f7ca2f started
2017/07/28 10:50:57 pump.pumpLogs(): f28130fa353a started
2017/07/28 10:50:57 pump.pumpLogs(): 464d4d0b24bd started
2017/07/28 10:50:57 pump.pumpLogs(): 139fccfc578f started
2017/07/28 10:50:57 pump.pumpLogs(): 341f311f43c0 started
2017/07/28 10:50:57 pump.pumpLogs(): f08a73de23ab started
2017/07/28 10:50:57 pump.pumpLogs(): 7e88c606f51b started
2017/07/28 10:50:57 pump.pumpLogs(): c21bde2e91e7 started
2017/07/28 10:50:57 pump.pumpLogs(): 52fe6826666a started
2017/07/28 10:51:02 pump.pumpLogs(): c21bde2e91e7 stopped
2017/07/28 10:51:02 pump.pumpLogs(): c21bde2e91e7 started
2017/07/28 10:51:06 pump.pumpLogs(): c21bde2e91e7 stopped
2017/07/28 10:51:06 pump.pumpLogs(): c21bde2e91e7 started
2017/07/28 10:51:09 pump.pumpLogs(): c21bde2e91e7 stopped
2017/07/28 10:51:09 pump.pumpLogs(): c21bde2e91e7 started
2017/07/28 10:51:11 pump.pumpLogs(): c21bde2e91e7 stopped
2017/07/28 10:51:11 pump.pumpLogs(): c21bde2e91e7 started
2017/07/28 10:51:14 pump.pumpLogs(): c21bde2e91e7 stopped
2017/07/28 10:51:14 pump.pumpLogs(): c21bde2e91e7 started

Thank @ko-christ for more detail! Next time this happens it would be great to see if there's anything odd about the output of the container with the "thrashing"  log pump. (i.e. docker logs c21bde2e91e7 in this particular case)
Same request to others that have noticed the stop/start thrash symptom.

Hi all,
We've just encountered this issue recently. (logspout starts and CPU jumps to ~100%)
Here are my insights after investigating this for a couple of hours:
In our environment, this was caused due to malformed JSON output in the container logs (when using json-file logging).
What apparently happened is that the machine ran out of space (we had a clear indication for that) and log lines could not be written properly - so we got json log line that were cut in the middle.
When that happens, docker will print out the following whenever someone tries to fetch the logs (even a simple docker logs command will reproduce this)

"Error streaming logs: invalid character 'l' after object key:value pair"

Apparently, docker will print this error and stop returning the log lines, even if there were many more lines after the corrupt json line.
On our machine we had about a dozen services constantly logging and this JSON line corruption was found in most of them.
It seems that docker encounters this and emits an error( https://github.com/moby/moby/blob/8af4db6f002ac907b6ef8610b237879dfcaa5b7a/daemon/logs.go#L109 ) yet Logspout immediately starts requesting logs again and, again, docker bombards the docker service logs with the above error message (tens per second).
Possible workarounds for such an issue (I didn't verify these options yet):

Setting the BACKLOG env var and restarting services
Stopping the service and moving/truncating the existing log files so that they would start off clean.

So this actually seems to be a docker issue. The best way to deal with this, IMHO, is to have docker detect such cases and roll the log file and put the corrupt log file aside (while prompting a meaningful warning about this)
One thing that puzzles me is why reverting to v3.1 "fixes" this issue (I didn't try reverting back to v3.1 yet, I'm using v3.3).

Hello,
I am having the same issue. Docker log is flooded with messages like this:
time="2017-08-30T22:30:31.922624624Z" level=error msg="Error streaming logs: invalid character '\\x00' looking for beginning of value"
time="2017-08-30T22:30:31.977376555Z" level=error msg="Error streaming logs: invalid character '\\x00' looking for beginning of value"
time="2017-08-30T22:30:32.172320949Z" level=error msg="Error streaming logs: invalid character '\\x00' looking for beginning of value"
time="2017-08-30T22:30:32.570219033Z" level=error msg="Error streaming logs: invalid character '\\x00' looking for beginning of value"
time="2017-08-30T22:30:32.662642497Z" level=error msg="Error streaming logs: invalid character '\\x00' looking for beginning of value"
time="2017-08-30T22:30:32.860254549Z" level=error msg="Error streaming logs: invalid character '\\x00' looking for beginning of value"

And the docker logs for bekt/logspout-logstash:master are:
2017/08/30 22:18:33 pump.pumpLogs(): 9972da45bf76 started
2017/08/30 22:18:33 pump.pumpLogs(): f6fddb18f835 started
2017/08/30 22:18:34 pump.pumpLogs(): 9972da45bf76 stopped
2017/08/30 22:18:34 pump.pumpLogs(): 9ab5e8cc0c36 stopped
2017/08/30 22:18:34 pump.pumpLogs(): 9972da45bf76 started
2017/08/30 22:18:34 pump.pumpLogs(): 9ab5e8cc0c36 started
2017/08/30 22:18:34 pump.Run() event: 42ad09d40141 network:connect
2017/08/30 22:18:34 pump.Run() event: orient-backu volume:mount
2017/08/30 22:18:34 pump.pumpLogs(): f6fddb18f835 stopped
2017/08/30 22:18:35 pump.pumpLogs(): f6fddb18f835 started
2017/08/30 22:18:35 pump.Run() event: orient-datab volume:mount
2017/08/30 22:18:35 pump.pumpLogs(): 9972da45bf76 stopped
2017/08/30 22:18:35 pump.Run() event: orient-log volume:mount
2017/08/30 22:18:35 pump.pumpLogs(): 9972da45bf76 started
2017/08/30 22:18:35 pump.Run() event: rancher-cni volume:mount
2017/08/30 22:18:35 pump.Run() event: efe51a4c5ed0 volume:mount
2017/08/30 22:18:36 pump.pumpLogs(): f6fddb18f835 stopped
2017/08/30 22:18:36 pump.pumpLogs(): 9972da45bf76 stopped
2017/08/30 22:18:36 pump.pumpLogs(): 9ab5e8cc0c36 stopped
2017/08/30 22:18:36 pump.Run() event: 58a80e107fc1 start
2017/08/30 22:18:36 pump.pumpLogs(): 9ab5e8cc0c36 started
2017/08/30 22:18:36 pump.pumpLogs(): 9972da45bf76 started
2017/08/30 22:18:36 pump.pumpLogs(): f6fddb18f835 started
2017/08/30 22:18:36 pump.pumpLogs(): 58a80e107fc1 started
2017/08/30 22:18:36 pump.pumpLogs(): 9972da45bf76 stopped
2017/08/30 22:18:36 pump.pumpLogs(): 9972da45bf76 started
2017/08/30 22:18:36 pump.pumpLogs(): 9ab5e8cc0c36 stopped
2017/08/30 22:18:36 pump.pumpLogs(): 9ab5e8cc0c36 started
2017/08/30 22:18:37 pump.pumpLogs(): 9972da45bf76 stopped
2017/08/30 22:18:37 pump.pumpLogs(): 9972da45bf76 started
2017/08/30 22:18:37 pump.pumpLogs(): f6fddb18f835 stopped
2017/08/30 22:18:37 pump.pumpLogs(): f6fddb18f835 started
2017/08/30 22:18:37 pump.pumpLogs(): 9ab5e8cc0c36 stopped
2017/08/30 22:18:37 pump.pumpLogs(): 9ab5e8cc0c36 started
2017/08/30 22:18:37 pump.pumpLogs(): 9972da45bf76 stopped
2017/08/30 22:18:37 pump.pumpLogs(): 9972da45bf76 started
2017/08/30 22:18:38 pump.pumpLogs(): be68d546188d stopped
2017/08/30 22:18:38 pump.pumpLogs(): be68d546188d started
2017/08/30 22:18:38 pump.pumpLogs(): f6fddb18f835 stopped
2017/08/30 22:18:38 pump.pumpLogs(): f6fddb18f835 started
2017/08/30 22:18:38 pump.pumpLogs(): 9972da45bf76 stopped
2017/08/30 22:18:38 pump.pumpLogs(): 9972da45bf76 started
2017/08/30 22:18:38 pump.pumpLogs(): 9ab5e8cc0c36 stopped
2017/08/30 22:18:38 pump.pumpLogs(): 9ab5e8cc0c36 started
2017/08/30 22:18:39 pump.pumpLogs(): 9972da45bf76 stopped
2017/08/30 22:18:39 pump.pumpLogs(): 9972da45bf76 started
2017/08/30 22:18:39 pump.pumpLogs(): 9ab5e8cc0c36 stopped
2017/08/30 22:18:39 pump.pumpLogs(): 9ab5e8cc0c36 started
2017/08/30 22:18:39 pump.pumpLogs(): f6fddb18f835 stopped
2017/08/30 22:18:39 pump.pumpLogs(): 9972da45bf76 stopped
2017/08/30 22:18:39 pump.pumpLogs(): f6fddb18f835 started
2017/08/30 22:18:39 pump.pumpLogs(): 9972da45bf76 started
2017/08/30 22:18:40 pump.pumpLogs(): 9ab5e8cc0c36 stopped
2017/08/30 22:18:40 pump.pumpLogs(): 9972da45bf76 stopped
2017/08/30 22:18:40 pump.pumpLogs(): 9972da45bf76 started
2017/08/30 22:18:40 pump.pumpLogs(): 9ab5e8cc0c36 started
2017/08/30 22:18:40 pump.pumpLogs(): f6fddb18f835 stopped
2017/08/30 22:18:40 pump.pumpLogs(): f6fddb18f835 started`

I have trimmed some of the docker-json logs and restarted the containers and it is not helping, once the logstput container starts it makes dockerd to eat almost all the CPU.
Thanks in advance!

@mattatcha thoughts on this given new info?

@pythiancastaneda To figureout what container is giving you the trouble:
# tail -f /var/log/syslog | grep docker
Second term:
# docker logs "for each container you have"

That way you will see at logs:
time="2017-08-30T22:30:32.860254549Z" level=error msg="Error streaming logs: invalid character '\\x00' looking for beginning of value"

And then you can remove the log or the wrong line.
Hope that helps.
I recently had the same problem and i did the steps above and fixed the issue.

Thanks for the update @stevecalvert ! In my case issue was the same corrupt container logs. For me, 3.2.2 also was logging an additional error. Posting that just in case
logspout_1  | 2017/09/15 17:42:43 pump.pumpLogs(): 0e1c957edf2d started
logspout_1  | 2017/09/15 17:42:43 pump.pumpLogs(): 0e1c957edf2d stopped with error: Unrecognized input header: 3
logspout_1  | 2017/09/15 17:42:43 pump.pumpLogs(): 0e1c957edf2d started
logspout_1  | 2017/09/15 17:42:43 pump.pumpLogs(): 0e1c957edf2d stopped with error: Unrecognized input header: 3

Only way for me to fix that issue, like for others, was to remove and recreate container.

@leriel
I get the same issue  with logspout 3.2.2 and gelf plugins.
But i'm sure to understand the workarround ...
You recreate which container ? Logspout or containers with corrupteur json-file logs ?
Is there any way to correct logspout code ?

Don't know if thee is any relationship ...
but I get this message if I tried to tail on a *-json.log, given by logspout in debug mode
tail: inotify cannot be used, reverting to polling: Too many open files
seem to by a default limit on ubuntu ? that could explain a some corrupted json-file logs ?
see https://nefaria.com/2014/08/tail-inotify-resources-exhausted/

Another info ...
when I purge all json-file logs given "stopped" by logspout, and restart it, then cpu usage on dockerd is flat
here my code:
for containerid in $(docker logs $(docker ps -a -q --filter "label=io.rancher.stack_service.name=graylog2/logspout")  2>&1| grep stopped | cut -d " " -f 4 | sort -u); do
     ls -l /var/lib/docker/containers/${containerid}*/${containerid}*-json.log
   > /var/lib/docker/containers/${containerid}*/${containerid}*-json.log
done

One more information,
I use docker engine 17.03.2-ce. Last week, we had a power shutdown on the nodes.
In my case, I think that it was the cause, why we get corrupted json-file logs (Error streaming logs: invalid character 'l' after object key:value pair)
Do you know if these issue can be solved with a new docker-ce release ?

Here some news ...
But don't know if there is a relationship
I get all  containerid given stopped by logspout:
docker logs $(docker ps -a -q --filter "label=io.rancher.stack_service.name=graylog/logspout")  2>&1| grep stopped | cut -d " " -f 4 | sort -u
c062de60016b
c4d6b098968c
f8f18bba6203

I effectivelly found  '\x00' of each file that is corrupted, and files are big

for containerid in $(docker ps -a -q); do
     grep -Pa '\x00' /var/lib/docker/containers/${containerid}*/${containerid}*-json.log
     if [ $? -eq 0 ]; then
          echo ${containerid}
          ls -lh /var/lib/docker/containers/${containerid}*/${containerid}*-json.log
    fi
done

{"log":"Creating volume folders...\n","stream":"stdout","time":"2017-09-16T15:50:34.379859731Z"}
c4d6b098968c
-rw-r----- 1 root root 564M Sep 21 12:11 /var/lib/docker/containers/c4d6b098968c475a82ed7ff34cc4e50344f097155f10d0d9d07528437b4417e7/c4d6b098968c475a82ed7ff34cc4e50344f097155f10d0d9d07528437b4417e7-json.log
{"log":"Creating volume folders...\n","stream":"stdout","time":"2017-09-16T15:50:33.076131673Z"}
c062de60016b
-rw-r----- 1 root root 478M Sep 21 12:11 /var/lib/docker/containers/c062de60016bd13faeebceb0301c121b35a0dfbc620048584a3d06c3d0449761/c062de60016bd13faeebceb0301c121b35a0dfbc620048584a3d06c3d0449761-json.log
{"log":"Creating volume folders...\n","stream":"stdout","time":"2017-09-16T15:50:47.996912867Z"}
f8f18bba6203
-rw-r----- 1 root root 806M Sep 21 12:11 /var/lib/docker/containers/f8f18bba62032e188dd75f76b9cc061936e804995289d7406b89d43e02d7cc36/f8f18bba62032e188dd75f76b9cc061936e804995289d7406b89d43e02d7cc36-json.log

We experienced the same issue yesterday.
We don't use logspout, but it is the closest issue we found, with this kind of error logs:
Jun 28 14:07:20 xxx docker[7686]: time="2017-06-28T14:07:20.892703572Z" level=error msg="Error streaming logs: invalid character 'l' after object key:value pair"

And the invalid char being anything in our case.
We are still investigating, but would like to see if we can find the issue together.
It looks like with our current information that one of our containers is outputting some kind of json under specific conditions, and that dockerd doesn't like it at all. Didn't find any moby related issue though :/
We'll let you know of our findings.

Hello I have:
$ docker version
Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:42:09 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:40:48 2017
 OS/Arch:      linux/amd64
 Experimental: false

System: Debian 9.1
logspout setup in Rancher:

and in syslog dockerd errors:
Oct 24 10:03:39 web4-vultr dockerd[31377]: time="2017-10-24T10:03:39.819294736+01:00" level=error msg="Error streaming logs: invalid character '\x00' looking for beginning of value" container=db62afc089ec method="(*Daemon).ContainerLogs" module=daemon
Oct 24 10:03:41 web4-vultr dockerd[31377]: time="2017-10-24T10:03:41.950449384+01:00" level=error msg="Error streaming logs: invalid character '\x00' looking for beginning of value" container=db62afc089ec method="(*Daemon).ContainerLogs" module=daemon
Oct 24 10:03:42 web4-vultr dockerd[31377]: time="2017-10-24T10:03:42.717452402+01:00" level=error msg="Error streaming logs: invalid character '\x00' looking for beginning of value" container=6488521fdc8f method="(*Daemon).ContainerLogs" module=daemon

and CPU usage is very high
I dont know why, but downgrade logspout to version v3.1 fix it..

We're on AWS and see this issue regularly. We have ~100 EC2 instances and when AWS suddenly retires or shuts down an instance the server restarts but some of the docker json log files end up corrupt.
I can see where the corruption is and what line by doing the following:
for FILE in /var/lib/docker/containers//-json.log; do cat $FILE | jq . > /dev/null; done
I used to just shutdown the container that corresponded to that corrupt logs(s), manually remove the corrupted (half written json line), then restart the container and all was well. Though with newer version of docker that no-longer fixes the issues I have to remove all the containers and re-deploy them to fix it now. I think this is really a Docker issue. If Docker detects a half written file I think it needs to handle this correctly. Basically 100% of the time AWS restarts a node unexpectedly this happens for us. Typically it's faster to completely replace the node than spend a bunch of time trying to fix this by hand.

We have the same issue but found the offending log files using grep below.
grep -P '\x00' /var/lib/docker/containers/**/*json.log

After manually removing the null "\x00" characters from the log the issue went away.
It seems to be a docker issue docker/for-linux#140 related to unexpected host/daemon restarts.

me to same issue ... any suggestion ?

We resolved to use #321 as a workaround for the time being. (note that BACKLOG should not be set, or be set to true).

I've faced with 100% CPU usage and error:
Mar 23 16:48:52 burger dockerd[569]: time="2018-03-23T16:48:52.971499270+02:00" level=error msg="Error streaming logs: EOF" container=b1dc4286ba72 method="(*Daemon).ContainerLogs" module=daemon

on cadvisor container.
My config:
version: '2.3'

services:

  logspout:
    image: gliderlabs/logspout:v3.2.4
    container_name: logspout
    command: syslog+tcp://logs-01.loggly.com:514
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      TAIL: 10

+upvoting rollback to 3.1
I've just force pushed ca675f3030a87f41536415114800470c0c74cd0d and manually added:
- glide.lock
- glide.yaml
- Dockerfile
- build.sh
- and also /adapters/multiline/multiline.go to use multiline

from latest master to build an image
new container works good

Hi! I had this problem today and I had a corrupted log.
2018/11/28 19:29:49 pump.pumpLogs(): 5ed6da2c05f0 started, tail: all 2018/11/28 19:29:49 pump.pumpLogs(): 5ed6da2c05f0 stopped with error: Unrecognized input header: 3 2018/11/28 19:29:49 pump.pumpLogs(): 5ed6da2c05f0 started, tail: all 2018/11/28 19:29:49 pump.pumpLogs(): 5ed6da2c05f0 stopped with error: Unrecognized input header: 3 2018/11/28 19:29:49 pump.pumpLogs(): 5ed6da2c05f0 started, tail: all 2018/11/28 19:29:49 pump.pumpLogs(): 5ed6da2c05f0 stopped with error: Unrecognized input header: 3 2018/11/28 19:29:49 pump.pumpLogs(): 5ed6da2c05f0 started, tail: all
/tmp# docker logs -f 5ed6da2c05f0 2018-11-28T07:28:07Z [INFO] Loading configuration 2018-11-28T07:28:07Z [INFO] Unable to parse user data: invalid character '#' looking for beginning of value 2018-11-28T07:28:07Z [INFO] Amazon ECS agent Version: 1.22.0, Commit: 26518174 error from daemon in stream: Error grabbing logs: invalid character '{' in \u hexadecimal character escape
So, if you kill that container, remove and re create the problem will be solved. Hope it will help.

I just wanted to confirm about this issue with the latest CE version of Docker and logspout (syslog+tls). We've had production servers crashing because of this. Dockerd was using 1600% CPU and then became unresponsive on a 32 CPU server. Noticed a very high number of goroutines shwoing with docker info in debug mode. Stopping the logspout container and waiting a few hours resolved the issue and the Dockerd CPU was back to normal. Tried a workarounds like TAIL=0 without success. Reverting to v3.1 solves the issue for now.
We had about 30 containers running on that machine, logging was heavy but not excessive. It's easily reproducible with the latest logspout version: starting logspout with a lot of containers logging show dockerd using high CPU almost immediately and increasing over time.

I just wanted to confirm about this issue with the latest CE version of Docker and logspout (syslog+tls). We've had production servers crashing because of this. Dockerd was using 1600% CPU and then became unresponsive on a 32 CPU server. Noticed a very high number of goroutines shwoing with docker info in debug mode. Stopping the logspout container and waiting a few hours resolved the issue and the Dockerd CPU was back to normal. Tried a workarounds like TAIL=0 without success. Reverting to v3.1 solves the issue for now.
We had about 30 containers running on that machine, logging was heavy but not excessive. It's easily reproducible with the latest logspout version: starting logspout with a lot of containers logging show dockerd using high CPU almost immediately and increasing over time.


Guard the substring calls in StateMachine parser to avoid bounds issues

Summary of the Pull Request
Perform checking on std::basic_string_view<T>.substr() calls to
prevent running out of bounds and sporadic Privileged Instruction throws
during x86 tests.
PR Checklist

[x] Closes the x86 tests failing all over the place since #4125 for no
apparent reason
[x] I work here
[x] Tests pass

Detailed Description of the Pull Request / Additional comments
It appears that not all std::basic_string_view<T>.substr() calls are
created equally. I rooted around for other versions of the code in our
source tree and found several versions that were less careful about
checking the start position and the size than the one that appears when
building locally on dev machines.
My theory is that one of these older versions is deployed somewhere in
the CI. Instead of clamping down the size parameter appropriately or
throwing correctly when the position is out of bounds, I believe that
it's just creating a substring with a bad range over an
invalid/uninitialized memory region. Then when the test operates on
that, sometimes it turns out to trigger the privileged instruction
NTSTATUS error we are seeing in CI.
Test Procedure

Fixed the thing
Ran the CI and it worked
Reverted everything and turned off all of the CI build except just
the parser tests (and supporting libraries)
Ran CI and it failed
Put the fix back on top (cherry-pick)
It worked.
Ran it again.
It worked.
Turn all the rest of the CI build back on

@miniksa After resolving a conflict, #4105 was merged in. Then I faced the trouble with the x86 tests.

/azp run

@DHowett-MSFT, seems fine (TM)

k, push forced back to just the change on master.

@msftbot merge this in 5 minutes


Implement Run Energy depleting/restoring

Addresses https://github.com/apollo-rsps/apollo/issues/87 in which run energy depletion and restoring is requested

Codecov Report

Merging #423 into kotlin-experiments will decrease coverage by 0.03%.
The diff coverage is 5.55%.

@@                   Coverage Diff                    @@
##             kotlin-experiments     #423      +/-   ##
========================================================
- Coverage                 23.23%   23.19%   -0.04%
  Complexity                  811      811
========================================================
  Files                       643      643
  Lines                     11123    11138      +15
  Branches                   1632     1634       +2
========================================================
  Hits                       2584     2584
- Misses                     8251     8266      +15
  Partials                    288      288

Impacted Files
Coverage Δ
Complexity Δ

...o/game/sync/task/PrePlayerSynchronizationTask.java
0% <0%> (ø)
0 <0> (ø)
:arrow_down:

...main/java/org/apollo/game/model/entity/Player.java
36.52% <8.33%> (-1.49%)
24 <0> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cd06a69...ec165b2. Read the comment docs.


refactor

Cool! Again huge work! Looks good :)

Cool! Again huge work! Looks good :)

thanks! I did this a long time ago, now I'm trying to get familiar with the code again lol


postgres_exporter takes >5mins to generate metrics on large database

What happened?
$title
Did you expect to see something different?
postgres_exporter /metrics page must return data quickly in few seconds.
How to reproduce it (as minimally and precisely as possible):
Run against large postgres with quite bit of child tables/hyper tables.
Environment
I've been running postgres/timescale for 2 weeks with >5000 hyper tables and quite a bit of data(1.2TiB).

tobs version:
tobs version

Kubernetes version information:
kubectl version

Kubernetes cluster kind:
insert how you created your cluster: kops, bootkube, etc.

postgres_exporter logs

ts=2022-11-18T12:55:45.767Z caller=postgres_exporter.go:617 level=debug msg="Querying PostgreSQL version" server=/var/run/postgresql:5432
ts=2022-11-18T12:55:45.776Z caller=pg_setting.go:35 level=debug msg="Querying pg_setting view" server=/var/run/postgresql:5432
ts=2022-11-18T12:55:45.792Z caller=namespace.go:193 level=debug msg="Querying namespace" namespace=pg_stat_replication
ts=2022-11-18T12:55:45.794Z caller=namespace.go:193 level=debug msg="Querying namespace" namespace=pg_replication_slots
ts=2022-11-18T12:55:45.797Z caller=namespace.go:193 level=debug msg="Querying namespace" namespace=pg_stat_archiver
ts=2022-11-18T12:55:45.809Z caller=collector.go:200 level=debug msg="collector succeeded" name=bgwriter duration_seconds=0.045750953
ts=2022-11-18T12:55:45.817Z caller=namespace.go:193 level=debug msg="Querying namespace" namespace=pg_stat_activity
ts=2022-11-18T12:55:45.830Z caller=namespace.go:193 level=debug msg="Querying namespace" namespace=pg_stat_database
ts=2022-11-18T12:55:45.851Z caller=namespace.go:193 level=debug msg="Querying namespace" namespace=pg_stat_database_conflicts
ts=2022-11-18T12:55:45.866Z caller=namespace.go:193 level=debug msg="Querying namespace" namespace=pg_locks
ts=2022-11-18T13:00:20.934Z caller=collector.go:200 level=debug msg="collector succeeded" name=database duration_seconds=275.167551805

tobs Logs:

insert tobs logs relevant to the issue here

Anything else we need to know?:

I will move this to upstream postgres_exporter once I find the root cause.

The query which lists database along with it's size takes the time.
SELECT pg_database.datname
		,pg_database_size(pg_database.datname)
		FROM pg_database;

Looks like the large database is the culprit.
SELECT pg_database.datname
                ,pg_database_size(pg_database.datname)
                FROM pg_database;
  datname  | pg_database_size
-----------+------------------
 postgres  |<PHONE_NUMBER>619
 template1 |         10265091
 template0 |          8602115

SELECT pg_database_size('postgres') takes a lot of time(~5mins) to complete. Probably we should have an option to disable db size population from the exporter?

I see that there is an option collector.database to disable it, but that doesn't seem to work!

--no-collector.database should fix the problem

@paulfantom May be we should consider bringing the change to cloud as well.
CC: @cevian


fix(config): remove unsupported double tap on GE 26932; add double tap to 12730; fix parameters

Closes #2042. Also removes errnonous parameters reported by @freshcoast, and adds double tap support to the 12730 fan controller.

So as far as I understand this: There are two different versions of a device we cannot distinguish based on the information we currently use for that. One is ZW+, supports Associations and has broken Supervision support. One is legacy Z-Wave, does not support associations and (probably) has no Supervision support.
In that case, tough luck for the legacy device. The supervision flag is a no-op then, so it doesn't hurt either.

So as far as I understand this: There are two different versions of a device we cannot distinguish based on the information we currently use for that.

For the record, these devices have different product IDs and different config files. They are distinguished just fine.

Okay, can you summarize the problem in a way that someone who doesn't know anything about those devices (i.e. me) can grasp it?

I don't think there's any problem at this point, as the additional items in the config file are inconsequential as has been discussed. The conversation just devolved into an argument about whether the devices are the same or not, and I'm stubborn enough not to concede on that point.
If you guys want to keep the config files as it is just to be safe, go for it, there was no objection from me, just observations. If the argument is that there's a 12730 somewhere out there in the world that, despite all documentation, is Z-Wave Plus and identifies as a Fan Switch, well you got me, that's impossible to disprove.
In my defense I did not ask for any changes to this PR (aside from the removed config parameters which was quickly fixed).
GE 12730:

It is a fan controller marketed as a ZW4002.
It is Z-Wave Classic, using a 300-series chip.
It has its own product ID of 0x3034.
It uses config file packages/config/config/devices/0x0063/ge_12730_zw4002.json.
Because it is Z-Wave Classic, the specific device type is "Multilevel Power Switch", not "Fan Switch". Thus the Supervision CC is not a mandatory CC and it is unaffected by the Supervision CC bug.
It does not support Supervision CC according to the logs I posted.
It does not support Association CC according to the logs I posted.

GE 14287:

It is also a fan controller marketed as a ZW4002.
It is Z-Wave Plus, using a 500-series chip.
It has its own product ID of 0x3131.
It uses config file packages/config/config/devices/0x0063/14287.json
It presumably supports Association CC, I don't have anything to say in regards to that because I don't have the switch installed and can't check it.
It is a Z-Wave Plus "Fan Switch" DT, but doesn't support Supervision CC (as others have pointed out). Thus, it is affected by the mandatory CC problem which requires removing that CC.

There are many of these fan controllers marketed as a ZW4002. I won't argue that they aren't the same hardware or don't share FCC IDs, that's fine. I will continue to state they are not identical in functionality:

Different product IDs
Various z-wave chips (300 vs 500)
Different versions of Z-Wave: Classic, Plus without S2, Plus with S2
Different config parameters, some documented and some undocumented

Adding the associations for the compat flag to the non-zwave-plus 12730 may not have any consequences. Removing the Supervision CC from the fans that do support S2 and Supervision would be! For example, (according to the alliance DB) the 55259 version of the switch supports Supervision, yet the config file removes it. It also supports Central Scene (according to the alliance DB an manual), but it's automatically enrolled into the Group 3 Double Taps association group. Are those the right things to do? I don't know, just saying not everything is identical... 🤷

We purposefully include double tap for central scene controllers. Both work and server different functions. Many people that control devices by direct association must react to the double tap to refresh devices that don't call on their change via a scene event. While you could do this via central scene, most people have multiple generations of switches and so doing it the same way is easier.
More importantly, however, if we don't setup for double tap and the controller somehow gets added to group 3...and double tap up or down takes the state out of sync with HA and causes havoc.

Again, I was not making a judgement about whether it is right or wrong, and not asking for any changes, just pointing out that the "same" devices are in fact different.

More importantly, however, if we don't setup for double tap by unmapping the basic sets and the controller somehow gets added to group 3...any double tap up or down takes the state out of sync with HA

I am probably not understanding this use case, but it sounds like a problem exclusive to zwavejs2mqtt? For HA's zwave_js integration, these Basic Sets are just events. The double taps would have no effect on the switch states, which use Binary Switch or Multilevel CCs, not Basic CC.
I personally don't think it's right to force every user to have the controller in group 3 if central scene is available, just my opinion.  It generates unnecessary network traffic. There's no way to opt out is there, I'd have to manage my own config file?

if we can assume that every device reporting itself as a 12730 doesn't have the firmware or features intended for the 14287. I'm just not sure I'm willing to take that bet.

I've seen the same product with different firmware having different functionality, but if you're telling me devices lie about their identity, well I have no response to that. I guess chalk it up to another example of a poor certification process.

If not unmapped, the basic set is mapped to the primary CC and will be sent through as a value change by the node. This affects every front end that uses node as the backend. It's only sent as an event if we do the compat flag...that's what it does.
Whether to force the lifeline is of course a point for debate. It doesn't actually cause any additional traffic on the zwave network, and a minuscule difference in traffic for the event bus. I suppose that we could setup the compat flag but not the lifeline flag for central scene devices, meaning those users would have an additional step to enable double tap. That creates a difference in apparent functionality that is best avoided to reduce service calls, though. I'd entertain that if you wanted to track down each and every device that is central scene, and for which the model was never sold without central scene, and then PR those changes. I have enough on my plate.

If not unmapped, the basic set is mapped to the primary CC (Binary or MuliLevel) per the spec and will be sent through as a value change by the node. This affects every front end that uses node as the backend. It's only sent as an event if we do the compat flag...that's what it does. It unmaps it and sends it as an event.

Fair enough, thanks for the explanation, I forget zwave-js maps the Basic Sets. I doubt GE intended for the double taps to be used in this way, which is why it's a strange implementation.

We all get it. You know the zwave spec very well and I'm sure will be a great resource. But we're trying to foster a more positive community than what ultimately led to OZW's demise.

Wow, all that was unnecessary. I'm sorry I struck such a nerve. 😫 I never asked for any changes to be made nor demanded anything from you.

 But that's my point exactly. If you aren't raising a concern or asking for changes to be made that why is it relevant in a PR discussion thread?

Your comments are quite unfair, and you are clearly attacking me. Normally I'd just let this pass, but you continue to misrepresent my comments and are suggesting things I've never said; therefore I feel I need to defend myself.
First, after I did "raise a concern" about the association groups, you became hostile merely because I challenged your assertion that the devices were identical. You falsely accused me of arguing about the compat flag, and blamed me for wasting your time. That comment thread was about the association groups! I also did not force you to engage and I stopped after you became upset. Let me repeat, I stopped engaging in any arguments at this point.
Second, in the actual conversation about the compat flag I didn't even argue about anything! All I did was make an observation that it probably wasn't necessary! No demands, nothing. I didn't even challenge your response, I only mentioned the devices files were different.
My comments may have been blunt, so my apologies for that, but it wasn't deserving of the hostility.

If you aren't raising a concern or asking for changes to be made then why is it relevant in a PR discussion thread? This isn't the proper forum for debate.

Now you accuse me of wanting to debate, but I was literally asked a question by Al. I replied to the question I was asked. I did not start any debate. I did not push for any changes. I did explain my position that not all of the fan devices are the same. I did give an opinion that maybe not all config files should be the same. Again, I was asked a question! The fact is, you were the one who inserted yourself into that second conversation, engaging me and starting a "debate". I had no intention of commenting further until you replied to me, where I assumed you were open for a conversation at the point. That I regret.
Out of nowhere you attacked me a second time, accusing me of grandstanding, being toxic and creating a negative environment, wanting to "start a debate", and blaming me for wasting your time. For what?  I'm honestly confused about that one.
You're right, this PR is no place for a debate, but I'll remind you again that you started this new conversation, so accusing me of the one debating is in bad faith.

This non-issue argument over which you aren't objecting to what we're doing has delayed the entire Aeotec set by a day and frankly put me out of the mood of working on it tomorrow.

I agree it's a non-issue argument and I'm truly sorry for wasting your time, because I feel the same! However, once again I didn't force you to engage in a conversation that you decided to initiate with me. To blame me for your delays and mood is not fair at all.
Feel free to lock this conversation, it's not doing anyone any good.


Connect to Heroku SSL problem

I'm trying to connect to Heroku Database prisma2 init, blank template.
everything is filed correctly but I'm getting an error:
 ERROR  self signed certificate

Pretty sure this is related to #884

Try this

Adding sslaccept=accept_invalid_certs to your datasource url should work for now

Originally posted by @williamluke4 in https://github.com/prisma/prisma2/issues/884#issuecomment-551697427

Try this

Adding sslaccept=accept_invalid_certs to your datasource url should work for now

Originally posted by @williamluke4 in #884 (comment)

Yes, I'm connecting to local DB and then sslaccept=accept_invalid_certs is helping

I am going to close this as a duplicate of https://github.com/prisma/prisma2/issues/884


Required a pipeline  for updating more then on tile.

Hi Team,
I am not facing any issue. Its just an request for updating more then one tile using concourse.
I have done with updating single tile "https://github.com/pivotal-cf/pcf-pipelines/tree/master/upgrade-tile"
with little modification. Able to update the tile with Apply changes.
Now required to update more then one tile. for example 2 tile. please share me the pipeline and parameter for the same.
Thanks,
Vasanth

I created a pipeline. but getting error.
"fly validate-pipeline --config multi-tile-pipeline.yml
could not resolve template vars: yaml: did not find expected "
groups: []
resource_types:

name: pivnet
type: docker-image
source:
uri: https://hub.docker.com/r/pivotalcf/pivnet-resource
repository: pivotalcf/pivnet-resource
tag: latest-final

resources:

name: pcf-pipelines
type: git
source:
uri: https://github.com/pivotal-cf/pcf-pipelines.git
branch: master

name: pivnet-product
type: pivnet
source:
api_token: xxxxx
product_slug: p-mysql
sort_by: semver

name: pivnet-product1
type: pivnet
source:
api_token: xxxxx
product_slug: p-rabbitmq
sort_by: semver

jobs:

name: upload-tile
max_in_flight: 1
plan:

aggregate:

get: pcf-pipelines
get: pivnet-product
params:
product_slugs: p-mysql
trigger: true

task: upload-tile-and-stemcell
file: pcf-pipelines/tasks/upload-product-and-stemcell/task.yml
params:
IAAS: aws
OPSMAN_DOMAIN_OR_IP_ADDRESS: xxxxx
OPS_MGR_USR: xxxxx
OPS_MGR_PWD: xxxxx
PIVNET_API_TOKEN: xxxxx

name: upload-tile
max_in_flight: 1
plan:

aggregate:

get: pcf-pipelines
get: pivnet-product1
params:
product_slugs: p-rabbitmq
trigger: true

task: upload-tile-and-stemcell
file: pcf-pipelines/tasks/upload-product-and-stemcell/task.yml
params:
IAAS: aws
OPSMAN_DOMAIN_OR_IP_ADDRESS: xxxxx
OPS_MGR_USR: xxxxx
OPS_MGR_PWD: xxxxx
PIVNET_API_TOKEN: xxxxx

Hi @vasanthg1210 please review our repo page for more information about managing updating more than one tile. Thanks.


huntr.dev - Command Injection Fix

https://huntr.dev/app/users/69 has fixed a security vulnerability (Command Injection) :hammer:. 69 has been awarded $25 for fixing the vulnerability through the huntr bug bounty program :dollar:. Think you could fix a vulnerability like this? Get involved at https://huntr.dev/!Q | AVersion Affected | ALLBug Fix | YESOriginal Pull Request | https://github.com/418sec/compass-compile/pull/1

@quaertym - let me know if you have any questions! 🍰


test(Dashboard): add dashboard system tests

Overview:
System tests for dashboard page DCOS-15059. The tests check if the following five items update when adding and deleting a service:

CPU allocation (checks if increases/decreases when service added/removed)
Memory allocation (checks if increases/decreases when service added/removed)
Disk allocation (checks if increases/decreases when service added/removed)
Service name in list (checks if service name appears/disappears in list when service added/removed)
Task count chart (checks if number of tasks increments/decrements when service added/removed)

Testing:
Everyday I usually create an open cluster named "andrew-(random-id)-open". Feel free to use it for testing.
docker run -it --rm --ipc=host \
    -v `pwd`:/dcos-ui \
    -e CLUSTER_URL=$OPEN_CLUSTER \
    mesosphere/dcos-ui dcos-system-test-driver --only "dashboard" \
    ./system-tests/driver-config/dev.sh

Checklist

[x] Did you add a JIRA issue in a commit message or as part of the branch name?
[ ] Did you add new unit tests?
[ ] Did you add new integration tests?
[ ] If this is a regression, did you write a test to catch this in the future?

Feel free to merge as velocity tests pass! ✨

run tests

run unit tests

run integration tests

run integration tests


Should have more catagories & tags.

We should add more catagories like Open source Tools or VS Code Extensions or any tools or modules related to any perticular development tool or IDE.
Also we could add more tags, if any of tag is  not exist it should create new one so we can use it again. Tags will help us to search better and more accurate.

Hy @vinugawade, if the tag or category that you want to add is not available in the dropdown you can create one by entering the category or tag name in the input box

@dishant0406 Thanks for help! I'll do that. Closing this issue now.


internal errors with b2

When filing an issue, please include the following information if
possible as well as a description of the problem.  Make sure you are
using the latest version of rclone.

What is your rclone version (eg output from rclone -V) 1.33

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Mac os el capitan 64 bit

Which cloud storage system are you using? (eg Google Drive)

Backblaze b2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Rclone sync

A log from the command with the -v flag (eg output from rclone -v copy /tmp remote:tmp)

log.zip
When using the rclone sync (I've seen this wiht the rclone copy to) I'll get unable to remove copy internal error. I noticed it today at about 19:00 or maybe a bit earlier. Do I need to run a rclone cleanup on the flder to make sure it does remove these versions or is somethingg broken someware? Sorry the log file is so big.

It looks like this is the same issue as #689 - you have a file called "Icon" with a carriage return on the end.
I'll post a beta when I've done a fix for it.
In the mean time you could add --exclude 'Icon?' to your command line to not sync those files.

Actually I was referring to the section that went
2016/09/14 19:10:36 Robert Jordan/Robert Jordan - 05 - The Fires of Heaven/Jordan, Robert - 05 - The Fires of Heaven/53 Chapter 52_ Choices.mp3: Clearing upload URL because of error: Not enough members for write of 4_zb048f0bb5d84d7865b660a15_f103707d48c08a63b_d20160915_m014628_c000_v0001010_t0017: neighbor_down; neighbor_down (500 internal_error)
2016/09/14 19:10:36 pacer: Rate limited, increasing sleep to 20ms
2016/09/14 19:10:36 pacer: low level retry 1/1 (error Not enough members for write of 4_zb048f0bb5d84d7865b660a15_f103707d48c08a63b_d20160915_m014628_c000_v0001010_t0017: neighbor_down; neighbor_down (500 internal_error))
2016/09/14 19:10:36 Robert Jordan/Robert Jordan - 05 - The Fires of Heaven/Jordan, Robert - 05 - The Fires of Heaven/53 Chapter 52_ Choices.mp3: Failed to copy: Not enough members for write of 4_zb048f0bb5d84d7865b660a15_f103707d48c08a63b_d20160915_m014628_c000_v0001010_t0017: neighbor_down; neighbor_down (500 internal_error)
2016/09/14 19:10:36 Robert Jordan/Robert Jordan - 05 - The Fires of Heaven/Jordan, Robert - 05 - The Fires of Heaven/53 Chapter 52_ Choices.mp3: Removing failed copy
2016/09/14 19:10:36 pacer: Reducing sleep to 10ms
2016/09/14 19:10:36 Robert Jordan/Robert Jordan - 05 - The Fires of Heaven/Jordan, Robert - 05 - The Fires of Heaven/53 Chapter 52_ Choices.mp3: Failed to remove failed copy: failed to delete file: File not present: 3archives/books/Robert Jordan/Robert Jordan - 05 - The Fires of Heaven/Jordan, Robert - 05 - The Fires of Heaven/53 Chapter 52_ Choices.mp3 (400 no_such_file)
2016/09/14 19:10:37 graphic audio/Eileen Wilks/World of the Lupi/6 Blood Magic/6 Blood Magic.rar: Sending chunk 6 length 100663296
Also  the excerpt
2016/09/14 18:51:44 Robert Jordan/Robert Jordan - 06 - Lord of Chaos/Jordan, Robert - 06 - Lord of Chaos/49 Chapter 45_ A Bitter Thought.mp3: Failed to remove failed copy: failed to delete file: File not present: 3archives/books/Robert Jordan/Robert Jordan - 06 - Lord of Chaos/Jordan, Robert - 06 - Lord of Chaos/49 Chapter 45_ A Bitter Thought.mp3 (400 no_such_file)
2016/09/14 18:51:44 auto biographies/Claire McCaskill Plenty Ladylike/Claire McCaskill Plenty Ladylike/Plenty Ladylike Description.txt: Failed to remove failed copy: failed to delete file: File not present: 3archives/books/auto biographies/Claire McCaskill Plenty Ladylike/Claire McCaskill Plenty Ladylike/Plenty Ladylike Description.txt (400 no_such_file)
So why won't rclone  remove them in the destination ? My source is fine, I have these files.

When there is an upload failure rclone deletes the upload so you don't get a partial upload (something I'm going to modify in #559). If it didn't get uploaded then the delete fails.

Ah, Ok.  So I  or anyone else  don't really need to worry about those then?
Thanks.

No need to worry :-)


Footnotes are missing post name in the link to the footnote and the backref

Sorry I'm back 😅
Footnotes point back to the proper anchor, but do not include the post name in the link, so they all go back to the home page. Examples for to and from:

Properly formed the link for each would be:

to -> ${SITE}/{POST_PATH}/${POST}#fn:${REF}
from -> ${SITE}/${POST_PATH}/${POST}#fnref:${REF}

e.g. this: #fn:4
should be this: posts/you-dont-know-gif#fn:4
But both appear to be missing the post part.
Example post with footnotes (can go to the bottom of the page for a direct link to some): https://blog.darrien.dev/posts/you-dont-know-gif/
Currently investigating, I'll post a PR if I figure it out. But just posting in the meantime in case you had any ideas.

Actually I wonder if there's an extra / somewhere. The current anchor syntax should work in its current form too 🤔

Found it :D I'll post a PR


Updated "Search command" + Added logo

@snehasp13 Do we have permission to use the stock art in the logo? What is the license?


Enhance conversion e2e test

What kind of PR is this?:

/kind feature
Make the conversion e2e test works
Minor improvement on TransactionAuthorization, add one more state to block any potential repeated signTransaction call
What this PR does / why we need it:
Which issue(s) does this PR fixes?:

Fixes #
Additional comments?:

#574 is super set


Archive this Application

As the last commit is 11 months ago and I don't think you will develope this application anymore? You could archive it, and no one would lose their time anymore by creating new issues. Thank you

What are people using now?

What alternative app are people using now?

You could use gnome-pomodoro on linux, PomoDoneApp on win10 and macos, or win 11 build in feature. You could also keep using this app here, there are no security issues, but might get incompatible with os stuff and break more and more, as no longer updates. Btw, there are also a ton of websites with a simple pomodoro timer! 😄


First versions of e2e test w/gateway and rest api

Still not integrated with CI since I need to create the gateway docker image yet.
There's still work to do around improving the tests and the error reporting.
The tests should be run with ./scripts/run-e2e.sh with an empty DB.
The tests are written in rust and can be found in e2e-tests directory.
Also for now the way tests are written require lots of boilerplate, plan to get rid of it with macros down the line.

Closing this for cleanup purposes.


Fix #1304: prevent git.refresh when no providers to avoid notification storm

Fixes #1304

/azp run

Seems like there was a transient Docker issue


onOpen doesn't fire when specifying c-search

I'm using my own custom search function, so I use <c-search> to supply my own <input>. onOpen doesn't fire when using c-search.
Example:
https://stackblitz.com/edit/angular-3tcuzj

what about this man ?

Hello?

add event click to select
  static openSelect($event) {     let inputField = <HTMLInputElement>document.querySelector('.dropdown-list:not([hidden]) input');     inputField && !($event.target.tagName == "INPUT") && setTimeout(() => {       inputField.focus()     }, 0)   }
this hack work for me

I have the same issu too,


Resolve includes file name.

Added the file name as a 2nd argument to resolve.
Expanded resolve test with 3rd case that test the name.

bump, really missing this feature

Hey, sorry I didn't see this. Can you explain this pull request some more? I see you wrote "Added the file name as a 2nd argument to resolve." but that's not what's in this change. match[1] is simply the first capture group in the regular expression, which can be anything, not just the file name.
Can you online the use-case you are trying to solve with this pull request?
Also, can you add documentation to the README for the feature that this pull request is adding?

I can't speak for the guy who made the PR, but for me it goes as follows:
I have a folder commands, which contains, well, API commands for the module. I use resolve to wrap them in the same wrapper function. This part works fine. However, when I wanted to add some trace logging in the lines of command "command name" was called with arguments "xyz", I found that there was no beautiful way of doing that. So temporarily I moved the logging into the commands themselves.

Hi @KristjanTammekivi very interesting. I'm not sure if I understood what you are doing, though. Perhaps would you be able to whip up the code you would use pretending like this PR was merged? That may help me understand. Currently this PR doesn't actually add any information you cannot already get at, it's just embedded in the full file path in the first argument.
Can you also speak towards the implementation question as well? The PR says it's adding the file name to the arguments, but it's actually adding match[1], not the file variable. Which one are you expecting?

Since the resolve is run recursively, match[1] is effectively the file name (without the .js extension, not captured by the DEFAULT_FILTER).
And the first argument of resolve only contains whatever it is in the respective module.export variable, so the file path is not embedded there.
I already documented my use case :) forgot to add a reference: Could be useful if resolve includes file name.
Knowing which file are you working with (during resolve) can be extremely useful; in my case is because files have semantic meaning, they carry the desired name of the task so I don't have to repeat it. And for this other guy, I imagine this is what he wants to do (using my code as the example):
var gulp = require('gulp');

require('require-all')({
    dirname: __dirname + '/src/gulp',
    resolve: function(item, name) {
        gulp.task(name, item.deps, function(cb) {
            console.log(`Running task: ${name}...`);
            item.task(cb);
        });
    },
});

Which looks pretty good to me!
I'll amend the README first chance I get :)

Hi @jaimegomez your example isn't very helpful; this module is not a directly walking module; you should not be using the module for side-effects. I don't see you using the return value anywhere.

Your example should be the following:
var gulp = require('gulp');

var tasks = require('require-all')(dirname: __dirname + '/src/gulp);

Object.keys(tasks).forEach(function(name) {
    var item = tasks[name];
    gulp.task(name, item.deps, function(cb) {
        console.log(`Running task: ${name}...`);
        item.task(cb);
    });
});

Or similar.

You're right, I had forgotten about that; tried it a long time ago, before switching to using resolve (and bumping into this issue). Why? I guess it's only because it fits my coding style better, looks "simpler" to me, more straight-forward, and most of the code that I use works in a similar fashion.
I'm sure, based on the previous comments, that @KristjanTammekivi would use it the way I described, though I don't know his motivation.
But I get what you're saying, this can already be accomplished just using the language, and resolve was never intended to be used this way, so it doesn't make sense from that perspective.
If I had to argue in favor of this change, I'd say that it adds flexibility (as we can see in this thread), but I agree that's not necessarily a good thing. Your call!

I'm al for adding flexibility, but adding a dozen ways to do the same things ends up as a bloated API. If someone has the need for the file name or something in the resolve function, we can add it, but I don't see the case for it currently. And this PR doesn't even add the filename, it adds match[1], which is even less useful (though it's hard to judge that without understanding the use-cases).
Maybe if you can use lodash or similar, at least until Object.entries is part of the language?
_.each(require('require-all')(__dirname + '/src/gulp'), function(item, name) {
    gulp.task(name, item.deps, function(cb) {
        console.log(`Running task: ${name}...`);
        item.task(cb);
    });
});

While we're at it, I have a question, what else could we find in the match[1] result? You've mentioned a few times that it's not the file name, but I can't see what you mean; it's already been used as the key value for modules (assuming map is not overridden), which is touted as the file name in the documentation.
But I'm convinced, this change only makes sense if we added each as a function and use it instead, but then again, why not use the tools we already have in the language.
Guess I'll keep using my private repo for now :) the resulting code just looks beautiful in my eyes.

which is touted as the file name in the documentation.

Where does it say that? I would like to correct that if it says somewhere it is the file name.

While we're at it, I have a question, what else could we find in the match[1] result?

It is simply whatever the user specified as the first capture group in the filter option. Yes, it comes from the file name, but it's just a part of the file name, not the actual file name (or even the entire name minus the extension). It's whatever the first capture group captured. I believe PR #31 is even trying to make that be the result of some function, making it more likely to not be the file's name.

this change only makes sense if we added each as a function and use it instead, but then again, why not use the tools we already have in the language.

I'm not sure what you mean by this. Can you elaborate?

Oh right, you can change the filter, I was missing that completely.
Line 48 (redacted): "...The map function is called on both file and directory names...".
As for adding each as a function, I was thinking on this:
var gulp = require('gulp');

require('require-all')({
    dirname: __dirname + '/src/gulp',
    each: function(key, value) {
        // exactly what I want
    },
});

It'd be a convenience method, I recon there are other ways to accomplish the same thing (you have mentioned three already). But it's also not unheard of 😄

Yep:
_.each(require('require-all')(__dirname + '/src/gulp'), function(item, name) {
    gulp.task(name, item.deps, function(cb) {
        console.log(`Running task: ${name}...`);
        item.task(cb);
    });
});


Auto-complete registerCompletionItemProvider is not working when language is yaml.

I am  testing vscode 1.43.1 example from vscode-extension-sample: completions and have some issues.
When I set registerCompletionItemProvider language option to "yaml", the auto complete is not working anymore until I uninstall the yaml extension.
It worked on vscode 1.42.1 with yaml extension. Does anybody encounter the same issue?

VSCode Version: 1.43.1
OS Version: Mac os mojave 10.14.5
YAML Extension Version: 0.7.2

Steps to Reproduce:

Install YAML extension.
Using vscode-extension-sample: completions and modify the language  option to "yaml".

the auto complete is not working anymore until I uninstall the yaml extension.

What does that mean? Isnt the provider called anymore or aren't its results not showing anymore?

When I run with debug mode that show the provider called successful.
But Its results not showing anymore.

But Its results not showing anymore.

In what scenarios? When doing ctrl+space on an empty line?

But Its results not showing anymore.

In what scenarios? When doing ctrl+space on an empty line?

I am testing this case from vscode-extension-sample: completions and modify the language to yaml.
code here:
https://github.com/microsoft/vscode-extension-samples/blob/master/completions-sample/src/extension.ts#L49
There trigger character is ..
I input console. In the empty yaml file and there has no item to show.

When I disable YAML extension, it seems work.

When I disable YAML extension, it seems work.

what do you expect me to do? this seems to be something with yaml, maybe its word definition?

When I disable YAML extension, it seems work.

what do you expect me to do? this seems to be something with yaml, maybe its word definition?

I expect the yaml auto-complete results will show in vscode version 1.43.1 with yaml extension. Because it worked on vscode 1.42.1 with yaml extension. After I upgrade vscode version to 1.43.1 then it did not work.

Ok - then please file an issue against the yaml extension

@jrieken
Let me clear the question:
In vscode 1.42.1, I develop an auto-complete extension sample with the official guide vscode-extension-sample: completions which make sure that can work on YAML.
https://github.com/microsoft/vscode-extension-samples/blob/master/completions-sample/src/extension.ts#L49
But after upgrading to 1.43.1, the sample extension is not working anymore on YAML file unless I disable the YAML Extension and in the mean time the sample code is not modified. On the other hands, it's still working on TEXT file.
I try to debug the sample extension code but everything seems to go well as usual except the CompletionList Popup does not show up.
I thought that it suppose to relate to the language identifiers determination rather than YAML Extension.
Do you have any further idea about this issue?


Introduce backup code authenticator

Purpose
Introduce backup code authenticator to IS server.
Related PRs
Merge this PR after merging follwoing PRs

https://github.com/wso2-extensions/identity-outbound-auth-backup-code/pull/1
https://github.com/wso2/carbon-identity-framework/pull/4009
https://github.com/wso2/carbon-identity-framework/pull/4010
https://github.com/wso2/identity-api-user/pull/146
https://github.com/wso2-extensions/identity-inbound-provisioning-scim2/pull/416

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/2269094003

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/2269094003
Status: success


Moving Show Intercom Feature to Sauce

Explanation of Bugfix/Feature/Enhancement:
title
Recommended Release Notes:
none

Just noticed that @falkencreative has already submitted a PR to saucify these feature.

@dbaldon - Josh's pull request is more recent, whereas I still have fixes to do due to confusion over shouldInvoke(). His can be added, and I'll close my pull request.


Vector3d.AngleTo method misleading

var angle = vector.Coord.Xaxis.AngleTo(vector);
Not sure if I'm explaining this right.
The above method returns the angle between the x axis and the vector, but the method name implies that the result should be the directional angle from the x-axis to the vector (hence "AngleTo"). The difference is that the method currently returns the absolute value of the angle without direction, meaning vectors (1,-1,0) and (1,1,0) both return pi/4 and there's no indication of rotation from v1 to v2. I believe standard convention is that a positive angle implies CCW rotation and a negative angle implies CW rotation.

Hi. This standard convention works in 2D space. In 3D space you also need to specify axis of rotation, and this is what Rotation object do. The name "AngleTo" may be misleading, but probably it is to late to rename it.


'NoneType' object has no attribute 'group'

Hi there,
I'm trying Petrel with wordcount sample but find this problem:
$petrel submit --config topology.yaml --logdir pwd
'NoneType' object has no attribute 'group'
Traceback (most recent call last):
File "/home/jonas/.local/lib/python2.7/site-packages/petrel-<IP_ADDRESS>.3-py2.7.egg/petrel/cmdline.py", line 111, in main
func(**args.dict)
File "/home/jonas/.local/lib/python2.7/site-packages/petrel-<IP_ADDRESS>.3-py2.7.egg/petrel/cmdline.py", line 32, in submit
sourcejar = get_sourcejar()
File "/home/jonas/.local/lib/python2.7/site-packages/petrel-<IP_ADDRESS>.3-py2.7.egg/petrel/cmdline.py", line 23, in get_sourcejar
storm_version = get_storm_version()
File "/home/jonas/.local/lib/python2.7/site-packages/petrel-<IP_ADDRESS>.3-py2.7.egg/petrel/cmdline.py", line 19, in get_storm_version
return m.group(0)
AttributeError: 'NoneType' object has no attribute 'group'
I also try:
$./buildandrun --config topology.yaml
~/.local/lib/python2.7/site-packages ~/wordcount
Traceback (most recent call last):
File "/home/jonas/Downloads/Petrel-master/petrel/setup.py", line 66, in
build_petrel()
File "/home/jonas/Downloads/Petrel-master/petrel/setup.py", line 29, in build_petrel
version_string, version_number = get_storm_version()
File "/home/jonas/Downloads/Petrel-master/petrel/setup.py", line 21, in get_storm_version
return m.group(0), m.group(1)
AttributeError: 'NoneType' object has no attribute 'group'
Error on or near line 20; exiting with status 1
I'd tried with virtualenv:
$source petrel/bin/activate
and without
$easy_install --install-dir /home/username/.local/lib/python2.7/site-packages  petrel==<IP_ADDRESS>.3
Thanks in advance

What version of Storm are you running? The Storm version should match the first 3 digits of the Petrel version.

I'm using 0.10.0. So this is the problem! Is there a 0.10 version or 1.0!?
Thanks in advance.

Are you aware of Streamparse?
—r

On 15 Apr 2016, at 21:39, CogumelosMaravilha<EMAIL_ADDRESS>wrote:
I'm using 0.10.0. So this is the problem! Is there a 0.10 version or 1.0!?
Thanks in advance.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub https://github.com/AirSage/Petrel/issues/72#issuecomment-210636922

Russ Ferriday -- Software Product Architect, Developer, Mentor
Founder & CTO Topia Systems Ltd.
<EMAIL_ADDRESS>--  +44 7429 518822

There is a PR which is intended to add 0.10 support. It needs to be redone because it has a lot of unnecessary (even incorrect) changes. I will try and work on it some time, or I'd be happy to accept help. :-)

Yes i'm aware of Streamparse. But having a 30k Python code a 15M jar file, perhaps there must be a better way!
Thanks.

I'll work on 0.10 support next week. I think it's just a few hours work at most. In the meantime, can you use Storm 0.9.4?

Hi,
I already downgrade my Apache Strom. Let's see what happens. If i can live with this version!
Thanks a lot for you work Barry.

I pushed some fixes to the repo tonight which should make Petrel work with Storm 0.10.0. Can you test and confirm? Note I have not pushed these changes to pypi yet.

Hello Barry,
First off all congrats for your great book.
It's a great book about Petrel, Storm, python important packages, and cool stuff like MongoDB and Redis, code profiling and debugging. Great examples integration with Twitter, etc.
And better of all the created jar files are so small! And we can debugging python code inside Strom!
In my honest opinion with better advertisement, like parse.ly does, talking in PyCon, etc. your project become the default highway to integrate Python and Strom.
Be aware in the code chapter 4/twitter_stream.py you forgot to remove your keys!
Now I'm going to test you new version.
Just a thought, why not a pip package!?
Thanks.

I have a big problem with my box. Is the motherboard or the CPU.
On 19-04-2016 03:06, Barry Hart wrote:

I pushed some fixes to the repo tonight which should make Petrel work
with Storm 0.10.0. Can you test and confirm? Note I have not pushed
these changes to pypi yet.
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/AirSage/Petrel/issues/72#issuecomment-211688167

Oops, I should fix those Twitter keys.
I can create a pip package, I was just wondering if you could test it before I upload to pypi (pip). To create a package yourself, all you have to do is "python setup.py sdist".
Thanks for the kind words about the book and Petrel! I should definitely do more marketing. I'm kind of shy, and I'm not using Storm at my current job, so it's kind of been on the back burner. The book was an unplanned project that came about because the publisher already wanted to do a book like that, and they found me.

I'm using a new box.
Zookeeper, Kafka and Storm are ok.
After build and pip your package I'm getting:
(petrel) jonas@jonas:/$ petrel
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/jonas/petrel/lib/python2.7/site-packages/petrel/cmdline.py", line 12, in
from .package import build_jar
File "/home/jonas/petrel/local/lib/python2.7/site-packages/petrel/package.py", line 13, in
from topologybuilder import TopologyBuilder
File "/home/jonas/petrel/local/lib/python2.7/site-packages/petrel/topologybuilder.py", line 6, in
from petrel.generated.storm.ttypes import ComponentCommon, Grouping, NullStruct, GlobalStreamId
ImportError: No module named generated.storm.ttypes
But perhaps is something wrong with my new environment!

It looks like the step that generates Python wrappers around the Storm Thrift objects failed. I'm not sure why.
I'll push a build to pypi tonight to save us the trouble of diagnosing this.

I uploaded version <IP_ADDRESS>.3 to pypi. Give it a try...

Looks fine to me!
Thanks a lot.


Computing the parameter lambda from a MolecularHamiltonian

I'm trying to understand how to compute the lambda parameter of a Molecular Hamiltonian. For example, let us say I have the following Molecular Hamiltonian data
self.molecule_data = openfermion.chem.MolecularData(self.molecule_geometry, self.basis, charge = charge, multiplicity = 1)

One way I had thought of calculating the lambda parameter is as the sum of absolute values of the amplitudes in the Jordan-Wigner operator, eg
self.molecule_psi4 = openfermionpyscf.run_pyscf(self.molecule_data,run_scf=True)
molecular_hamiltonian = self.molecule_data.get_molecular_hamiltonian()
fermion_operator = openfermion.transforms.get_fermion_operator(molecular_hamiltonian)
JW_op = openfermion.transforms.jordan_wigner(fermion_operator)
l = abs(np.array(list(JW_op.terms.values())))
self.lambda_value = sum(l)

On the other hand though, one may use the amplitudes of the Hamiltonian, similarly as suggested in Qubitization of Arbitrary Basis Quantum Chemistry Leveraging Sparsity and Low Rank Factorization, in which case lambda could be computed as
H1 = molecular_hamiltonian.one_body_tensor
eri = molecular_hamiltonian.two_body_tensor
T = H1 - 0.5 * np.einsum("pqqs->ps", eri) + np.einsum("pqrr->pq", eri)
lambda_V = 0.5 * np.sum(np.abs(eri))
lambda_T = np.sum(np.abs(T))
lambda =  lambda_T + lambda_V

I think these two values should be then the same but with some examples, one can see that they are not exactly the same, the former being smaller than the latter by a few percentage points depending on the molecule. Is there anything wrong with one of those forms of calculating lambda or does the reordering of terms from physics -> chemistry affect the value of the lambda and that is why I don't get the same result?
Thanks in advance!

By the way, a third way I can think of is
lambda_0 = np.sum(np.abs(H1))+0.5*np.sum(np.abs(eri))

To give an example, with water the values I get are
self.lambda_value = 256
lambda = 261
lambda_0 = 268

Indeed, it makes sense that these are all slightly different values. The correct lambda value for a qubit Hamiltonian would be the first one (but you could substract the constant if you want). In https://arxiv.org/abs/2103.14753 appendix C we elaborate on why these values are slightly different, it has to do with the reordering of integrals when writing the Hamiltonian as a weighted sum of Pauli strings (https://quantum-journal.org/papers/q-2019-12-02-208/ actually makes a small mistake in that regard). The QubitOperator in Openfermion actually has an easy way to compute the 1-norm/lambda value, as JW_op.induced_norm(). If you want to compute the lambda values without doing an JW transformation - as these can be costly - you can use the code in #725

@Emieeel I don't think the lambda value in our paper is "incorrect" (if that is what you mean to claim then can you describe this a bit better?). If I understand correctly, you are pointing out there is an additional symmetry in the Hamiltonian that can be taken advantage of. In particular, I think what you are pointing out is equivalent to the following. If you write it out in the normally ordered form, then you can see that there should be antisymmetry swapping p,q or r,s.  That is,
V_{pqrs} a'_p a'r a_q a_s + V{rqps} a'r a'p a_q a_s = V{pqrs} a'p a'r a_q a_s - V{rqps} a'p a'r a_q a_s = (V{pqrs} - V{rqps}) a'p a'r a_q a_s
What this means is that if the LCU can take advantage of that symmetry the effective lambda from that term is |V{pqrs} - V{rqps}| rather than |V{pqrs}| + |V{rqps}|.  We only get that lambda value if we take advantage of it.
So we're not taking advantage of that particular symmetry and as a result of lambda is slightly higher. It might be a good idea to take advantage of this. But that's different from saying the analysis is incorrect, right?

Hey @Emieeel thanks a lot for your answer. So if I get it right, the reason why self.lambda_value < lambda above, is due to equation C18 in your article, |g_{pqrs} - g_{psrq}| < |g_{pqrs}| + |g_{psrq}|due to the grouping of terms @babbush is mentioning above (thanks also Ryan!).
Do we also know why lambda_0 happens to be even larger than lambda? (sorry for the notation)

@babbush Hi Ryan, thank you for pointing this out. Indeed our main point here was that one could take advantage of this symmetry of the Hamiltonian, such that you get the exact same lambda value for a qubit Hamiltonian (obtained for example by a JW transformation in OpenFermion) written in terms of unique Pauli operators -- for which this symmetry is not apparent -- but I agree that this not incorrect. However as a small sidenote, I believe that the THC paper https://arxiv.org/pdf/2011.03494.pdf pointed out a small mistake concerning a factor 1/2 on page 37 (sorry if I'm a bit pedantic):

Note that these λ definitions differ from those given in [9]. At first glance λV appears to be a factor of 8 smaller.
The first reason for this is because we deviated from the convention of absorbing the factor of 1/2 into V (since
this is just a difference in convention, it is consistent with prior work). However, there is roughly another factor
of 4 difference that comes in because the work of [9] accidentally left out a factor of 1/2 in Eq. (A3). This leads
to a λV that is reduced by a factor of 4.

@PabloAMC Indeed, but I'm not sure if there is such a simple reason that lambda_0 is larger than lambda. It makes sense that they're different though, since you rewrite your Hamiltonian (a'_p a'_r a_s a_q = - a'_p a_s a'r a_q + delta_rs a'p a_q). The lambda value is dependent on how you write the Hamiltonian. Also, beware of the openfermion convention: h{pqrs} = g{psqr}.

@Emieeel yep that is correct, I agree with everything you write in your most recent comment.


Visual Studio Code tries to close my workspace when I open a folder

Hi,
So when I have an opened project, and I try to open a new folder (File -> Open File…), Visual Studio Code tries to close my opened files.

This looks like this is intentional, but it doesn't make sense.
I ask to OPEN a folder, why does VSCode force me to close my current workspace to do that?
The only workaround I've found is to first create a new window (File -> New Window), and then open a new folder inside it. That's too many things to do for such a simple task. 😆

Could you describe what you'd like to happen with your existing files and maybe I could show you a way to make that work?

Sorry, I thought I already replied, but I didn't.
I don't want anything to happen to my existing files.
And I don't want to be asked what I want to happen to my existing files, since my intent to simply open a new folder (in a new window), without any side-effect whatsoever on existing files.

@JacksonKearl yes, I do get the desired behaviour. It opens a new window when I open a folder, without asking any questions.
Switching window.openFoldersInNewWindow to on solves my issue.
:)