Getting

➜ winixctl login
You need to signup for a Winix account & associate your device in the phone app before using this.
Username (email<EMAIL_ADDRESS>Password: <not shown>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/jose/backends/__init__.py", line 2, in <module>
    from jose.backends.cryptography_backend import get_random_bytes  # noqa: F401
  File "/usr/local/lib/python3.10/site-packages/jose/backends/cryptography_backend.py", line 4, in <module>
    from cryptography.exceptions import InvalidSignature, InvalidTag
ModuleNotFoundError: No module named 'cryptography'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/winixctl", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/winix/cmd.py", line 326, in main
    cls(args, config=Configuration("~/.config/winix/config.json")).execute()
  File "/usr/local/lib/python3.10/site-packages/winix/cmd.py", line 111, in execute
    return self._login()
  File "/usr/local/lib/python3.10/site-packages/winix/cmd.py", line 120, in _login
    self.config.cognito = login(username, password)
  File "/usr/local/lib/python3.10/site-packages/winix/auth.py", line 25, in login
    from warrant_lite import WarrantLite
  File "/usr/local/lib/python3.10/site-packages/warrant_lite/__init__.py", line 16, in <module>
    from jose import jwt, JWTError
  File "/usr/local/lib/python3.10/site-packages/jose/jwt.py", line 6, in <module>
    from jose import jws
  File "/usr/local/lib/python3.10/site-packages/jose/jws.py", line 5, in <module>
    from jose import jwk
  File "/usr/local/lib/python3.10/site-packages/jose/jwk.py", line 1, in <module>
    from jose.backends.base import Key
  File "/usr/local/lib/python3.10/site-packages/jose/backends/__init__.py", line 5, in <module>
    from jose.backends.pycrypto_backend import get_random_bytes  # noqa: F401
  File "/usr/local/lib/python3.10/site-packages/jose/backends/pycrypto_backend.py", line 7, in <module>
    from Crypto.PublicKey import RSA
  File "/usr/local/lib/python3.10/site-packages/Crypto/PublicKey/RSA.py", line 137
    e, d, n, p, q, u = [self._key[comp] for comp in 'e', 'd', 'n', 'p', 'q', 'u']
                                                       ^
SyntaxError: invalid syntax

Potential causes:

My Winix email login uses a + -- a nice feature offered by Google G-Suite so I can see if my registered email address ends up on some spam list
My Winix password is very complicated, so that might also be getting things in trouble.  Perhaps it needs to be surrounded with quotes?

I have this same issue, even trying something like a normal password im getting this too

I got around that error doing pip install cryptography. but now get this one
requests.exceptions.SSLError: HTTPSConnectionPool(host='us.mobile.winix-iot.com', port=443): Max retries exceeded with url: /registerUser (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))

seems to be certificate related.

ended up using this for now to get my device id. https://github.com/banzalik/homebridge-winix-c545/wiki/How-to-get-device-ID

Yeah needed to pip install —upgrade cryptography


Configuration: make sure the shared_sources table doesn't contain empty elements [1.1.1 & 1.1.0]

Fixes #7634

Master ? I tested there and didn't get the same issue.
I saw that 1.1.0 may have a similar issue, so cherry picking there.

@levitte Master ? I tested there and didn't get the same issue.

Perhaps I am mistaken, I did not look recently.

Yeah, master is different, and does not exhibit the same symptoms in my tests.

Please go ahead and merge.

Thanks!
Merged
1.1.1:
0c594ccc29f6ba241627f436ba3d05fc400d1066 Configuration: make sure the shared_sources table doesn't contain empty elements
1.1.0:
013aefc5f03c95f5366cc854ffb02b0df49af90d Configuration: make sure the shared_sources table doesn't contain empty elements


memory leak on nginx reload

RAM usage constantly grows on nginx -s reload
Having modsecurity rules loaded (even with modsecurity off) causes RAM usage to grow with each nginx -s reload and ultimately  leads nginx to stuck with messages like:
Logs and dumps (/var/log/nginx/error.log)
Output of:
2020/08/06 20:00:20 [alert] 1962#1962: fork() failed while spawning "worker process" (12: Cannot allocate memory)
2020/08/06 20:00:20 [alert] 1962#1962: sendmsg() failed (9: Bad file descriptor)
2020/08/06 20:00:20 [alert] 1962#1962: fork() failed while spawning "worker process" (12: Cannot allocate memory)
2020/08/06 20:00:20 [alert] 1962#1962: sendmsg() failed (9: Bad file descriptor)
2020/08/06 20:00:20 [alert] 1962#1962: fork() failed while spawning "cache manager process" (12: Cannot allocate memory)
2020/08/06 20:00:20 [alert] 1962#1962: sendmsg() failed (9: Bad file descriptor)
To Reproduce

Configure nginx to load rules:
/etc/nginx/nginx.conf

http {
...
   modsecurity off;
   modsecurity_rules_file /etc/nginx/modsec/main.conf;
..
}

Restart Nginx and check rules were loaded (/var/log/nginx/error.log):

2020/08/06 08:57:13 [notice] 13627#13627: ModSecurity-nginx v1.0.1 (rules loaded inline/local/remote: 0/911/0)

Generate load:

./nikto.pl -h https://your-site-name

Run several 'nginx -s reload' (with 3-4 minutes interval) and check RAM consumption with free -m command before and after nginx reload:

# free -m
              total        used        free      shared  buff/cache   available
Mem:           3951         433        2122          30        1395        3136
Swap:          2043          49        1994

# nginx -s reload

# free -m
              total        used        free      shared  buff/cache   available
Mem:           3951         451        2103          30        1395        3117
Swap:          2043          49        1994

# nginx -s reload

# free -m
              total        used        free      shared  buff/cache   available
Mem:           3951         464        2083          30        1404        3104
Swap:          2043          49        1994

# nginx -s reload

# free -m
              total        used        free      shared  buff/cache   available
Mem:           3951         481        2051          30        1417        3086
Swap:          2043          49        1994

.....

# free -m
              total        used        free      shared  buff/cache   available
Mem:           3951         901        1534          30        1515        2666
Swap:

Expected behavior
'RAM used' should not steadily grow and should stay around the same level as it does for example without modsecurity rules loaded (in which case 'ram used' stays about 300 MB)
Server

ModSecurity v3 master - https://github.com/SpiderLabs/ModSecurity/commit/51d06d7a8edc7a400782c06e24ebe71834736f77  with nginx-connector v1.0.1
WebServer:  nginx-1.18.0
OS : Ubuntu 16.04

Rule Set:

https://github.com/coreruleset/coreruleset/archive/v3.3.0.tar.gz

Additional context
The same happens with modsecurity on in server's context.
Using SecResponseBodyAccess Off in modsecurity.conf

Same issue here. I have disabled modsec but i doubt this will be fixed...

Hi,
We are working on a lot regarding rules reload at: v3.1-experimetal. Is this issues happens on v3.0.4 or only in v3/master?

Hi,  zimmerle, you are right - can't reproduce this with v3.0.4.

I'm still not able to resolve this. As nginx works with the ModSecurity  + CRS ruleset, the memory size grows and grows until Nginx crashes. I've hacked around it by forcing nginx to restart as soon as it crashes, but that's a really bad fix. I tried to use the experimental version, however that didn't work out (the vhost on which ModSecurity was not working with the following error)[alert] 7955#0: worker process 9835 exited on signal 11 (core dumped) ...but thats for another topic. Any idea how I can get nginx and ModSecurity to behave?
nginx version: nginx/1.18.0
built by gcc 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)
ModSecurity-nginx v1.0.1
ModSecurity v3/master

I'm still not able to resolve this. As nginx works with the ModSecurity + CRS ruleset, the memory size grows and grows until Nginx crashes. I've hacked around it by forcing nginx to restart as soon as it crashes, but that's a really bad fix. I tried to use the experimental version, however that didn't work out (the vhost on which ModSecurity was not working with the following error)[alert] 7955#0: worker process 9835 exited on signal 11 (core dumped) ...but thats for another topic. Any idea how I can get nginx and ModSecurity to behave?
nginx version: nginx/1.18.0
built by gcc 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)
ModSecurity-nginx v1.0.1
ModSecurity v3/master

hi,
in my case, the issue, i think it was because i was loading the rules on each server directive which means many times as the number of server directives. Moving the loding of the rules on the main nginx.conf file fixed 2 issues: 1)The memory leak one which is the current issue and also 2)the slow start of nginx

I'm still not able to resolve this. As nginx works with the ModSecurity + CRS ruleset, the memory size grows and grows until Nginx crashes. I've hacked around it by forcing nginx to restart as soon as it crashes, but that's a really bad fix. I tried to use the experimental version, however that didn't work out (the vhost on which ModSecurity was not working with the following error)[alert] 7955#0: worker process 9835 exited on signal 11 (core dumped) ...but thats for another topic. Any idea how I can get nginx and ModSecurity to behave?
nginx version: nginx/1.18.0
built by gcc 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)
ModSecurity-nginx v1.0.1
ModSecurity v3/master

hi,
in my case, the issue, i think it was because i was loading the rules on each server directive which means many times as the number of server directives. Moving the loding of the rules on the main nginx.conf file fixed 2 issues: 1)The memory leak one which is the current issue and also 2)the slow start of nginx

Unfortunately each vhost is running a different application, and the ruleset is fine tuned for each one. So loading the rules in the nginx.conf isn't something that's possible. Is the 3.0.4 or v3/master the latest stable version?

oh i see.  That's a big issue.
If i'm not wrong, i read some times ago that this issue with the loading of the rules is being fixed but it will take a lot of time...
btw i always use v3/master

The last

I'm still not able to resolve this. As nginx works with the ModSecurity + CRS ruleset, the memory size grows and grows until Nginx crashes. I've hacked around it by forcing nginx to restart as soon as it crashes, but that's a really bad fix. I tried to use the experimental version, however that didn't work out (the vhost on which ModSecurity was not working with the following error)[alert] 7955#0: worker process 9835 exited on signal 11 (core dumped) ...but thats for another topic. Any idea how I can get nginx and ModSecurity to behave?
nginx version: nginx/1.18.0
built by gcc 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)
ModSecurity-nginx v1.0.1
ModSecurity v3/master

hi,
in my case, the issue, i think it was because i was loading the rules on each server directive which means many times as the number of server directives. Moving the loding of the rules on the main nginx.conf file fixed 2 issues: 1)The memory leak one which is the current issue and also 2)the slow start of nginx

Unfortunately each vhost is running a different application, and the ruleset is fine tuned for each one. So loading the rules in the nginx.conf isn't something that's possible. Is the 3.0.4 or v3/master the latest stable version?

The last released version is 3.0.4.
The experimental branch is the 3.1-experimental

oh i see. That's a big issue.
If i'm not wrong, i read some times ago that this issue with the loading of the rules is being fixed but it will take a lot of time...
btw i always use v3/master

did you had the chance to test this particular issue in 3.1-experimental?

oh i see. That's a big issue.
If i'm not wrong, i read some times ago that this issue with the loading of the rules is being fixed but it will take a lot of time...
btw i always use v3/master

did you had the chance to test this particular issue in 3.1-experimental?

Nope. I can test it but it will take same take because it is not very deterministic the appeareance of the memory leak bug.
The slow start on the other hand can be tested. Will try tomorrow and let you know.

oh i see. That's a big issue.
If i'm not wrong, i read some times ago that this issue with the loading of the rules is being fixed but it will take a lot of time...
btw i always use v3/master

did you had the chance to test this particular issue in 3.1-experimental?

Nope. I can test it but it will take same take because it is not very deterministic the appeareance of the memory leak bug.
The slow start on the other hand can be tested. Will try tomorrow and let you know.

thank you.

hi zimmerle,
i cannot compile that branch...
cd /root
rm -rdf /root/ModSecurity/
sudo git clone --depth 1 -b v3/dev/3.1-experimental --single-branch https://github.com/SpiderLabs/ModSecurity
cd ModSecurity
sudo git submodule init
sudo git submodule update
sudo ./build.sh
sudo ./configure --with-maxmind=no #--with-maxmind=no sepse jepte problem ne update
sudo make #here the errors

Can you confirm the last commit hash in your branch?

The following is what git log says, so seems the last one..

[root@ReverseProxy ModSecurity]# git log
commit baf189938e9edf6e2f71142f69d58e021d49cac1
Author: Felipe Zimmerle<EMAIL_ADDRESS>Date:   Mon Oct 19 12:33:59 2020 -0300
Having RuleWithActionsProperties()

[root@ReverseProxy ModSecurity]#

This is another nginx installation with ModSec and the same issue even here. Just tried on another machine to be sure that is not environment issue.

I am going to verify and get back to you. thanks

I've tried everything, even boosting the RAM on my instance, however it is still crashing at times. Will using another operating system help? I have a similar setup running on Debian 10 (not as many vhosts though), that has not yet crashed.

@hazardousmonk  Have you had a chance to test v3/dev/3.1-experimental ?

ping.

hi, its for me the ping?

btw, does not compile again

We are also experiencing a memory leak on reload signal. It seems that the leak started manifesting when we enabled SecAuditLog /dev/stdout in our rule set (however, we have not confirmed this yet). On reload, the memory of the master process consistently grows about 6 MB.
The leak is definitely present in the latest v3/master branch (afefda53c69bb17e2ec16d24dd6fbc2c8fd7d063) but it seems that versionv3.0.4 is fine. By bisecting the changes I identified commit 7a48245aed517c5cba0455b5d4e99cdaea14129e as the culprit. I tried to track the leak to a specific call-stack with the memleak script (part of the BCC tools), however it was not very helpful (even for debug builds with optimizations disabled, may be limitation of the script):
1153930 bytes in 16780 allocations from stack
		calloc+0x35 [ld-musl-x86_64.so.1]
		[unknown]
1268410 bytes in 18320 allocations from stack
		calloc+0x35 [ld-musl-x86_64.so.1]
		[unknown]
1536770 bytes in 22510 allocations from stack
		calloc+0x35 [ld-musl-x86_64.so.1]
		[unknown]
1914850 bytes in 30490 allocations from stack
		calloc+0x35 [ld-musl-x86_64.so.1]
		[unknown]
2177110 bytes in 28440 allocations from stack
		calloc+0x35 [ld-musl-x86_64.so.1]
		[unknown]
3771110 bytes in 54810 allocations from stack
		calloc+0x35 [ld-musl-x86_64.so.1]
		[unknown]
4531060 bytes in 62420 allocations from stack
		calloc+0x35 [ld-musl-x86_64.so.1]
		[unknown]
4897110 bytes in 72520 allocations from stack
		calloc+0x35 [ld-musl-x86_64.so.1]
		[unknown]
5016560 bytes in 156793 allocations from stack
		calloc+0x35 [ld-musl-x86_64.so.1]
5531980 bytes in 31644 allocations from stack
		operator new(unsigned long)+0x15 [libstdc++.so.6.0.28]

I am going to run this through Valgrind and gdb to find out more soon. HTH.

@hazardousmonk Have you had a chance to test v3/dev/3.1-experimental ?

Sorry for the delayed response. I built it, however it crashed on launch - it was a production server down for maintenance so I had to revert back to safer territory. I will try this again and keep you posted. Probably Nginx 1.19 and the Dev version. I'm just curious as to how everyone is dealing with this so far? I've hacked a quick script to restart nginx the moment it sees those evil lines in the error log.

Hi @hazardousmonk  thank you for have it tested. You could have it compiled and just changed library file or export a different LD_LIBRARY_PATH together with the nginx .so addon.
At this phase of the development I would prefer if you don't have it running headless. Specially if it not starting properly. Do you happens to have the logging related to the event?

I actually compiled it into nginx rather than building a dynamic module - I will try to load it as a dynamic module. I'm not quite sure how to run it in non-headless mode. If you could point me in the right direction I'll be more than happy to help out on this and send you the full verbose logs.
I can also confirm that v3.0.4 is stable and does not have this condition.

As mentioned on #2376 we no longer have leaks on the reloads. That is valid for 3.1-experimental.

As mentioned on #2376 we no longer have leaks on the reloads. That is valid for 3.1-experimental.

I have the same problem. How can I fix it?

I have the same problem. How can I fix it?

Is anyone there?

Is anyone there?

I have the same problem. How can I fix it? The impact of this problem is very serious. I hope that the government will pay close attention to the repair and repair version 3.0.4 as soon as possible

(a) Try to restart instead of reload; (b) Experiment the master branch and provide feedback; (c) experiment 3.1 and provide feedback.

I have the same problem. How can I fix it? The impact of this problem is very serious. I hope that the government will pay close attention to the repair and repair version 3.0.4 as soon as possible

(a) Try to restart instead of reload; (b) Experiment the master branch and provide feedback; (c) experiment 3.1 and provide feedback.

Is anyone there?

?

Is anyone there?

?

＠zimmerle Has the master branch been fixed the memory leak?

＠zimmerle Has the master branch been fixed the memory leak?

@zimmerle hello admin. I used the one you specified (3.1-experimental) but I noticed the problem still occurs if you need the environment to test, I'll give it to you. this issues not fix in 3.1-experimental. thank you!

@zimmerle   It often leads memory leak on nginx reload for branch 3.1-experimental
Describe the bug
We publish more frequently every day, and often reload nginx，every few days nginx has a memory leak,How to fix it?
Logs and dumps
Output of:
2021/01/21 11:04:08 [alert] 25264#25264: fork() failed while spawning "worker process" (12: Cannot allocate memory)
2021/01/21 11:04:08 [alert] 25264#25264: sendmsg() failed (9: Bad file descriptor)
2021/01/21 11:04:08 [alert] 25264#25264: sendmsg() failed (9: Bad file descriptor)
2021/01/21 11:04:08 [alert] 25264#25264: sendmsg() failed (9: Bad file descriptor)

@zimmerle It often leads memory leak on nginx reload using modsecurity branch 3.1-experimental and master,I hope that the official will fix this problem as soon as possible, which has a serious impact on the production environment.
And There is also why this problem #2381 was closed without being resolved!!!
I suggest that you officially do a test yourself. First, create a lot of virtual hosts, each virtual host quotes the modsecurity rules separately, and then constantly reload, the memory leak should be reproduced!
Describe the bug
We publish more frequently every day, and often reload nginx，every few days nginx has a memory leak,How to fix it?
Logs and dumps

I feel that this kind of problem has been going on for a long time, and I hope that the official will seize the time to fix it. This is a fatal problem and the priority should be the highest.

+1
Server goes down every two weeks or so until I restart nginx. Been doing this for half a year now.

For the past six months, I've been waiting for 3.0.10 to become an official release so it comes down the regular update channels on my server (while patiently restarting a crashed nginx about every other week).
Should I still hold my breath, or go ahead and compile the experimental version?

@GutHib what version are you currently running? did you ever tried v3/master?

I am running 3.0.4. on a server with Directadmin installed. Like with most control panels, by default you get to install all stable releases that come down through the regular update channels. I'm aware that I can go ahead and slap 3.1 on my server manually, but I don't want to make a mess of things. (Directadmin has its own install paths, and I don't want to end up with two competing versions or break the regular updates.) I understand that my concerns may be unfounded, but weirder things have happened.
So basically I'm just trying to figure out my options. If manual installation is my only choice, I will give it a try. Thanks!

Anyone?

Hi there,
First of all thanks a lot for the project, we use it to protect several resources in our company and we really appreciate the work you are doing, keep it up!
I've compiled the library in v3/master, v3/dev/3.1 and v3/dev/3.1-experimental and I see memory leaks in v3/master and v3/dev/3.1 once I reload nginx, I don´t see them in v3/dev/3.1-experimental as expected since the last commit there is with the fix. The size of the leaks is directly proportional on the amount of rules you have loaded.
We would use the experimental branch, however we are using owasp crs as well and their rules depend on ctl:forceRequestBodyVariable which is not supported in the experimental branch, so I was wondering how much effort it would take to port the fix on the experimental branch to a more up to date branch like master or dev/3.1.
Kind regards

I'm not seeing the 'nginx -s reload' memory leak fixed using the v3/dev/3.1-experimental branch, checking either using /proc//smaps directly or using valgrind.  I am running nginx single process in foreground:
master_process off;
daemon off;
worker_processes    1;
The memory leak is in modsecurity code called from the parser, the two worst offenders are:
nginx version 1.21.1
modsec nginx-connector version 1.0.2
modsecurity version v3/dev/3.1-experimental
==16116== **1,200,064 bytes in 41,681 blocks** are possibly lost in loss record 1,114 of 1,115 ==16116==    at 0x4837B65: calloc (vg_replace_malloc.c:752) ==16116==    by 0x4C25E58: acmp_add_pattern (acmp.cc:517) ==16116==    by 0x4C17C27: modsecurity::operators::PmFromFile::init(std::shared_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) (pm_from_file.cc:71) ==16116==    by 0x4B034E6: yy::seclang_parser::parse() (seclang-parser.yy:889) ==16116==    by 0x4B7B68A: modsecurity::Parser::Driver::parse(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (driver.cc:210) ==16116==    by 0x4B7B8C1: modsecurity::Parser::Driver::parseFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (driver.cc:254) ==16116==    by 0x4BA4973: modsecurity::RulesSet::loadFromUri(char const*) (rules_set.cc:55) ==16116==    by 0x4BA6F1A: msc_rules_add_file (rules_set.cc:368) ==16116==    by 0x283F21: ngx_conf_set_rules_file (ngx_http_modsecurity_module.c:363) ==16116==    by 0x159693: ngx_conf_handler (ngx_conf_file.c:463) ==16116==    by 0x1591A0: ngx_conf_parse (ngx_conf_file.c:319) ==16116==    by 0x197EC3: ngx_http_core_server (ngx_http_core_module.c:2892) ==16116==  ==16116== **4,668,272 bytes in 41,681 blocks** are possibly lost in loss record 1,115 of 1,115 ==16116==    at 0x4837B65: calloc (vg_replace_malloc.c:752) ==16116==    by 0x4C25E10: acmp_add_pattern (acmp.cc:512) ==16116==    by 0x4C17C27: modsecurity::operators::PmFromFile::init(std::shared_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) (pm_from_file.cc:71) ==16116==    by 0x4B034E6: yy::seclang_parser::parse() (seclang-parser.yy:889) ==16116==    by 0x4B7B68A: modsecurity::Parser::Driver::parse(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (driver.cc:210) ==16116==    by 0x4B7B8C1: modsecurity::Parser::Driver::parseFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (driver.cc:254) ==16116==    by 0x4BA4973: modsecurity::RulesSet::loadFromUri(char const*) (rules_set.cc:55) ==16116==    by 0x4BA6F1A: msc_rules_add_file (rules_set.cc:368) ==16116==    by 0x283F21: ngx_conf_set_rules_file (ngx_http_modsecurity_module.c:363) ==16116==    by 0x159693: ngx_conf_handler (ngx_conf_file.c:463) ==16116==    by 0x1591A0: ngx_conf_parse (ngx_conf_file.c:319) ==16116==    by 0x197EC3: ngx_http_core_server (ngx_http_core_module.c:2892)
nginx version 1.21.1
modsec nginx-connector version 1.0.2
modsecurity version v3.0.4
==25494== **1,200,064 bytes in 41,681 blocks** are possibly lost in loss record 785 of 786 ==25494==    at 0x4837B65: calloc (vg_replace_malloc.c:752) ==25494==    by 0x4B63038: acmp_add_pattern (acmp.cc:517) ==25494==    by 0x4B55FCB: modsecurity::operators::PmFromFile::init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) (pm_from_file.cc:71) ==25494==    by 0x4A6E267: yy::seclang_parser::parse() (seclang-parser.yy:871) ==25494==    by 0x4AD3162: modsecurity::Parser::Driver::parse(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (driver.cc:154) ==25494==    by 0x4AD3393: modsecurity::Parser::Driver::parseFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (driver.cc:185) ==25494==    by 0x4AF5117: modsecurity::Rules::loadFromUri(char const*) (rules.cc:98) ==25494==    by 0x4AF720C: msc_rules_add_file (rules.cc:370) ==25494==    by 0x283F21: ngx_conf_set_rules_file (ngx_http_modsecurity_module.c:363) ==25494==    by 0x159693: ngx_conf_handler (ngx_conf_file.c:463) ==25494==    by 0x1591A0: ngx_conf_parse (ngx_conf_file.c:319) ==25494==    by 0x197EC3: ngx_http_core_server (ngx_http_core_module.c:2892) ==25494==  ==25494== **4,668,272 bytes in 41,681 blocks** are possibly lost in loss record 786 of 786 ==25494==    at 0x4837B65: calloc (vg_replace_malloc.c:752) ==25494==    by 0x4B62FF0: acmp_add_pattern (acmp.cc:512) ==25494==    by 0x4B55FCB: modsecurity::operators::PmFromFile::init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) (pm_from_file.cc:71) ==25494==    by 0x4A6E267: yy::seclang_parser::parse() (seclang-parser.yy:871) ==25494==    by 0x4AD3162: modsecurity::Parser::Driver::parse(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (driver.cc:154) ==25494==    by 0x4AD3393: modsecurity::Parser::Driver::parseFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (driver.cc:185) ==25494==    by 0x4AF5117: modsecurity::Rules::loadFromUri(char const*) (rules.cc:98) ==25494==    by 0x4AF720C: msc_rules_add_file (rules.cc:370) ==25494==    by 0x283F21: ngx_conf_set_rules_file (ngx_http_modsecurity_module.c:363) ==25494==    by 0x159693: ngx_conf_handler (ngx_conf_file.c:463) ==25494==    by 0x1591A0: ngx_conf_parse (ngx_conf_file.c:319) ==25494==    by 0x197EC3: ngx_http_core_server (ngx_http_core_module.c:2892)
Total reported RSS change (which is almost completely USS) is
modsecurity version v3/dev/3.1-experimental  (about 20MB for v3.3.0 rules, several rules removed that had load errors)
`before
PID   TID CLS RTPRIO STAT    VSZ   RSS COMMAND
16789 16789 TS       - S+    92460 41020 nginx
after reload
PID   TID CLS RTPRIO STAT    VSZ   RSS COMMAND
16789 16789 TS       - S+<PHONE_NUMBER>6 nginx
`
modsecurity version v3.0.4  (about 17MB for stock v3.3.0 rules)
` before
PID   TID CLS RTPRIO STAT    VSZ   RSS COMMAND
26010 26010 TS       - S+    86460 36564 nginx-static
after
PID   TID CLS RTPRIO STAT    VSZ   RSS COMMAND
26010 26010 TS       - S+<PHONE_NUMBER>6 nginx
`
I do agree the amount of leaked memory is (at least partly) a function of the size of the ruleset being reloaded.
Happy to provide more data and/or test different builds.
Thanks.

Do we open a new issue or reopen this one?

Hi @scaarup ,
There is still an open issue for this general issue:  https://github.com/SpiderLabs/ModSecurity/issues/2552
Unless there are very distinctive cases, I expect having 2552 open is sufficient.  If there is indeed a distinctive case that need to be considered separately, it might make more sense to open a new issue rather than re-open this old one.


refactor(flow): add type checking via flow

We don't necessarily need to merge this. I mostly was trying this out as an experiment in how flow would integrate into the existing workflow.
It ends up being pretty nice in my opinion.

More terse, and readable prop type definitions and doesn't rely on external package
Better intellisense including knowledge of whats being imported from other modules
Superb type inference even before you explicitly provide types
Inline/"compile time" errors for invalid types via VS Code plugin and doesn't rely on runtime like prop-types
Fully opt-in via // @flow so if for some reason providing types for a file doesn't make sense, it's simple to turn off type checks
Even in a smaller project such as this, it managed to catch a potential bug!

To be fair, there is a learning curve to understanding the syntax. And there are some gotchas (example, props + defaultProps type checking seem somewhat cumbersome -- but this could be considered part of the learning curve).
While the documentation is great, if you need to do anything that perhaps fall out of their documentation, help seems a bit hard to find. I wish they had a gitter or other chat channel.
It also took a bit of fiddling to get thing set up correctly with my editor + configuring flexgrid for flowtypes.
So far, I have found that the gains in developer experience, and additional guarantees we can make about our codebase outweigh potential downsides. Also being given escape hatches (turning off types, or defaulting to any in extreme cases) should help keep things moving along.
Like any new technology - especially complex, sophisticated systems like flow - it takes time to understand how it all fits together. But I see real potential here for a great developer experience.

I like it! And good catch on the bug!

I've a couple questions, could we turn it on by default?
Is the flow task meant as built step? If so, should it be part of a webpack build?
Do both tasks (start and build work with it?)

These are all great questions. I'll answer them the best I can in order:
Can we turn flow on by default (instead of having to add anything to the file)?
I had looked into this possibility, but didn't find very many good answers. I think we have two options (kinda):

We can include an eslint rule that has an autofixable option to add // @flow to the beginning of all files. So running eslint --fix will add the line to all files that don't have it
there is an flow check --all but I'm not sure this is a feasible option. My computer just hangs when trying to do this as I think it tried checking node_modules regardless of [ignore] options

Neither of these solutions are exactly what we want, but are a decent enough band-aid perhaps. I'm open to other explorations.
Is the flow task meant as built step? If so, should it be part of a webpack build? Should it run on CI?
I think it would make sense to include it as a build step. There seems to be a couple good options here:

flowtype-loader
tcomb-babel-plugin

I haven't tried these, and we may be unhappy with how much time it adds to our watch step. So may consider one of the following:
Only run loader for CI build and rely on IDE feedback during development
Run loader in parallel and get feedback in terminal, but dont block bundling
Do both tasks (start and build work with it?)
As this PR stands, start and build both work since the types are just ignored and stripped out. Ideally, CI would run type checking and fail if any errors are found.
Is there a Component type? If not, can we define a type for "function / Component extended class" as we would do on TS?
There are a couple ways to define Components (might be good to set a standard there). In this PR I added a prop attribute to the class you can also do the same for state typing.
The other approach is similar to TypeScript where you define your types as ... extends Component<DefaultProps, Props, State> where each definition is a type declared independently.
I ran into issues defining Component types for HOCs. I didn't find much in documentation on how best to declare these types. We may need to define our own "Component as parameter" types that flow will be happy with. I actually went with any here since I found an open source Facebook app that had a similar situation. See here. There are likely more examples in there worth studying!
Can/Should we get rid off any any (either by setting the type or removing the type)?
any should definitely be used sparingly. Flow also has a mixed type which may be more appropriate in many circumstances. (For instance, one spot I said any[] when mixed[] was probably more accurate). any can be a nice escape hatch, and may be overly limiting to completely ban it.

Can we turn flow on by default (instead of having to add anything to the file)?

Would something like git ls-files *.js | xargs flow check-contents work to parse individual files? If that works we may want to also run it as a pre push check
Good point, we should run eslint --fix as precommit hook anyway, that would make sure that when it runs on CI, if there are any issues, things fail rather than skipping the file

Is the flow task meant as built step? If so, should it be part of a webpack build? Should it run on CI?

Got it, yeah I agree it makes sense to combine it with the lint task locally. That would also ensure it runs on CI too (since it checks for linting issues already)
If code is already stripped out automatically then yeah I agree no need to add additional loaders. Particularly when the editor should provide the same feedback already

Is there a Component type? If not, can we define a type for "function / Component extended class" as we would do on TS?

Makes sense, we also don't need to fix every single issue on this PR so as you have it now looks good to me

Can/Should we get rid off any any (either by setting the type or removing the type)?

Same thing, we can take another pass after this PR lands


ToDo: diffs FF60-FF61

211 diffs ( 90 new, 101 gone, 20 different )
FF61 is scheduled for release 26-June
other

[x] 1450448 re 2730+2750 pushed to ESR60 - https://github.com/ghacksuserjs/ghacks-user.js/commit/70abeda9d495edd72d5ca580233af17994e9ba49 & https://github.com/ghacksuserjs/ghacks-user.js/commit/d34894e9656cddcdf32c177a15e088e2316e0543
[ ] 0100s UI changes & cleanup all prefs for home & newtab, related to 0514 (Activity Stream)

new in v61.0:

[x] pref("network.file.disable_unc_paths", true); // hidden pref - https://github.com/ghacksuserjs/ghacks-user.js/commit/56acb4cff590941bb5b1e2a0fe491941bb653f96

1413868 currently access denied, 26424

[x] pref("shield.savant.enabled", false); 1462725 - https://github.com/ghacksuserjs/ghacks-user.js/commit/dceef9d1dbb8060be767f9fce3d7f3f07f3803f5
pref("security.block_ftp_subresources", true); - Article, 1452701 - added here for visibility

removed, renamed or hidden in v61.0:

[x] pref("browser.storageManager.enabled", true); 1428306 - https://github.com/ghacksuserjs/ghacks-user.js/commit/a4a9b9a675df5f43e32d82062bd32e29f1ff4936
[x] experiments 1450801 & 1420908 - https://github.com/ghacksuserjs/ghacks-user.js/commit/9386fb55819f960842a5328cb614e13899f5d805

pref("experiments.enabled", true);
pref("experiments.manifest.uri", "https://telemetry-experiment.cdn.mozilla.net/manifest/v1/firefox/%VERSION%/%CHANNEL%");
pref("experiments.supported", true);
pref("experiments.activeExperiment", false);

[x] remote jar 1427726 - https://github.com/ghacksuserjs/ghacks-user.js/commit/9386fb55819f960842a5328cb614e13899f5d805

pref("network.jar.block-remote-files", true);
pref("network.jar.open-unsafe-types", false);

[x] pref("services.blocklist.signing.enforced", true); 1397230 - https://github.com/ghacksuserjs/ghacks-user.js/commit/a4a9b9a675df5f43e32d82062bd32e29f1ff4936

changed in v61.0:

pref("privacy.usercontext.about_newtab_segregation.enabled", true); // prev: false 1703 (inactive)

ignore
click me for details
==NEW
pref("accessibility.monoaudio.enable", false);
pref("apz.android.chrome_fling_physics.enabled", false);
pref("apz.android.chrome_fling_physics.friction", "0.015");
pref("apz.android.chrome_fling_physics.inflexion", "0.35");
pref("apz.android.chrome_fling_physics.stop_threshold", "0.1");
pref("apz.max_tap_time", 300);
pref("browser.newtabpage.activity-stream.asrouterExperimentEnabled", false);
pref("browser.newtabpage.activity-stream.asrouter.snippetsUrl", "");
pref("browser.newtabpage.activity-stream.feeds.aboutpreferences", true);
pref("browser.newtabpage.activity-stream.feeds.asrouterfeed", true);
pref("browser.newtabpage.activity-stream.feeds.theme", true);
pref("browser.newtabpage.activity-stream.section.highlights.includeBookmarks", true);
pref("browser.newtabpage.activity-stream.section.highlights.includeDownloads", true);
pref("browser.newtabpage.activity-stream.section.highlights.includeVisited", true);
pref("browser.tabs.20FpsThrobber", false);
pref("browser.tabs.30FpsThrobber", false);
pref("browser.tabs.closeTabByDblclick", false);
pref("browser.tabs.insertAfterCurrent", false);
pref("devtools.accessibility.enabled", false);
pref("devtools.application.enabled", false);
pref("devtools.browserconsole.html", false);
pref("devtools.debugger.alphabetize-outline", false);
pref("devtools.debugger.component-stack-visible", true);
pref("devtools.debugger.component-visible", true);
pref("devtools.debugger.features.component-stack", false);
pref("devtools.debugger.features.pause-points", true);
pref("devtools.debugger.features.skip-pausing", false);
pref("devtools.debugger.skip-pausing", false);
pref("devtools.debugger.tabsBlackBoxed", "[]");
pref("devtools.inspector.fonteditor.enabled", false);
pref("devtools.inspector.show-three-pane-tooltip", false);
pref("devtools.inspector.three-pane-enabled", false);
pref("devtools.inspector.three-pane-toggle", false);
pref("devtools.netmonitor.responseBodyLimit", 1048576);
pref("devtools.netmonitor.saveRequestAndResponseBodies", true);
pref("devtools.toolbox.tabsOrder", "");
pref("dom.event.default_to_passive_touch_listeners", true);
pref("dom.indexedDB.storageOption.enabled", false);
pref("dom.performance.enable_scheduler_timing", false);
pref("dom.performance.time_to_dom_content_flushed.enabled", false);
pref("dom.use_components_shim", true);
pref("dom.vr.external.enabled", false);
pref("extensions.webcompat.perform_injections", true);
pref("extensions.webextensions.background-delayed-startup", false);
pref("gfx.downloadable_fonts.validate_variation_tables", true);
pref("gfx.webrender.async-scene-build", 2);
pref("gfx.webrender.blob.invalidation", true);
pref("gfx.webrender.dcomp-win.enabled", true);
pref("identity.fxaccounts.messages.enabled", false);
pref("intl.ime.hack.on_any_apps.fire_key_events_for_composition", false); // ANDROID IME-unaware apps
pref("layers.enable-tiles-if-skia-pomtp", false);
pref("layers.tiles.edge-padding", false);
pref("layers.tiles.retain-back-buffer", true);
pref("layout.css.emulate-moz-box-with-flex", false);
pref("layout.css.getPropertyCSSValue.enabled", false);
pref("layout.css.moz-document.url-prefix-hack.enabled", true);
pref("layout.css.parsing.parallel", true);
pref("marionette.contentListener", false);
pref("media.audio-max-decode-error", 3);
pref("media.av1.enabled", true);
pref("media.clearkey.persistent-license.enabled", false);
pref("media.eme.audio.blank", false);
pref("media.eme.video.blank", false);
pref("media.forcestereo.enabled", false);
pref("media.hls.enabled", false);
pref("media.mediasource.experimental.enabled", false);
pref("media.resampling.enabled", false);
pref("media.ruin-av-sync.enabled", false);
pref("media.videocontrols.lock-video-orientation", false);
pref("media.video-max-decode-error", 2);
pref("media.webaudio.audiocontextoptions-samplerate.enabled", true);
pref("media.webspeech.recognition.force_enable", false);
pref("media.webspeech.synth.force_global_queue", false);
pref("media.webspeech.test.enable", false);
pref("media.webspeech.test.fake_fsm_events", false);
pref("media.webspeech.test.fake_recognition_service", false);
pref("mousewheel.autodir.enabled", false);
pref("mousewheel.autodir.honourroot", false);
pref("network.cookie.move.interval_sec", 10);
pref("network.http.fallback-connection-timeout", 5);
pref("network.predictor.doing-tests", false); // 1448220 ?
pref("security.pki.mitm_canary_issuer", ""); // 1450967 ?
pref("security.pki.mitm_canary_issuer.enabled", true);
pref("security.sandbox.gmp.win32k-disable", false);
pref("services.settings.changes.path", "/buckets/monitor/collections/changes/records");
pref("services.settings.default_bucket", "main");
pref("services.settings.default_signer", "remote-settings.content-signature.mozilla.org");
pref("shield.savant.loglevel", "warn");

==REMOVED or HIDDEN
pref("apz.minimap.visibility.enabled", false);
pref("browser.newtabpage.activity-stream.collapseTopSites", false);
pref("browser.newtabpage.activity-stream.enableWideLayout", true);
pref("browser.newtabpage.activity-stream.section.highlights.collapsed", false);
pref("browser.newtabpage.activity-stream.section.topstories.collapsed", false);
pref("browser.newtabpage.activity-stream.showTopSites", true);
pref("browser.tabs.restorebutton", 0);
pref("devtools.browserconsole.filter.csserror", true);
pref("devtools.browserconsole.filter.csslog", false);
pref("devtools.browserconsole.filter.cssparser", false);
pref("devtools.browserconsole.filter.exception", true);
pref("devtools.browserconsole.filter.jslog", true);
pref("devtools.browserconsole.filter.jswarn", true);
pref("devtools.browserconsole.filter.netwarn", true);
pref("devtools.browserconsole.filter.network", true);
pref("devtools.browserconsole.filter.networkinfo", false);
pref("devtools.browserconsole.filter.secerror", true);
pref("devtools.browserconsole.filter.secwarn", true);
pref("devtools.browserconsole.filter.servererror", false);
pref("devtools.browserconsole.filter.serverinfo", false);
pref("devtools.browserconsole.filter.serverlog", false);
pref("devtools.browserconsole.filter.serverwarn", false);
pref("devtools.browserconsole.filter.serviceworkers", true);
pref("devtools.browserconsole.filter.sharedworkers", true);
pref("devtools.browserconsole.filter.windowlessworkers", true);
pref("devtools.browserconsole.new-frontend-enabled", false);
pref("devtools.changesview.enabled", false);
pref("devtools.computed.boxmodel.opened", true);
pref("devtools.debugger.features.breakpoints-dropdown", false);
pref("devtools.eventsview.enabled", false);
pref("devtools.hud.loglimit.console", 1000);
pref("devtools.hud.loglimit.cssparser", 1000);
pref("devtools.hud.loglimit.exception", 1000);
pref("devtools.hud.loglimit.network", 1000);
pref("devtools.inspector.colorWidget.enabled", false);
pref("devtools.inspector.split-rule-enabled", false);
pref("devtools.inspector.split-sidebar-toggle", false);
pref("devtools.webconsole.filter.csserror", true);
pref("devtools.webconsole.filter.csslog", false);
pref("devtools.webconsole.filter.cssparser", false);
pref("devtools.webconsole.filter.exception", true);
pref("devtools.webconsole.filter.jslog", false);
pref("devtools.webconsole.filter.jswarn", true);
pref("devtools.webconsole.filter.netwarn", true);
pref("devtools.webconsole.filter.network", true);
pref("devtools.webconsole.filter.networkinfo", false);
pref("devtools.webconsole.filter.secerror", true);
pref("devtools.webconsole.filter.secwarn", true);
pref("devtools.webconsole.filter.servererror", false);
pref("devtools.webconsole.filter.serverinfo", false);
pref("devtools.webconsole.filter.serverlog", false);
pref("devtools.webconsole.filter.serverwarn", false);
pref("devtools.webconsole.filter.serviceworkers", true);
pref("devtools.webconsole.filter.sharedworkers", false);
pref("devtools.webconsole.filter.windowlessworkers", false);
pref("devtools.webconsole.new-frontend-enabled", true);
pref("dom.animations-api.pending-member.enabled", true);
pref("dom.compartment_per_addon", true);
pref("dom.forms.number", true);
pref("dom.ipc.cpows.allow-cpows-in-compat-addons", "{b9db16a4-6edc-47ec-a1f4-b86292ed211d},firegestures@xuldev.org,{DDC359D1-844A-42a7-9AA1-88A850A938A8},privateTab@infocatcher,mousegesturessuite@lemon_juice.addons.mozilla.org,treestyletab@piro.sakura.ne.jp,cliqz@cliqz.com,{AE93811A-5C9A-4d34-8462-F7B864FC4696},contextsearch2@lwz.addons.mozilla.org,{EF522540-89F5-46b9-B6FE-1829E2B572C6},{677a8f98-fd64-40b0-a883-b8c95d0cbf17},images@wink.su,fx-devtools,url_advisor@kaspersky.com,{d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d},{dc572301-7619-498c-a57d-39143191b318},dta@downthemall.net,{86095750-AD15-46d8-BF32-C0789F7E6A32},screenwise-prod@google.com,{91aa5abe-9de4-4347-b7b5-322c38dd9271},secureLogin@blueimp.net,ich@maltegoetz.de,come.back.block.image.from@cat-in-136.blogspot.com,{7b1bf0b6-a1b9-42b0-b75d-252036438bdc},s3crypto@data,{1e0fd655-5aea-4b4c-a583-f76ef1e3af9c},akahuku.fx.sp@toshiakisp.github.io,{aff87fa2-a58e-4edd-b852-0a20203c1e17},{1018e4d6-728f-4b20-ad56-37578a4de76b},rehostimage@engy.us,lazarus@interclue.com,{b2e69492-2358-071a-7056-24ad0c3defb1},flashstopper@byo.co.il,{e4a8a97b-f2ed-450b-b12d-ee082ba24781},jid1-f3mYMbCpz2AZYl@jetpack,{8c550e28-88c9-4764-bb52-aa489cf2efcd},{37fa1426-b82d-11db-8314-0800200c9a66},{ac2cfa60-bc96-11e0-962b-0800200c9a66},igetter@presenta.net,killspinners@byo.co.il,abhere2@moztw.org,{fc6339b8-9581-4fc7-b824-dffcb091fcb7},wampi@wink.su,backtrack@byalexv.co.uk,Gladiator_X@mail.ru,{73a6fe31-595d-460b-a920-fcc0f8843232},{46551EC9-40F0-4e47-8E18-8E5CF550CFB8},acewebextension_unlisted@acestream.org,@screen_maker,yasearch@yandex.ru,sp@avast.com,s3google@translator,igetterextension@presenta.net,{C1A2A613-35F1-4FCF-B27F-2840527B6556},screenwise-testing@google.com,helper-sig@savefrom.net,ImageSaver@Merci.chao,proxtube@abz.agency,wrc@avast.com,{9AA46F4F-4DC7-4c06-97AF-5035170634FE},jid1-CikLKKPVkw6ipw@jetpack,artur.dubovoy@gmail.com,nlgfeb@nlgfeb.ext,{A065A84F-95B6-433A-A0C8-4C040B77CE8A},fdm_ffext@freedownloadmanager.org");
pref("dom.ipc.cpows.forbid-cpows-in-compat-addons", true);
pref("dom.ipc.scheduler", false);
pref("dom.requestcontext.enabled", false);
pref("experiments.manifest.fetchIntervalSeconds", 86400);
pref("extensions.{972ce4c6-7e08-4474-a285-3208198ce6fd}.description", "chrome://browser/locale/browser.properties");
pref("extensions.{972ce4c6-7e08-4474-a285-3208198ce6fd}.name", "chrome://browser/locale/browser.properties");
pref("extensions.allow-non-mpc-extensions", true);
pref("extensions.dss.switchPending", false);
pref("general.skins.selectedSkin", "classic/1.0");
pref("gfx.downloadable_fonts.keep_variation_tables", false);
pref("layers.advanced.background-color", false);
pref("layers.advanced.background-image", 2);
pref("layers.advanced.border-layers", 2);
pref("layers.advanced.bullet-layers", 2);
pref("layers.advanced.canvas-background-color", 2);
pref("layers.advanced.caret-layers", false);
pref("layers.advanced.columnRule-layers", 2);
pref("layers.advanced.image-layers", 2);
pref("layers.advanced.outline-layers", 2);
pref("layers.advanced.solid-color", false);
pref("layers.advanced.table", false);
pref("layout.accessiblecaret.always_show_when_scrolling", true);
pref("layout.css.expensive-style-struct-assertions.enabled", false);
pref("layout.css.float-logical-values.enabled", true);
pref("layout.css.scoped-style.enabled", false);
pref("layout.css.scope-pseudo.enabled", true);
pref("layout.css.servo.chrome.enabled", true);
pref("layout.css.servo.enabled", true);
pref("layout.css.text-combine-upright.enabled", true);
pref("media.wmf.skip-blacklist", false);
pref("network.predictor.redirect-likely-confidence", 75);
pref("services.blocklist.changes.path", "/buckets/monitor/collections/changes/records");
pref("svg.path-caching.enabled", true);
pref("toolkit.telemetry.infoURL", "https://www.mozilla.org/legal/privacy/firefox.html#telemetry");

==CHANGED
pref("app.normandy.run_interval_seconds", 21600); // prev: 86400
pref("browser.newtabpage.activity-stream.feeds.section.topstories.options", "{\"api_key_pref\":\"extensions.pocket.oAuthConsumerKey\",\"hidden\":true,\"provider_icon\":\"pocket\",\"provider_name\":\"Pocket\",\"read_more_endpoint\":\"https://getpocket.com/explore/trending?src=fx_new_tab\",\"stories_endpoint\":\"https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=3&consumer_key=$apiKey&locale_lang=en-US&feed_variant=default_spocs_off\",\"stories_referrer\":\"https://getpocket.com/recommendations\",\"topics_endpoint\":\"https://getpocket.cdn.mozilla.net/v3/firefox/trending-topics?version=2&consumer_key=$apiKey&locale_lang=en-US\",\"show_spocs\":false,\"personalized\":true}"); // prev: "{\"api_key_pref\":\"extensions.pocket.oAuthConsumerKey\",\"hidden\":true,\"provider_description\":\"pocket_description\",\"provider_icon\":\"pocket\",\"provider_name\":\"Pocket\",\"read_more_endpoint\":\"https://getpocket.com/explore/trending?src=fx_new_tab\",\"stories_endpoint\":\"https://getpocket.cdn.mozilla.net/v3/firefox/global-recs?version=3&consumer_key=$apiKey&locale_lang=en-US&feed_variant=default_spocs_off\",\"stories_referrer\":\"https://getpocket.com/recommendations\",\"privacy_notice_link\":\"https://www.mozilla.org/privacy/firefox/#suggest-relevant-content\",\"disclaimer_link\":\"https://getpocket.com/firefox/new_tab_learn_more\",\"topics_endpoint\":\"https://getpocket.cdn.mozilla.net/v3/firefox/trending-topics?version=2&consumer_key=$apiKey&locale_lang=en-US\",\"show_spocs\":false,\"personalized\":true}"
pref("browser.tabs.remote.warmup.enabled", true); // prev: false
pref("devtools.debugger.features.map-scopes", true); // prev: false
pref("extensions.geckoProfiler.getSymbolRules", "localBreakpad,remoteBreakpad,dump_syms.exe"); // prev: "localBreakpad,remoteBreakpad"
pref("extensions.langpacks.signatures.required", true); // prev: false
pref("extensions.legacy.exceptions", "testpilot@cliqz.com,@testpilot-containers,jid1-NeEaf3sAHdKHPA@jetpack,@activity-streams,pulse@mozilla.com,@testpilot-addon,@min-vid,tabcentertest1@mozilla.com,snoozetabs@mozilla.com,speaktome@mozilla.com,hoverpad@mozilla.com"); // prev: "{972ce4c6-7e08-4474-a285-3208198ce6fd},testpilot@cliqz.com,@testpilot-containers,jid1-NeEaf3sAHdKHPA@jetpack,@activity-streams,pulse@mozilla.com,@testpilot-addon,@min-vid,tabcentertest1@mozilla.com,snoozetabs@mozilla.com,speaktome<EMAIL_ADDRESS>pref("extensions.webextensions.tabhide.enabled", true); // prev: false
pref("font.name-list.emoji", "Segoe UI Emoji, Twemoji Mozilla"); // prev: "Segoe UI Emoji, EmojiOne Mozilla"
pref("gfx.webrender.blob-images", true); // prev: 1
pref("image.mem.shared", true); // prev: 2
pref("layers.omtp.paint-workers", -1); // prev: 1
pref("layers.tile-height", 512); // prev: 256
pref("layers.tile-width", 512); // prev: 256
pref("layout.css.moz-document.content.enabled", false); // prev: true
pref("lightweightThemes.selectedThemeID", "default-theme@mozilla.org"); // prev: ""
pref("media.wmf.amd.vp9.enabled", false); // prev: true
pref("network.http.throttle.version", 2); // prev: 1
pref("network.trr.blacklist-duration", 1200); // prev: 259200

I think we should just remove browser.storageManager.enabled rather than deprecate it. I'm guessing it was only ever added to handle rolling it out and/or testing. If ESR60 users disable Storage API, it could be handy, other than that it's a waste of space (note: haven't looked up the ticket or even know if they will push this to ESR)

bugzilla tickets for some of the remaining prefs

browser.cache.offline.insecure.enable
Bug 1429732 - Use a pref to disable registerProtocolHandler in insecure contexts.

browser.storageManager.enabled
Bug 1428306 - Remove browser.storageManager.enabled pref

browser.tabs.remote.warmup.enabled
Bug 1457651 - Re-enable tab warming on Tier3 platforms.
Bug 1456602 - Enable tab warming for Windows and Linux on all channels.
Bug 1423220 - Enable tab warming by default for Nightly builds.

dom.indexedDB.storageOption.enabled
Bug 1451486 - Part 1 - Ignore the storage attribute on indexedDB.open() by default.

dom.keyboardevent.dispatch_during_composition
Bug 1446401 - Start to dispatch "keydown" event and "keyup" event even during in composition in Nightly and early Beta

experiments.enabled
Bug 1420908: Remove telemetry experiments.

experiments.manifest.uri
Bug 1420908: Remove telemetry experiments.

experiments.supported
Bug 1450801 Part 1: Remove front-end (about:addons) support for telemetry experiments.

extensions.legacy.exceptions
Bug 1372694 - Stop making the default theme a heavyweight theme.

extensions.webextensions.tabhide.enabled
Bug 1455040 - Enable tab hiding API by default.

gfx.downloadable_fonts.validate_variation_tables
Bug 1341088 - Enable validation of opentype variation tables in downloadable fonts, and replace the keep_variation_tables pref with validate_variation_tables (default=true).

layout.css.moz-document.content.enabled
Bug 1422245 Disable @-moz-document in content pages

network.cookie.move.interval_sec
Bug 1454100 - Move cookies around every X seconds

network.jar.block-remote-files
Bug 1427726 - remove support for remote JAR files,

network.jar.open-unsafe-types
Bug 1427726 - remove support for remote JAR files,

security.block_ftp_subresources
Bug 1452701 - Add pref to allow FTP subresources.

services.blocklist.signing.enforced
Bug 1397230 - Generalize blocklist clients to remote settings clients

I was going to check on that, because it would seem to be a leftover they missed. I haven't looked at the tickets (except a few others I looked up in the pile of new items I moved) - but I assume experiments (all of 0501) is now replaced by FF60's app.shield.optoutstudies.enabled (0512)

^^ Edit: bump: I think we should just remove browser.storageManager.enabled
Pants wanna remove pref. Pants is smart, be like Pants.

we can also remove services.blocklist.signing.enforced rather than deprecate it, because it's default true since FF50

NEW: security.block_ftp_subresources - Article: https://blog.mozilla.org/security/2018/05/07/blocking-ftp-subresource-loads-within-non-ftp-documents-in-firefox-61/ Not sure if we need to add this, default true is what we want. I'd be more interested in making 0708 active

dom.indexedDB.storageOption.enabled - safe to ignore, enabled=false is what we want. see https://www.fxsitecompat.com/en-CA/docs/2018/storage-option-for-indexeddb-open-has-been-removed/
extensions.webcompat.perform_injections ( + extensions.webcompat.perform_ua_overrides )
if they are user-set they get reset when the system addon starts up. The webcompat SA allows mozilla to inject scripts or css into a site and/or change the User-agent string, presumably to unbreak certain sites. ( see https://wiki.mozilla.org/Compatibility/Go_Faster_Addon/Override_Policies_and_Workflows ) ATM (in nightly) there are no real live settings just a test config (injections, ua-overrides). To disable it we can't use user_pref but pref() should probably work. IDK if we need/want to do that though, up to you.

dom.indexedDB.storageOption.enabled - I've read this has been breaking extensions, or I may be confused over the removal of a deprecated or non-standard "persistent" IDB feature. I'll have to dig a little more - yup, that compat link is what I was thinking, I doubt any extensions we use break. Agreed, ignore.
Webcompat - agreed, ignore them. Would be interesting to see if webcompat can override RFP.

OT: I have no idea what this means - is it meant to be scary?

^^ sounds like an Ad for VulnDB and the blame is put on Mitre, not mozilla.

In this case, CVE delivered information on 318 vulnerabilities up to seven years late. Compare that to VulnDB, which had information on all of these issues available immediately after disclosure

sounds like an Ad

I know. I just thought I'd draw attention to CVE - I've had loads of problems trying to find things in there.. in other words, I think it's a bit sh^t. No idea why that article focused on FF

61b14 changes since 61b6
new
pref("browser.newtabpage.activity-stream.asrouter.snippetsUrl", "");
pref("layout.css.moz-document.url-prefix-hack.enabled", true); // 61b6: false
pref("services.sync.engine.bookmarks.validation.enabled", true);
pref("services.sync.engine.passwords.validation.enabled", true);
pref("shield.savant.enabled", false);
pref("shield.savant.loglevel", "warn");
removed, renamed or hidden
pref("geo.provider.ms-windows-location", false);
changed
pref("browser.cache.offline.insecure.enable", false); // prev: true
pref("browser.newtabpage.activity-stream.telemetry.ut.events", true); // prev: false
pref("browser.startup.blankWindow", true); // prev: false
pref("device.sensors.ambientLight.enabled", false); // prev: true
pref("device.sensors.proximity.enabled", false); // prev: true
pref("dom.keyboardevent.dispatch_during_composition", true); // prev: false
pref("dom.registerContentHandler.enabled", false); // prev: true
pref("geo.wifi.uri", "https://location.services.mozilla.com/v1/geolocate?key=%MOZILLA_API_KEY%"); // prev: "https://www.googleapis.com/geolocation/v1/geolocate?key=%GOOGLE_API_KEY%"
pref("layout.display-list.retain", true); // prev: false
pref("media.wmf.amd.vp9.enabled", false); // prev: true
pref("network.tcp.tcp_fastopen_enable", true); // prev: false
pref("privacy.usercontext.about_newtab_segregation.enabled", true); // prev: false

61.0 changes since 61b14
changed
pref("app.update.channel", "beta"); // prev: "release"
pref("app.update.url.details", "https://www.mozilla.org/%LOCALE%/firefox/beta/notes"); // prev: "https://www.mozilla.org/%LOCALE%/firefox/notes"
pref("app.update.url.manual", "https://www.mozilla.org/firefox/beta"); // prev: "https://www.mozilla.org/firefox/"
pref("devtools.devedition.promo.enabled", true); // prev: false
pref("toolkit.telemetry.enabled", true); // prev: false

new hidden pref:
// Disable using UNC paths (TBB bug 26424 and Mozilla's bug 1413868)
user_pref("network.file.disable_unc_paths", true);

uplifted all the way back to and including ESR52

I guess we better finish this - have been uber busy (and still am), so just as well 61 is a bit of a non event for our changes
FYI: https://jakearchibald.com/2018/i-discovered-a-browser-bug/ - interesting read (and https://github.com/whatwg/fetch/issues/144#issuecomment-368040980 which is linked in the article

@Thorin-Oakenpants do you need anything from me to finish this?

@earthlng nah, I can do it - I'm just not all that bothered right now to be honest. Almost everything left in OP can be ignored (that's the ones not bulleted for us to action)
Got to say, I am so busy (with some other puter shit I have put off for far too long due to here) and I have had to deal with some other personal things outside the puter world
Might wrap it up by the end of next month :grinning: .. or you could do the whole thing and I'll just check and bitch about anything I don;t like

moving all from changed to ignore
pref("browser.tabs.remote.warmup.enabled", true); // prev: false
pref("extensions.legacy.exceptions", "testpilot@cliqz.com,@testpilot-containers,jid1-NeEaf3sAHdKHPA@jetpack,@activity-streams,pulse@mozilla.com,@testpilot-addon,@min-vid,tabcentertest1@mozilla.com,snoozetabs@mozilla.com,speaktome@mozilla.com,hoverpad@mozilla.com"); // prev: "{972ce4c6-7e08-4474-a285-3208198ce6fd},testpilot@cliqz.com,@testpilot-containers,jid1-NeEaf3sAHdKHPA@jetpack,@activity-streams,pulse@mozilla.com,@testpilot-addon,@min-vid,tabcentertest1@mozilla.com,snoozetabs@mozilla.com,speaktome<EMAIL_ADDRESS>pref("extensions.webextensions.tabhide.enabled", true); // prev: false
pref("layout.css.moz-document.content.enabled", false); // prev: true
pref("media.wmf.amd.vp9.enabled", false); // prev: true

PS: when we start v62, I think we can turn on container tab prefs - all of section 1700 - esp now 1703 is now default true

pref("browser.newtabpage.activity-stream.asrouter.snippetsUrl", "");
pref("shield.savant.enabled", false);
pref("shield.savant.loglevel", "warn");

AS is covered by nixing the xpi
loglevel seems fine as "warn" (wouldn't we want a warning?) - whatever savant is (I'll look it up)
so WTF is savant and is this already covered? links? ticket?

https://bugzilla.mozilla.org/show_bug.cgi?id=1465694
https://bugzilla.mozilla.org/show_bug.cgi?id=1457226
more telemetry - aren't these covered by shield being disabled?

I will definitely also disable tab warmup because I don't want FF to start doing something whenever I hover over a background tab with my mouse. But IDK if it's worth adding under 5000.
And lets not forget about this either. The bugzilla for it is still Access Denied.
I think this is about a potential proxy bypass and is definitely more severe in TBB than vanilla FF. But since most people will never really need a website (??) to access a UNC path anyway, it shouldn't be a problem if we add it IMHO.

error is the log level below warn.

A fresh bugzilla. And it has Critical Priority.

shield.savant.enabled - do we stick it in 0330 or 0503 ?

fits better in 0503 IMO

@earthlng I need some help with the last commit re the changes with the new home page and new tab UI
/* 0102: set start page (0=blank, 1=home, 2=last visited page, 3=resume previous session)
 * [SETTING] General>Startup>When Firefox starts>Restore previous session ***/
   // user_pref("browser.startup.page", 0);
/* 0103: set your "home" page (see 0102) or use an extension
 * about:home=Activity Stream (default, see 0514), custom URL (see example below), about:blank
 * [SETTING] Home>New Windows and Tabs>Homepage and new windows ***/
   // user_pref("browser.startup.homepage", "https://www.example.com/");
/* 0104: set your "newtab" page
 * true=Activity Stream (default, see 0514), false=blank page (or set by an extension)
 * [SETTING] Home>New Windows and Tabs>New tabs ***/
user_pref("browser.newtabpage.enabled", false);
user_pref("browser.newtab.preload", false); // not used now?

0102: I haven't got a pre61 handy to look at the previous UI, so I am not sure what the old UI had for options. The new UI only has an option for checking/unchecking "restore previous session". So think we (and I'm all tired out) need to check that those values 0=blank 1=home still exist, or if they get set by 0102 or 0103, and also work out what is the default is and what it changes to/from when flicking between the options in the new UI (I am so confused right now).
0103: i think this is right
0104: this was 0360 - it only has 2 options, even if you use an extension for your newtab. I do not think the preload does anything any more (because all the defaults for AS are handled by AS prefs), so if we can check that we could remove that pref - yes/no?
tied 0103+0104 into 0514, but not sure if I need to add anything more to 0514?

The defaults for both newtab and homepage and new windows are to use AS. The default for startup seems to be 1 = home = AS (need to confirm that)

^ bump

this is what it looks like in FF60:

0102+0103 could use a [SETTING-ESR60].
re: 0102 ...

need to check that those values 0=blank 1=home still exist

they do

work out what is the default is and what it changes to/from when flicking between the options in the new UI

default is 1. In FF61+ it toggles between 1 and 3
0103: I don't really like the "or use an extension". A newtab extension has nothing to do with privacy/security
re: 0104: "I do not think the preload does anything any more" - in FF61+ it's obsolete because it's checked together with browser.newtabpage.enabled and if that's false then so is newtab preloading. But in ESR60 and ESR52 it's slightly different and for that reason I'd say we keep it.

if 0102 is only about previous session then maybe this could be with the other session stuff?

it's about what happens on startup and thus fits perfectly where it is right now, IMO.
ps: I wouldn't mind activating both 0102 and 0103 and setting them to 0 and "about:blank" to make sure that no websites start loading before all extensions are ready.

this is what it looks like in FF60:

So that's the same as ESR60 then? Yup, we'll need a [SETTING-ESR60] tag.

it's about what happens on startup and thus fits perfectly where it is right now, IMO

was just a thought pending answers.   It's already complicated as it is.

ps: I wouldn't mind activating both 0102 and 0103 and setting them to 0 and "about:blank" to make sure that no websites start loading before all extensions are ready.

I thought about it yesterday - some people set a custom url, others use an extension (like myself - it sets a moz-extension address) - this is something that users need to set themselves IMO. There's a whole UI for it. For those who want they can set an override.
I thought about making the inactive value as about:blank, but I also wanted to keep the URL example

The answer that 60 vs 61 handles shit differently put a whole new spin on everything.

I don't even know how to do this commit anymore. There's no HOME option tab in ESR60, the 0102 prefs have different outcomes in 60 vs 61. I'm lost. Had enough of it. Please just do it for me (and I'll bitch later)

here's what 0102 looks like in 61+

draft - we could always move some stuff into the section header
/*** 0100: STARTUP ***/
user_pref("_user.js.parrot", "0100 syntax error: the parrot's dead!");
/* 0101: disable default browser check
 * [SETTING] General>Startup>Always check if Firefox is your default browser ***/
user_pref("browser.shell.checkDefaultBrowser", false);
/* 0102: set start page
 * FF61+ : 1=home 3=resume previous session
 * ESR60x: 0=blank, 1=home, 2=last visited page, 3=resume previous session
 * [SETTING] General>Startup>When Firefox starts>Restore previous session
 * [SETTING-ESR60] General>Startup>When Firefox starts ***/
   // user_pref("browser.startup.page", 0);
/* 0103: set your "home" page (see 0102) or use an extension
 * about:home=Activity Stream (default, see 0514), custom URL, about:blank
 * [SETTING] Home>New Windows and Tabs>Homepage and new windows ***/
   // user_pref("browser.startup.homepage", "about:blank");
/* 0104: set your "newtab" page
 * true=Activity Stream (default, see 0514), false=blank page (or set by an extension)
 * [SETTING] Home>New Windows and Tabs>New tabs ***/
user_pref("browser.newtabpage.enabled", false);
user_pref("browser.newtab.preload", false); // ESR60

This is getting messy

set browser.startup.page to 0 - this turns both homepage+newwindows and newtabs to "blank page" (you need to reload the options page)
changing homepage+newwindows to custom url or Firefox home (activity stream) changes browser.startup.page back to 1
changinghomepage+newwindows back to blank does not reset browser.startup.page to 0 (and there is not UI for that in 61+). Guess they really want the startup page to be AS

So that's the same as ESR60 then?

IDK, ESR60 portable isn't out yet

some people set a custom url, others use an extension

yeah but most people don't use either and IMHO it should be the people using custom urls or extensions that should set an override and not the other way around. Also, people might not realize that when they have a homepage set and startup set to 1 (default) that the site will start to load before all their extensions are ready.

So what happens if you enforce 0 or 2

should work just fine. You can have a homepage set and still startup with blank if you set this to 0. But I think if you then open a new window it will load the homepage and that's why you also may want to set homepage to about:blank (or your extension url or whatnot)
* FF61+ : 1=home 3=resume previous session
* ESR60x: 0=blank, 1=home, 2=last visited page, 3=resume previous session

I don't like this because all 4 values are still supported in all versions
re: about:home=Activity Stream (default, see 0514), custom URL, about:blank
I'm pretty sure that when you don't delete the AS xpi and you use the UI to set this to "Firefox Homepage" (or whatever it's called), it will set the value to "about:newtab". If you delete the xpi it will set "about:home", last I checked.
re: true=Activity Stream (default, see 0514), false=blank page (or set by an extension)
when you use a newtab extension FF will set these prefs for you automatically and even lock the UI, doesn't it? So, again, no need to mention extensions anywhere in this section
user_pref("browser.newtab.preload", false); // ESR60 - no need for that comment, please remove

FYI: I just tested with browser.startup.page = 0 (and I use an extension to control my newtab/home/newwindow) and it is independent of the extension. Haven't read your post above just yet, getting my thoughts down .. since there is no UI in FF61 to stop AS on startpage, then 0102 we could enforce as 0 - this also controls sets 103+104 to about:blank (but not those controlled by an extension - see my test I just did). thoughts?
Not sure what just doing 0102 as 0 is on ESr60

I don't like this because all 4 values are still supported in all versions

I re-edited the draft which is fast becoming moot again anyway

I'm pretty sure that when you don't delete the AS xpi and you use the UI to set this to "Firefox Homepage" (or whatever it's called), it will set the value to "about:newtab". If you delete the xpi it will set "about:home", last I checked.

If i delete the XPI (as in my current setup) it sets about:home = correct. WITH the XPI I will recheck but I got the values I wrote from a default portable 61 - will let you know

when you use a newtab extension FF will set these prefs for you automatically and even lock the UI, doesn't it? So, again, no need to mention extensions anywhere in this section

here is my UI

newtab is locked, but homepage+newwindow are not

0104 * true=Activity Stream (default, see 0514), false=blank page (or set by an extension)

we can drop the extension part here since they're mention in 0103 and in 0104 setting as blank has no effect - extension overrides it (and locks it)
I have a plan .. bear with me.
Let's just keep it simple and fuck all the infos and settings and shit
/* 0102: set a blank startup-, home- and newtab page ***/
user_pref("browser.startup.page", 0); // 0=blank, 1=home, 2=last visited, 3=resume previous session
user_pref("browser.startup.homepage", "about:blank");
user_pref("browser.newtabpage.enabled", false);
user_pref("browser.newtab.preload", false);

that we don't need to mention extensions.

nah, we do .. hang on .. i'm almost there

/*** 0100: STARTUP ***/
user_pref("_user.js.parrot", "0100 syntax error: the parrot's dead!");
/* 0101: disable default browser check
 * [SETTING] General>Startup>Always check if Firefox is your default browser ***/
user_pref("browser.shell.checkDefaultBrowser", false);
/* 0102: set start page (0=blank, 1=home, 2=last visited page, 3=resume previous session)
 * [NOTE] see next post earthlng will explain
 * [SETTING] General>Startup>When Firefox starts>Restore previous session
 * [SETTING-ESR60] General>Startup>When Firefox starts ***/
   // user_pref("browser.startup.page", 0);
/* 0103: set your "home" page (see 0102) or use an extension
 * about:home=Activity Stream (default, see 0514), custom URL, about:blank
 * [SETTING] Home>New Windows and Tabs>Homepage and new windows ***/
   // user_pref("browser.startup.homepage", "about:blank");
/* 0104: set your "newtab" page
 * true=Activity Stream (default, see 0514), false=blank page (or set by an extension)
 * [SETTING] Home>New Windows and Tabs>New tabs ***/
user_pref("browser.newtabpage.enabled", false);
user_pref("browser.new

my overrides
user_pref("browser.startup.page", 1);
user_pref("browser.startup.homepage", "moz-extension://8ablahblah");

more refinement coming

^ godamnit - my autocopy meant as I selected stuff to remove, it autocopied it and I pasted the same shit in - read it again in case

I want them separate: Start, Home+NewWindow + NewTab
No need to explain anything else. We enforce everything as about:blank. I'm happy with the final draft I just edited - nits?

edited: final v2 (merged the two settings in 0102)
/*** 0100: STARTUP ***/
user_pref("_user.js.parrot", "0100 syntax error: the parrot's dead!");
/* 0101: disable default browser check
 * [SETTING] General>Startup>Always check if Firefox is your default browser ***/
user_pref("browser.shell.checkDefaultBrowser", false);
/* 0102: set START page (0=blank, 1=home, 2=last visited page, 3=resume previous session)
 * [SETTING] General>Startup>When Firefox starts ***/
user_pref("browser.startup.page", 0);
/* 0103: set HOME+NEWWINDOW page
 * about:home=Activity Stream (default, see 0514), custom URL/extension, about:blank
 * [SETTING] Home>New Windows and Tabs>Homepage and new windows ***/
user_pref("browser.startup.homepage", "about:blank");
/* 0104: set NEWTAB page (an extension will ignore this value)
 * true=Activity Stream (default, see 0514), false=blank page
 * [SETTING] Home>New Windows and Tabs>New tabs ***/
user_pref("browser.newtabpage.enabled", false);
user_pref("browser.newtab.preload", false);

my overrides
user_pref("browser.startup.page", 1);
user_pref("browser.startup.homepage", "moz-extension://8ablahblah");

no need for explanations, no need to mention versions, no need to mention how the code works which will probably change
:+1: or :+1:  (no thumbs down allowed) and then I will commit

PS: we can relook at the settings tags when we get an ESR60 portable

@earthlng close this if you're happy and I'll do a 61-beta release

We could add this to 5000s

// user_pref("browser.tabs.remote.warmup.enabled", false); // disable tab warmup (FF57+)

Or add it to 0300s (quiet fox) with all the other "speculative" connections: IMO it is speculative, because you may not even want to load that tab or it caused by an errant wayward mouse - and what about mousing over a tab (let's say you have dozens open) just to get the page title
We could also add a link to https://www.ghacks.net/2018/01/12/firefox-tab-warming-explained/ if you want
I'm not 100% sure exactly when this works. I mean, this must only apply to people who load sessions, right? i.e the tabs haven't been loaded yet. Because we don't use sessions in our user.js (but I don't mind the fallback of this pref for users). And maybe it applies as well when users "open all" for a bunch of bookmarks etc. IDK.
@earthlng

AFAIK tab warmup is not about making connections and stuff. It's about doing the visual parsing and processing of the page, DOM, etc so that it displays faster when you switch to that tab. We don't need to add it because it's probably not a big issue and most people are so performance hungry these days.

ah OK .. i thought wanted it added as per an earlier comment.. no worries


more arguments ??

test_init (tests.test_policies.test_lstm_policy.TestLSTMPolicy) ... ERROR
======================================================================
ERROR: test_intrinsic_reward_and_gsum_calculation (tests.test_policies.test_feudal_batch_processor.TestFeudalBatchProcessor)
Traceback (most recent call last):
File "/Users/user/Desktop/feudal_networks/tests/test_policies/test_feudal_batch_processor.py", line 154, in test_intrinsic_reward_and_gsum_calculation
b = Batch(obs, a, returns, terminal, s, g, features)
TypeError: new() takes exactly 7 arguments (8 given)
Not sure why theres more arguments than what it can take.

The code is still in development, we'll update the readme when everything is working!

Thanks, looking forward to the full working code.


Hugo 0.58 - Use RegularPages if Available

Upgrading Hugo to 0.58.1 caused issues for me related to #138  and https://github.com/gohugoio/hugoThemes/issues/682.
This PR fixes my issues, and to the best of my knowledge (testing it with v0.57.2), it is backwards compatible.
Behavior without any changes:

Behavior with my changes:

Behavior with Best posts:

Edited to remove issues I was having with Best posts.

https://github.com/mtn/cocoa-eh-hugo-theme/pull/144 fixed this. closing


build: update travis to build against go 1.14

Panic in the channeldb tests:
fatal error: checkptr: unsafe pointer conversion

goroutine 4 [running]:
runtime.throw(0xb08ed5, 0x23)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/runtime/panic.go:1112 +0x72 fp=0xc0000b35d0 sp=0xc0000b35a0 pc=0x462da2
runtime.checkptrAlignment(0xc000027590, 0xabba60, 0x1)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/runtime/checkptr.go:18 +0xb7 fp=0xc0000b3600 sp=0xc0000b35d0 pc=0x434477
github.com/coreos/bbolt.(*Bucket).write(0xc0000b3778, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/coreos/bbolt@v1.3.3/bucket.go:624 +0x15c fp=0xc0000b3668 sp=0xc0000b3600 pc=0x8b8d1c
github.com/coreos/bbolt.(*Bucket).CreateBucket(0xc00019c0f8, 0xfe5c40, 0x10, 0x10, 0xc0001b01b8, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/coreos/bbolt@v1.3.3/bucket.go:181 +0x465 fp=0xc0000b3828 sp=0xc0000b3668 pc=0x8b57a5
github.com/coreos/bbolt.(*Tx).CreateBucket(...)
	/home/travis/gopath/pkg/mod/github.com/coreos/bbolt@v1.3.3/tx.go:108
github.com/lightningnetwork/lnd/channeldb.createChannelDB.func1(0xc00019c0e0, 0xc0001b0101, 0xc00019c0e0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/channeldb/db.go:259 +0x96 fp=0xc0000b3998 sp=0xc0000b3828 pc=0x9faab6
github.com/coreos/bbolt.(*DB).Update(0xc0001b0000, 0xc3dbd0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/coreos/bbolt@v1.3.3/db.go:694 +0xec fp=0xc0000b3a40 sp=0xc0000b3998 pc=0x8c28fc
github.com/lightningnetwork/lnd/channeldb.createChannelDB(0xc000027520, 0x17, 0x0, 0xc000028960)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/channeldb/db.go:258 +0x129 fp=0xc0000b3ae0 sp=0xc0000b3a40 pc=0x9f9519
github.com/lightningnetwork/lnd/channeldb.Open(0xc000027520, 0x17, 0x0, 0x0, 0x0, 0x17, 0x0, 0x0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/channeldb/db.go:150 +0x96f fp=0xc0000b3cc0 sp=0xc0000b3ae0 pc=0x9f937f
github.com/lightningnetwork/lnd/channeldb/migtest.MakeDB(0x9ff7f6, 0xc00004f568, 0xdf0588, 0xc58b74)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/channeldb/migtest/migtest.go:25 +0xa7 fp=0xc0000b3d50 sp=0xc0000b3cc0 pc=0x9fe467
github.com/lightningnetwork/lnd/channeldb/migtest.ApplyMigration(0xc000153440, 0xc3dbe8, 0xc3dbf0, 0xc3dbd8, 0x0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/channeldb/migtest/migtest.go:44 +0x48 fp=0xc0000b3e90 sp=0xc0000b3d50 pc=0x9fe678
github.com/lightningnetwork/lnd/channeldb/migration12_test.TestTLVInvoiceMigration.func1(0xc000153440)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/channeldb/migration12/migration_test.go:197 +0x66 fp=0xc0000b3ed0 sp=0xc0000b3e90 pc=0x9ff7f6
testing.tRunner(0xc000153440, 0xc0000ff4d0)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/testing/testing.go:992 +0x1ec fp=0xc0000b3fd0 sp=0xc0000b3ed0 pc=0x58216c
runtime.goexit()
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000b3fd8 sp=0xc0000b3fd0 pc=0x496681
created by testing.(*T).Run
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/testing/testing.go:1043 +0x661

Current candidate PR for the fix in bbolt: https://github.com/etcd-io/bbolt/pull/201

Updated to the latest version of bolt and walletdb, which now have the 1.14 fix.

As mentioned offline, everything seems ok, but we have new this error when running the tests without the race condition detector on:
runtime: pointer 0xc0002ee000 to unallocated span span.base()=0xc0002ee000 span.limit=0xc0002effe0 span.state=0
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)
runtime stack:
runtime.throw(0xa262de, 0x3e)
	/home/travis/.gimme/versions/go1.14.1.linux.amd64/src/runtime/panic.go:1114 +0x72 fp=0x7f5f504afde8 sp=0x7f5f504afdb8 pc=0x436202
runtime.badPointer(0x7f5f77f866d0, 0xc0002ee000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.14.1.linux.amd64/src/runtime/mbitmap.go:380 +0x230 fp=0x7f5f504afe30 sp=0x7f5f504afde8 pc=0x416140
runtime.findObject(0xc0002ee000, 0x0, 0x0, 0xc0002edfc0, 0x7f5f77e08450, 0x7f)
	/home/travis/.gimme/versions/go1.14.1.linux.amd64/src/runtime/mbitmap.go:416 +0x9b fp=0x7f5f504afe68 sp=0x7f5f504afe30 pc=0x4161eb
runtime.wbBufFlush1(0xc000044800)
	/home/travis/.gimme/versions/go1.14.1.linux.amd64/src/runtime/mwbbuf.go:288 +0xa5 fp=0x7f5f504afec0 sp=0x7f5f504afe68 pc=0x430675
runtime.wbBufFlush.func1()
	/home/travis/.gimme/versions/go1.14.1.linux.amd64/src/runtime/mwbbuf.go:218 +0x3a fp=0x7f5f504afed8 sp=0x7f5f504afec0 pc=0x463f4a
runtime.systemstack(0x7ffe80ab5890)
	/home/travis/.gimme/versions/go1.14.1.linux.amd64/src/runtime/asm_amd64.s:370 +0x66 fp=0x7f5f504afee0 sp=0x7f5f504afed8 pc=0x465e36
runtime.mstart()
	/home/travis/.gimme/versions/go1.14.1.linux.amd64/src/runtime/proc.go:1041 fp=0x7f5f504afee8 sp=0x7f5f504afee0 pc=0x43b010

Not sure what to attribute this to yet. Either the bbolt fix uncovered another underlying issue, or there's an issue with our access patterns as is (possibly using read bytes without copying them first).

Wish I had seen this sooner. We've been tracking this for the past few days as well.
Here's a minimal reproducer for bbolt: https://github.com/matheusd/bbolt-sweep-bug
And here's something that seems to be fixing the tests in dcrlnd: https://github.com/etcd-io/bbolt/compare/master...matheusd:bad-arith-error

@matheusd allegedly that issue above was fixed in Go 1.14.2. However, we're still running into this issue on top of that: https://github.com/etcd-io/bbolt/issues/214

@Roasbeef go 1.4.2 by itself doesn't seem to solve the issue for me but the most recent recent commit on https://github.com/jrick/bbolt/tree/memfix does.

Pushed up a new version that points to my fork (https://github.com/Roasbeef/btcwallet/commit/fdbd4946c409852964dc896525299e1a8c611340) which uses jrick's bbolt fix branch. Will monitor the build (restarting over n over) to see if the issue pops up again.

Updated now that the bbolt patch has landed. Also updated Github Actions (I think...).

fatal error: checkptr: unsafe pointer conversion

Should all be resolved now, it was still using the old version of bbolt since btcwallet hadn't been fully switched over.


[Language Text] Skip cancellation test

It is failing consistently due to a known service issue for now.

API change check
API changes are not detected in this pull request.


cocomac: Watch +1<PHONE_NUMBER>

cocomac requests the watch of the watch_number +1<PHONE_NUMBER>. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

Approved by tripleee in Charcoal HQ


Annotation: Label switches to category on save (from time to time)

After user selects a taxon, it shows as annotation label, but after a number of savings(Save as Draft) or page refreshing the label switches back to category.
Issue with DataAttributes field?

fixed


📚 - StackBlitz example not working for Notification Service -> Component

What is the affected URL?
https://taiga-ui.dev/services/notifications-service
Description
Run the StackBlitz link for component example https://taiga-ui.dev/services/notifications-service :

Error:
Import error, can't find file:
../../alert-example/alert-example.component

Which browsers have you used?

[X] Chrome
[ ] Firefox
[ ] Safari
[ ] Edge

Which operating systems have you used?

[ ] macOS
[X] Windows
[ ] Linux
[ ] iOS
[ ] Android

P.S. Component with data example also does not work.

Please, check it: https://taiga-ui.dev/next/services/notifications-service

@rvalitov
Thanks for the bug report!
It will not work in next version of our demo app.
I will fix it later.


I would like to add azure inventory Services Azure Database for PostgreSQL flexible servers.

I would like to add azure inventory Services Azure Database for PostgreSQL flexible servers.

Hello there @Weerayug
I just created the PostgreSQL Flexible Server module.
Can you test the script now and see if it works?
Thanks

Hello K. @Claudio-Merola
Tested PostgreSQL Flexible Server module and nothing happened.
thank you very much

Hello @Weerayug,
Nothing happened? It didn't work?
Did you cloned the repo and tried that latest version of the main script?

Hello @Claudio-Merola
I'm sorry for the late reply.
Thanks, I tested it and it works great.
thank you very much

thank you very much


Heat mode not selectable

I'm trying to operate an ASYG12KMCC unit using a UTY-TFSXW1 WiFi adaptor from homeassistant on HA Yellow. I can adjust temperatures and presets fine, and can change mode from Heat to Off successfully, but am unable to change mode to Heat from any other mode (critically, from 'Off' 😅 ).
Auto, Cool etc can be selected from HA, but not Heat; the mode just reverts to whatever was previous.
Heat can be selected from the FGLAir app and this gets updated in HA.
I've tried restarting HA and haven't been able to replicate the alternating success of #https://github.com/bigmoby/fglair_for_homeassistant/issues/36
so opened this as a new issue on advice of @mtbBrian who has a similar issue with their ASU12RLS3Y
Thanks again for this project though - even before getting to the potential for automations, apart from this issue it's so much more responsive than the FGLAir app!

Hi @andhya I refactored the code in the operation mode function, please try this: https://github.com/bigmoby/fglair_for_homeassistant/releases/tag/v0.2.6
Give me a feedback if it solve, please

Just updated and operated remotely - it sticks in the UI and matches and follows the FGLAir app, so looks very promising, thanks @bigmoby !
Just need double check that the unit operates as expected when I get home, but given it's agreeing with the FGLAir app it seems very likely.

All working, have sent you some beers @bigmoby thanks!

Thank you so much @andhya 🙏I really appreciate!


[stable/mysqldump] changed mysql password usage to prevent warnings

Signed-off-by: André Bauer<EMAIL_ADDRESS>
What this PR does / why we need it:

changed mysql password usage to prevent warnings
used labels the helm 2.14 way

Which issue this PR fixes
(optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged)

fixes #

Special notes for your reviewer:
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] DCO signed
[x] Chart Version bumped
[x] Variables are documented in the README.md
[x] Title of the PR starts with chart name (e.g. [stable/chart])

/assign @axdotl

/lgtm


We don't want them to look too GO-like, since we still need to make them distinctively ad-like.
There must be a middle-ground we can do?

dark-theme ads? White text on black blackground? (won't work for the drake meme ad, but works for text ads)

Fixed in #2103


tests/utils: use CARGO_BIN_EXE* to find tinty executable

Assuming that the executable always lives in ./target/release/tinty is not sound depending on the Cargo setup. Instead, one should rely on existing environment variables to be more robust (see https://doc.rust-lang.org/cargo/reference/environment-variables.html)

Thanks! I'll update to fix this


feat: add unique ids to generated html

Closes #15
I investigated whether there was an easy 'click to link to this admonition', but it's not clear to me what the UX of this should be, so that's open if anyone wants to work on it.

Probably best to also put a hyperlink  tag on the title pointing to the id, just like MarkDown titles.  This way it is much easier to "Copy Link".
As for the particular id itself, it doesn't really matter, as any non-conflicting ID would work fine.  The current implementation seems fine to me!

What is this?

Probably best to also put a hyperlink  tag on the title pointing to the id, just like MarkDown titles. This way it is much easier to "Copy Link".

Agreed - however this will require some level of styling/css changes which I don't have bandwidth for at the moment:

wrapping the title in an a tag causes some formatting issues that need to be ironed out
what happens if you have a hyperlink in the title already

maybe an icon that appears on hover would be better

For this PR I'll just merge the id attachment as that's simple and uncontroversial - if anyone wants to contribute a copy-on-link feature I'd be very open to it!

Without an anchor tag, some browsers (e.g. Chrome & FireFox) don't give a Copy Link Address option, which makes it difficult to actually get that auto-generated link id.
I've found that the following seems to work well enough, which is how mdbook does it for headings:

The admonition title


[Dev Env] CORS Error: API Endpoints Not Returning Access-Control-Allow-Origin

Seems like somehow Dev API is having CORS issue:

Thank you!

The backend lambda was failing due to a bad python import. Fixed by https://github.com/chanzuckerberg/corpora-data-portal/pull/761

Woohoo! Thanks so much, @Bento007 !! 🎉 🎉 🎉 🎉 🎉


drivers: neural_net: Use correct Kconfig variable

The Kconfig variable for selecting neural network accelerator driver(s)
was changed from NEURAL_NET to NEURAL_NET_ACCEL
The CMakeLists.txt however, was not updated to use the updated variable
This PR updates CMakeLists.txt to include neural_net if
CONFIG_NEURAL_NET_ACCEL is defined.

Codecov Report

Merging #11270 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #11270   +/-   ##
=======================================
  Coverage    53.1%    53.1%
=======================================
  Files         218      218
  Lines       26887    26887
  Branches     5954     5954
=======================================
  Hits        14277    14277
  Misses      10174    10174
  Partials     2436     2436

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 896472a...6da35dd. Read the comment docs.


Added Dimensioned Containers and Pipelines guides

Title says it all.

Looks good at a glance. I'll merge and try them out myself from master.


support latest built_redux version 5.0.0

add test
add generic types to Thunk typedef and thunkMiddleware
add example to README.md

awesome, thanks!


Error in make on Mac OS

in /similarity_search/include/method/hnsw.h
: error: use of undeclared identifier '_MM_HINT_T0' _mm_prefetch((char _)(_iter)->getData(), _MM_HINT_T0);
-did notice issue for testing Mac OS on new version.
-issues also arise with "__m128" and others in /similarity_search/src/method/hnsw_distfunc_opt.cc when bypassing the previous errors with a workaround

Hi, thank you for reporting. Yes, the current version may not build on Mac. Our next goal is to make it fully Mac compatible. HNSW also won't work properly if you build using Clang. We are working on fixing this.
BTW, which compiler do you use?

i was using clang. thank you.

@chaz9456 could you try the develop branch? if clang doesn't work could you try gcc?
BTW, if you don't want to build fancy stuff, there are no dependencies on 3rd party libraries.
Thank you!

It looks like the build should work now thanks to @bileg fixes. I am closing the issue.

Thanks! can't wait to give it a try.

Actually @bileg used Clang, but if it works with GCC, it would be a nice confirmation.

I still get this error on both master and dev running macOS Sierra. Have manually installed gcc via HomeBrew and run cmake with -DCMAKE_C_COMPILER=/usr/bin/gcc` to no avail.

@ianseyer which branch?
Thanks!

Hi @ianseyer thanks to @bileg this should now work in master. If it doesn't feel free to re-open the ticket and bug us.


update package.json and year in .eslintrc.json

No clue how exactly yarn works so I'm hoping that these changes will still apply since apparently yarn is being used now. I had no problems with npm install after these changes, didn't try it beforehand.
npm run lint:

By adding --ext .js the linter will now only check files ending in that extension, which covers more than just ignoring .json files did.
By adding node_modules/.bin/ to the start eslint will now run the "local" eslint installed by npm install or by however yarn does it, and not require you to install and update eslint globally.

Dependency changes:

I updated all of the deps that were outdated (so almost all of them lol).
uws is unstable and isn't a massive speed boost, so I removed it in favor of zlib-sync which allows discord to send DEA compressed data which is a much more noticeable speed boost with no loss in stability.
I updated patron.js with all of the good changes from the fork DEA was using before, so I switched it back to the main patron.js github. If you find any more bugs please report them to me and I'll push a fix ASAP.

The main Patron.js doesn't fully support Discord.js master yet.  I was not sure if GuildMember#nickname would return either null or undefined so I used != null but it seems to indefinitely return undefined when they don't have a nickname.  This means that any Discord.js type reader that uses nicknames needs to check if the nickname is undefined rather than null.
For reference
Also an important reason why we're using this fork is because I was planning on reworking the cooldown system.  Command#cooldown would take either a number or object (was considering just making it accept only an object) where the object would contain a couple properties.  The main 2 would be limit, the number of times they can use the command without triggering the cooldown, and time, the length of the cooldown.
If the main version fixes the nickname thing and adds the reworked cooldown system I will merge.

For now I'll just change this PR back to the fork, besides that one problem it's good otherwise correct?

Also just dealt with that d.js bug, I just replaced all null/undefined checks with != null because the speed decrease is too minor to care about it and ensures there's never any confusion on which one is being used to represent the absence of a value.

ETA on merging this? I already had to rebase to get rid of merge conflicts once, would like to not have to do that more.