WIP: Feat: Use best practice ciphers in 2020/21

This PR updates the cipherlists based on the guidance from Mozilla and OWASP best practices.
I've referenced this Feb 2018 discussion and PR for guidance on what needed to be updated.
Modern TLS_LEVEL
OWASP cipherlists(github repo) last updated April 2019 according to git blame, commit. OWASP B (Broad Compatibility) has been chosen.

Includes solely the strongest Perfect Forward Secrecy (PFS) ciphers.
Protocols: TLSv1.2 (and newer or better).

OWASP B - Cipherlist TLS 1.2
DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256

Multi-line for better scanning over:
DHE-RSA-AES256-GCM-SHA384:
DHE-RSA-AES128-GCM-SHA256:
ECDHE-RSA-AES256-GCM-SHA384:
ECDHE-RSA-AES128-GCM-SHA256:
DHE-RSA-AES256-SHA256:
DHE-RSA-AES128-SHA256:
ECDHE-RSA-AES256-SHA384:
ECDHE-RSA-AES128-SHA256

Mozilla cipherlists last updated Jan 2020 (guidelines 5.4). Intermediate configuration has been chosen:

All cipher suites are forward secret and authenticated
Protocols: TLS 1.2, TLS 1.3
Certificate type: ECDSA (P-256) (recommended), or RSA (2048 bits)
While the goal is to support a broad range of clients, we reasonably disable a number of ciphers that have little support (such as ARIA, Camellia, 3DES, and SEED)

Mozilla Intermediate - Cipherlist TLS 1.2
ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384

Multi-line for better scanning over:
ECDHE-ECDSA-AES128-GCM-SHA256:
ECDHE-RSA-AES128-GCM-SHA256:
ECDHE-ECDSA-AES256-GCM-SHA384:
ECDHE-RSA-AES256-GCM-SHA384:
ECDHE-ECDSA-CHACHA20-POLY1305:
ECDHE-RSA-CHACHA20-POLY1305:
DHE-RSA-AES128-GCM-SHA256:
DHE-RSA-AES256-GCM-SHA384

See the links for more details about the advised cipherlists and compatibility / rationale.

Breakdown of changes to the `modern` cipherlist
Moved to the front of the list:
ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-GCM-SHA256

AES128-GCM-SHA256 may be weaker but is still considered quite secure, thus has been positioned to the front of the list for performance reasons according to mozilla. If clients were permitted to select the cipher preference, they could ask for something stronger(or chacha if no AES-NI support?).
Neither OWASP or Mozilla include the following two ciphers, so they've been removed. I think they may have been in a previous cipherlist from Mozilla when this list was last updated?
ECDHE-ECDSA-AES256-SHA384
ECDHE-ECDSA-AES128-SHA256

Mozilla ends its list with these two:
DHE-RSA-AES128-GCM-SHA256
DHE-RSA-AES256-GCM-SHA384

OWASP B adds non-GCM DHE variants, and retains the last two ciphers in the current list. These are the only difference between OWASP A and B grades, so could be dropped if choosing to opt for grade A instead:
DHE-RSA-AES256-SHA256
DHE-RSA-AES128-SHA256
ECDHE-RSA-AES256-SHA384
ECDHE-RSA-AES128-SHA256

OWASP B also lists 3 TLS 1.3 ciphers, but these rarely need to be specified for most software, as such they're not included:
TLS_AES_256_GCM_SHA384
TLS_CHACHA20_POLY1305_SHA256
TLS_AES_128_GCM_SHA256

If any reviewer is not confident about understanding the cipher suite names, this is a good article on the topic, Wikipedia also has a table on ciphers, which are apparently all covered through TLS 1.0-1.2?

Reduced cipherlist excludes
# From:
smtpd_tls_exclude_ciphers = aNULL, LOW, EXP, MEDIUM, ADH, AECDH, MD5, DSS, ECDSA, CAMELLIA128, 3DES, CAMELLIA256, RSA+AES, eNULL
# To (with no apparent change in cipher support):
smtpd_tls_exclude_ciphers = aNULL, SEED, CAMELLIA, RSA+AES

Mozilla doesn't include smtpd_tls_exclude_ciphers with their postfix config generator. While it's possible to find relatively recent blogs / guides making use of cipher excludes, they often don't explain any useful reasoning for the choices.
I think it has just become a bit of parroting of information and a "better safe than sorry if I don't understand this" approach. Looking at the postfix docs we have:

List of ciphers or cipher types to exclude from the SMTP server cipher list at all TLS security levels.

Excluding valid ciphers can create interoperability problems. DO NOT exclude ciphers unless it is essential to do so.

Cipher types listed in smtpd_tls_mandatory_exclude_ciphers or smtpd_tls_exclude_ciphers are excluded from the base definition of the selected cipher grade.

The underlying cipherlists for grades other than null include anonymous ciphers, but these are automatically filtered out if the server is configured to ask for remote SMTP client certificates. You are very unlikely to need to take any steps to exclude anonymous ciphers, they are excluded automatically as required.

Additionally aNULL also excludes ADH.
I have verified parity of the current excluded ciphers with this reduced set. It should be clear which ciphers listed are now responsible for, except for MEDIUM which handled two SEED ciphers. Since MEDIUM is ambiguous to what ciphers it covers (the other exclusions are still required for achieving parity), I have swapped MEDIUM to SEED.
For anyone curious, MEDIUM as an exclude, does not appear to exclude medium cipherlist, which I was initially concerned about that emptying out port 25 inbound cipherlist for STARTTLS, which would have meant never using TLS , thankfully that's not the case :sweat_smile:
In my testing these excluded ciphers are still graded as A, only the anon ciphers that aNULL excludes provide a lower F grade, but the postfix docs have stated it's a non-issue when dealing with remote connections. (TODO: only verified on the docker host, need to check remotely to confirm)
If we followed the mozilla generator config, and used tls_medium_cipherlist instead of tls_high_cipherlist, having excludes would not be necessary(they don't seem to be for our modern TLS_LEVEL, but still relevant with the current intermediate TLS_LEVEL unless we share the mandatory high cipherlist as medium instead).
Testing reduced cipher excludes for parity
nmap used to scan on port 25(mapped to docker-mailserver container), command ran on the docker host(<IP_ADDRESS> being all interfaces on IPv4):
nmap --script ssl-enum-ciphers -p 25 <IP_ADDRESS>

I went with what I saw first and worked for me based on this answer, which now viewing comments may suggest some further testing should be done with other tooling. My nmap package (Manjaro) included the mentioned script, I did notice no TLSv1.3 results, happy to test more with another tool if desired.

Results for current DockerHub image:
PORT   STATE SERVICE
25/tcp open  smtp
| ssl-enum-ciphers:
|   TLSv1.2:
|     ciphers:
|       TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (ecdh_x25519) - A
|       TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (dh 2048) - A
|       TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (ecdh_x25519) - A
|       TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (dh 2048) - A
|       TLS_DHE_RSA_WITH_AES_256_CCM_8 (dh 2048) - A
|       TLS_DHE_RSA_WITH_AES_256_CCM (dh 2048) - A
|       TLS_ECDHE_RSA_WITH_ARIA_256_GCM_SHA384 (ecdh_x25519) - A
|       TLS_DHE_RSA_WITH_ARIA_256_GCM_SHA384 (dh 2048) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (ecdh_x25519) - A
|       TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 (dh 2048) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (ecdh_x25519) - A
|       TLS_DHE_RSA_WITH_AES_256_CBC_SHA (dh 2048) - A
|       TLS_RSA_WITH_ARIA_256_GCM_SHA384 (rsa 2048) - A
|       TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (ecdh_x25519) - A
|       TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (dh 2048) - A
|       TLS_DHE_RSA_WITH_AES_128_CCM_8 (dh 2048) - A
|       TLS_DHE_RSA_WITH_AES_128_CCM (dh 2048) - A
|       TLS_ECDHE_RSA_WITH_ARIA_128_GCM_SHA256 (ecdh_x25519) - A
|       TLS_DHE_RSA_WITH_ARIA_128_GCM_SHA256 (dh 2048) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (ecdh_x25519) - A
|       TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (dh 2048) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (ecdh_x25519) - A
|       TLS_DHE_RSA_WITH_AES_128_CBC_SHA (dh 2048) - A
|       TLS_RSA_WITH_ARIA_128_GCM_SHA256 (rsa 2048) - A
|     compressors:
|       NULL
|     cipher preference: server
|_  least strength: A

Using the user-patches.sh support, I've tested some modifications. For anyone reproducing, note that at least with docker-compose, CTRL+C to stop the container after docker-compose up is insufficient, the main.cf file will still remain with the modifications (which prevents some other snippets shown here from working as they can no longer find the target line that got replaced), you need to make use of docker-compose down before the next up run.

Results (diff) with `smtpd_tls_exclude_ciphers` removed:
```
# user-patches.sh
sed -i "s/^smtpd_tls_exclude_ciphers.*/# nope/" /etc/postfix/main.cf
```
5a6,11
> |       TLS_DH_anon_WITH_AES_256_GCM_SHA384 - F
> |       TLS_DH_anon_WITH_AES_256_CBC_SHA256 - F
> |       TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA256 - F
> |       TLS_ECDH_anon_WITH_AES_256_CBC_SHA - F
> |       TLS_DH_anon_WITH_AES_256_CBC_SHA - F
> |       TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA - F
15a22,23
> |       TLS_ECDHE_RSA_WITH_CAMELLIA_256_CBC_SHA384 (ecdh_x25519) - A
> |       TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA256 (dh 2048) - A
17a26,29
> |       TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA (dh 2048) - A
> |       TLS_RSA_WITH_AES_256_GCM_SHA384 (rsa 2048) - A
> |       TLS_RSA_WITH_AES_256_CCM_8 (rsa 2048) - A
> |       TLS_RSA_WITH_AES_256_CCM (rsa 2048) - A
18a31,40
> |       TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 2048) - A
> |       TLS_RSA_WITH_CAMELLIA_256_CBC_SHA256 (rsa 2048) - A
> |       TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A
> |       TLS_RSA_WITH_CAMELLIA_256_CBC_SHA (rsa 2048) - A
> |       TLS_DH_anon_WITH_AES_128_GCM_SHA256 - F
> |       TLS_DH_anon_WITH_AES_128_CBC_SHA256 - F
> |       TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA256 - F
> |       TLS_ECDH_anon_WITH_AES_128_CBC_SHA - F
> |       TLS_DH_anon_WITH_AES_128_CBC_SHA - F
> |       TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA - F
26a49,50
> |       TLS_ECDHE_RSA_WITH_CAMELLIA_128_CBC_SHA256 (ecdh_x25519) - A
> |       TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA256 (dh 2048) - A
28a53,56
> |       TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA (dh 2048) - A
> |       TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 2048) - A
> |       TLS_RSA_WITH_AES_128_CCM_8 (rsa 2048) - A
> |       TLS_RSA_WITH_AES_128_CCM (rsa 2048) - A
29a58,64
> |       TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 2048) - A
> |       TLS_RSA_WITH_CAMELLIA_128_CBC_SHA256 (rsa 2048) - A
> |       TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
> |       TLS_RSA_WITH_CAMELLIA_128_CBC_SHA (rsa 2048) - A
> |       TLS_DH_anon_WITH_SEED_CBC_SHA - F
> |       TLS_DHE_RSA_WITH_SEED_CBC_SHA (dh 2048) - A
> |       TLS_RSA_WITH_SEED_CBC_SHA (rsa 2048) - A
33c68
< |_  least strength: A
---
> |_  least strength: F

Note in the diff that the only ciphers that were rated below A were the anonymous ciphers which aNULL excludes. We've already learned from postfix docs that they shouldn't be a concern. Thus it may be fine to just not exclude any ciphers when the server chooses?
Mozilla excludes some with the following reasoning:

While the goal is to support a broad range of clients, we reasonably disable a number of ciphers that have little support (such as ARIA, Camellia, 3DES, and SEED)

Results for current DockerHub image when testing the high cipherlist:
```
# user-patches.sh
sed -i "s/^smtpd_tls_exclude_ciphers.*/# nope/" /etc/postfix/main.cf
sed -i "s/^tls_high_cipherlist/tls_medium_cipherlist/" /etc/postfix/main.cf
```
PORT   STATE SERVICE
25/tcp open  smtp
| ssl-enum-ciphers:
|   TLSv1.2:
|     ciphers:
|       TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (ecdh_x25519) - A
|     compressors:
|       NULL
|     cipher preference: server
|_  least strength: A

Current tls_high_cipherlist (split to individual lines for readability):
ECDHE-ECDSA-AES256-GCM-SHA384:
ECDHE-RSA-AES256-GCM-SHA384:
ECDHE-ECDSA-CHACHA20-POLY1305:
ECDHE-RSA-CHACHA20-POLY1305:
ECDHE-ECDSA-AES128-GCM-SHA256:
ECDHE-RSA-AES128-GCM-SHA256:
ECDHE-ECDSA-AES256-SHA384:
ECDHE-RSA-AES256-SHA384:
ECDHE-ECDSA-AES128-SHA256:
ECDHE-RSA-AES128-SHA256

Note that ECDSA ciphers are not listed, despite being in the cipherlist override. I don't know this stuff that well, postfix docs seem to suggest such support is dependent on user supplied certs, but that RSA is typically what will be encountered:

RSA, DSA and ECDSA (Postfix ≥ 2.6) certificates are supported. Typically you will only have RSA certificates issued by a commercial CA. In addition, the tools supplied with OpenSSL will by default issue RSA certificates. You can configure all three at the same time, in which case the cipher used determines which certificate is presented. For Netscape and OpenSSL clients without special cipher choices, the RSA certificate is preferred.

Most clients will not be ECDSA capable, so you will likely also need an RSA or DSA certificate and private key.

That mentions Netscape as a client... so how accurate the information in those postfix docs are today is uncertain.. Mozilla does have some modern docs mentioning similar:

We recommend ECDSA certificates using P-256, as P-384 provides negligable improvements to security and Ed25519 is not yet widely supported
Certificate type: ECDSA (P-256) (recommended), or RSA (2048 bits)
ECDSA certificates are recommended over RSA certificates, as they allow the use of ECDHE with Windows 7 clients using Internet Explorer 11, as well as allow connections from IE11 on Windows Server 2008 R2
Administrators needing to provide access to IE 11 on Windows Server 2008 R2 and who are unable to switch to or add ECDSA certificates can add TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

Thus I assume the omission of ECDSA ciphers in my results is due to docker-mailserver generating an RSA cert when none is provided? Which means nmap is not able to query the full support(or rather postfix needs to be tested with an ECDSA cert provided as well?
Not that it matters.. with the current excludes, as they specifically have an exclude for ECDSA which I believe ignores any in the cipherlist override tls_high_cipherlist..

`smtpd_tls_exclude_ciphers` does affect `tls_high_cipherlist`
```
# user-patches.sh
sed -i "s/^smtpd_tls_exclude_ciphers.*/smtpd_tls_exclude_ciphers = AES128/" /etc/postfix/main.cf
sed -i "s/^tls_high_cipherlist/tls_medium_cipherlist/" /etc/postfix/main.cf
```
PORT   STATE SERVICE
25/tcp open  smtp
| ssl-enum-ciphers:
|   TLSv1.2:
|     ciphers:
|       TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (ecdh_x25519) - A
|     compressors:
|       NULL
|     cipher preference: server
|_  least strength: A

For completeness (kinda), to no surprise each of these ports return the same output(default modern TLS_LEVEL):

Ports 587, 465, 993 - Default DockerHub
PORT    STATE SERVICE
587/tcp open  submission
| ssl-enum-ciphers:
|   TLSv1.2:
|     ciphers:
|       TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (ecdh_x25519) - A
|     compressors:
|       NULL
|     cipher preference: server
|_  least strength: A

PORT    STATE SERVICE
465/tcp open  smtps
| ssl-enum-ciphers:
|   TLSv1.2:
|     ciphers:
|       TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (ecdh_x25519) - A
|     compressors:
|       NULL
|     cipher preference: server
|_  least strength: A

PORT    STATE SERVICE
993/tcp open  imaps
| ssl-enum-ciphers:
|   TLSv1.2:
|     ciphers:
|       TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (ecdh_x25519) - A
|     compressors:
|       NULL
|     cipher preference: server
|_  least strength: A

The intermediate TLS_LEVEL has had it's cipherlist truncated. None of these appear to remain available after applying the excludes on them. If the excludes are removed, each TLS level presents ciphers(when the cipherlist is only set to this removed list) with the warning:

Forward Secrecy not supported by any cipher

This is misleading when associating them with a high grade list intended for mandatory TLS support. Despite the warning, the nmap tool grades them all with with A for encryption strength(if that's at all trustworthy of a metric).
The intermediate cipherlist still produces the same cipher support with the new reduced list and exclusions.

Removed `intermediate` TLS_LEVEL ciphers as cipherlist (no excludes)
```
# user-patches.sh
sed -i "s/^smtpd_tls_exclude_ciphers.*/# nope/" /etc/postfix/main.cf
sed -i "s/^tls_high_cipherlist.*/tls_medium_cipherlist = ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS/" /etc/postfix/main.cf
```
PORT   STATE SERVICE
25/tcp open  smtp
| ssl-enum-ciphers:
|   TLSv1.0:
|     ciphers:
|       TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A
|     compressors:
|       NULL
|     cipher preference: server
|     warnings:
|       Forward Secrecy not supported by any cipher
|   TLSv1.1:
|     ciphers:
|       TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A
|     compressors:
|       NULL
|     cipher preference: server
|     warnings:
|       Forward Secrecy not supported by any cipher
|   TLSv1.2:
|     ciphers:
|       TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_GCM_SHA384 (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A
|     compressors:
|       NULL
|     cipher preference: server
|     warnings:
|       Forward Secrecy not supported by any cipher
|_  least strength: A

Changes to Consider
Change smtpd_tls_mandatory_ciphers from high to medium?
I'm having some trouble getting information about smtpd_tls_ciphers (opportunistic STARTTLS or the mandatory variant that only applies to implicit TLS).
We have it set to high while Mozilla is using medium, this could just be a lack of trust with their postfix template that was contributed, as their project is mostly focused on adding the cipher advice configs while relying on contributors for the various templates in their generator website afaik(eg, they've presently got an obsolete postfix config parameter smtpd_use_tls).
Do note what the postfix docs have to say(unsure of when those were last updated, some of the docs do seem to have outdated information, eg port 465):

The default grade ("medium") is sufficiently strong that any benefit from globally restricting TLS sessions to a more stringent grade is likely negligible, especially given the fact that many implementations still do not offer any stronger ("high" grade) ciphers, while those that do, will always use "high" grade ciphers.

So insisting on "high" grade ciphers is generally counter-productive. No known SMTP clients fail to support at least one "medium" or "high" grade cipher.

medium describes itself as "These use 128-bit or longer symmetric bulk-encryption keys.". All available cipher grades have the warning "you are strongly encouraged to not change.". Changing them based on OWASP and Mozilla advice and keeping them in sync between Postfix and Dovecot is probably fine..
There isn't much information what the default grades provide (needs to be inspected via CLI command afaik?). (EDIT: Used nmap to inspect, not sure if this is the best way)
This change would really only affect port 25 inbound traffic with STARTTLS. Ports 465 and 587 are submission ports where users are more likely to be using capable clients, TLS is mandatory for these by default, as is Dovecot(TODO: Confirm this) ports (110, 143, 993, 995) for retrieval via those same clients.
Enforcing stricter ciphers via high makes sense here, while leaving port 25 for more compatibility to receive email.

Statistics / info if useful

Google provides a transparency report about inbound/outbound traffic of their SMTP service. They don't break down encryption by protocol or ciphers however. Presently outbound is at 90%+, dropping down to 80% in a recurring monthly pattern(automated bulk, less important to worry about). Inbound is at ~93%. In the ranking section, there are some well known domains sending inbound email with low or 0% encryption, eg adobe(80%), samsung(60%), aliexpress / aol / yahoo japan all at 0%.
There is a July 2019 SMTP TLS report, which reports TLS 1.2 being negotiated 90% of the time, lower protocols were only < 3%. How likely those emals are not spam is unclear, the blog did not continue to post anymore SMTP stat updates and the twitter account no longer exists.
TLS 1.2 is the minimum supported protocol, as recommended by RFC 7525, PCI DSS, and others. RFC 7525 has declared this since 2015, any SMTP service that is legitimate should really have caught up by now as their service would be increasing in rejections(or sadly favoring plain text via STARTTLS as is required to be supported for inbound traffic on port 25).
The retirement of insecure protocols from the list of supported protocols was announced by all major browser makers including Microsoft, Google, Apple, and Mozilla back in October 2018. They all aim to accomplish that by mid 2020, but Microsoft and Mozilla have both announced delays due to covid-19. While that is browsers, still worth noting.

Allow client to choose cipher?
When allowing the client to choose the cipher from the list instead, which mozilla advises for their intermediate and modern profiles, the client can opt for what would suit it best, all ciphers the server advertises from this list are considered strong, so selection isn't a concern.
tls_preempt_cipherlist controls server/client preference in postfix. Mozilla has set it to no, which is the default in postfix docs. I have not changed it to no, as that would require port 25 inbound STARTTLS to also use the same high cipherlist to follow Mozilla's advice(or just modify cipherlist for medium and prefer that over high).
Probably a good idea to allow client cipher selection for modern profile at least?

TL;DR

The modern TLS_LEVEL cipherlist is updated to accommodate OWASP B and Mozilla Intermediate ciphers.
The intermediate TLS_LEVEL cipherlist has only received housekeeping. Removing ciphers that are presently excluded by smtpd_tls_exclude_ciphers anyway.
smtpd_tls_exclude_ciphers has also received housekeeping. It has been reduced to only the relevant excludes, matching the same cipher support presently offered - except for ECDSA, which is no longer excluded.
ECDSA, despite having ciphers for it in the cipherlist, has not been supported by this image afaik. It seems to require a user supplied ECDSA cert(RSA is default), however the exclude list ignored these ciphers from what I can tell. I've not kept it excluded as that seems like it was a bug.

Thank you for the detailed justification. For this kind of changes we should be relying on an auditing tool such as https://github.com/drwetter/testssl.sh.

For this kind of changes we should be relying on an auditing tool such as https://github.com/drwetter/testssl.sh.

Yes, that'd be a good idea. I came across that yesterday while working on this PR. It flags some features as negative, when they shouldn't be, but isn't always practical for it to know better. The linked issue is about allowing client preference for the cipher, which should be allowed when strong ciphers are only available to choose from.

Additionally, often browsers are better maintained than web servers updated. So the client's preferences are probably not bad.

I don't think this can be applied to mail clients.
Server preferred cipher order is a good practice. Mail clients are not always choosing the best cipher (android applications eyes).
While some false positive could happened when using this kind of tool, this image should stick to "security first, optimization as fine tuning".
The optimizations such as disabling server cipher preference by default order can decrease security and should be let as a fine-tuning done by the user (knowing what he's doing).

Server preferred cipher order is a good practice. Mail clients are not always choosing the best cipher

Define "best" cipher. If the OWASP ciphers are ignored, Mozilla states that all ciphers in their cipherlist are strong ciphers such that it doesn't matter which one is chosen by the client.
In that case the client can choose the cipher that is best suited for it. If they're all strong, it doesn't matter if it chooses one that it can be processed more quickly. Especially when it comes to hardware encryption support (AES-NI), chacha is said to be better here in that case, the server would not know if the client has AES-NI though would it?

this image should stick to "security first, optimization as fine tuning".

It does? Security isn't being compromised by this PR to the best of my knowledge (albeit I'm not a security professional).

The optimizations such as disabling server cipher preference by default order can decrease security

Is this referring to Mozilla's advice to push the AES128-GCM ciphers in front of the AES256-GCM ones? They do that with server preference in mind, stating it's still safe, but when the client doesn't have preference, it's best to offer up a more performant cipher(out of a list where all are considered to be fine).

I cannot say the same for the intermediate TLS_LEVEL setting in this image. That has support for ciphers less than TLS 1.2. Thus I would not advise leniency to let the client pick, only for modern.
OWASP B can drop the four non-GCM variants it adds, and we'd be using OWASP A instead. Really depends on the decision to allow modern to enable client preference and tighten up the ciphers more, possibly updating intermediate to be more strict(not sure about this for mail servers, I'm aware they lag more in support).
Do note under the statistics section in this PR notes(3rd post):

TLS 1.2 is the minimum supported protocol, as recommended by RFC 7525, PCI DSS, and others. RFC 7525 has declared this since 2015.

Granted, all modern browsers are only starting to enforce that later this year.
Other statistics show TLS 1.2 is negotiated for 90% of secured inbound traffic, where about 90%+ of inbound traffic is encrypted. Some is from valid big tech names, but as it's arriving on port 25, it will be delivered without encryption(this is on Googles SMTP servers).
Mozilla omits some of the ciphers in TLS 1.2, even though they may be still fine (postfix docs encourages not altering the cipherlist defaults it provides):

While the goal is to support a broad range of clients, we reasonably disable a number of ciphers that have little support (such as ARIA, Camellia, 3DES, and SEED)

For MUAs (users submitting / retrieving email from the mail server), we can expect stronger support, and do so via the current cipherlist override and enforcing mandatory TLS.
Client software won't necessarily improve unless it's required to, users that need broader compatibility need to choose a better client or understand the risks of poor support, and downgrade to intermediate TLS_LEVEL which should still support them fine.
Our modern is with SMTP in mind, so it's not matching Mozilla's modern which is TLS 1.3 only.

Mozilla states that all ciphers in their cipherlist are strong ciphers such that it doesn't matter which one is chosen by the client.

True, considering the cipher list is only sufficiently secured ciphers, allowing the client to choose permits him to choose the most efficient cipher for his architecture, it also prevents the client to be forced to use an unadapted cipher (e.g. the server has a weaker cipher in the top hist the list). source
Review in progress

:black_square_button: Reduced excludes in smtpd_tls_exclude_ciphers
:black_square_button: Intermediate TLS_LEVEL
:black_square_button:  Change smtpd_tls_mandatory_ciphers from high to medium?
:heavy_check_mark:  Allow client to choose cipher?

:heavy_check_mark: Allow client to choose cipher?

Needs confirmation on when that should be acceptable.
Presently, we only use our modern and intermediate cipherlists for smtpd_tls_mandatory_ciphers = high, that excludes inbound email on port 25 which uses smtpd_tls_ciphers = medium.
Especially on the intermediate TLS_LEVEL, where <TLSv1.2 is allowed, it may be best to go with the cipher order postfix medium default cipherlist, the first cipher that the client(another mail server) supports from that list would then be chosen.  Postfix defaults to letting the client choose though, and the postfix docs state

While server cipher selection may in some cases lead to a more secure or performant cipher choice, there is some risk of interoperability issues. In the past, some SSL clients have listed lower priority ciphers that they did not implement correctly. If the server chooses a cipher that the client prefers less, it may select a cipher whose client implementation is flawed.

But then goes on to say:

Most notably Windows 2003 Microsoft Exchange servers have flawed implementations of DES-CBC3-SHA, which OpenSSL considers stronger than RC4-SHA.

Neither of those ciphers should be used these days IIRC, that particular client is 17 years old, but I think the ciphers still are present in intermediate TLS_LEVEL.
However, since we're talking about only inbound port 25 (other ports are fine, they use our restricted cipherlist), flawed / insecure ciphers don't matter much, the client could always choose plain text anyway..
So just allow the client to choose ciphers? When considering users submitting / retrieving email, this is over mandatory TLS and unclear how intermediate TLS_LEVEL should be treated. I have not compared Mozilla's old configuration, but their generator will prefer server cipher selection. Thus it's probably best to follow that, or tighten mandatory protocols for intermediate TLS_LEVEL to use TLS 1.2 as a minimum there too(port 25 inbound can remain relaxed).

Good discussion and nice work. From my point of view there are two points to consider. One is backwards compatibility for the low security installations. Change those and there are immediately issues about ancient clients that no longer work. The other is good security and best practices for the high security (default) option.
I would generally let the server select the cipher. At least when it comes to browsers it is dangerous to let the client select as it can be used for attacks. We do not control the client; the attacker does. I haven't heard of that kind of attack for mail servers, but that doesn't mean it doesn't exist.
I would also prefer to have high scores in the TLS tests. Some organizations depend on getting good marks in those tests, so a top score with the default configuration is a win. Experts can change the settings, but when you don't know it is nice to get an official stamp.

One is backwards compatibility for the low security installations. Change those and there are immediately issues about ancient clients that no longer work.

Yes, that is taken into consideration. I've only modified the support for modern TLS_LEVEL.
Those using intermediate should not experience any difference, beyond the possibility of using ECDSA certificates now(assuming the exclude for that prevented ECDSA cipher suites, I would need to verify to confirm).
Other than that, the intermediate cipherlist has just removed ciphers that afaik get removed from the exclude list, so were never available in the present image(only verified via nmap as detailed in the 2nd post).
Cipher excludes were reduced, but parity checked. I may not have checked this correctly, as I've only tested locally on my machine(Manjaro, rolling release distro), if it's possible my local setup/openssl as a client would prevent nmap showing certain cipher support, then I may have made a mistake here. I've not looked into yet if excludes like MD5 have any relevance in TLS protocols, I know MD5 is insecure, but I'm assuming that must be from SSL protocols which are prohibited. I did test with no excludes and received many other ciphers listed as supported / insecure, so I just ensured based on that, the support with the reduced excludes was the same as what the image presently was showing as supported ciphers for modern and intermediate levels.

The other is good security and best practices for the high security (default) option.

This is mostly placing trust in the linked resources while I gain a better understanding myself to establish more confidence. I'm still doing some research on this and will share findings here for anyone to review.
Perhaps this PR should be changed to a draft with WIP prefix for now? I had amassed a fair amount and wanted to share it here for review / discussion in the meantime.

I would generally let the server select the cipher. At least when it comes to browsers it is dangerous to let the client select as it can be used for attacks.

As discussed with @youtous and earlier in this PRs details(it's lengthy, so I don't blame anyone for not reading through all that), mandatory TLS cipherlist (at least with Mozilla's list, unsure about additions from OWASP B) is considered that all ciphers are strong, thus it doesn't matter how they're ordered/selected (order has still been respected based on OWASP/Mozilla lists), there is no risk allowing the client to choose from the mandatory list, if anything it's considered more compatible/secure (see justification response to @youtous for details).
Mozilla is generic though, so it's also applied to port 25 inbound, using the same cipherlist as mandatory TLS. We don't do that here, as we opt for broader cipher support with third-party mail servers for receiving/delivering email. I believe it's still safe for clients to choose, at least on medium, and postfix docs also encourage it regardless.
There is no enforced TLS for receiving, nor authentication on port 25, so regardless of client cipher selection, the risk may not be there? The ports the user will interact with (465, 587, 110, 143, 993, 995), is more important for strong cipher selection.

Some organizations depend on getting good marks in those tests, so a top score with the default configuration is a win.

Feel free to list what tests you want to see run.
I don't see this PR lowering the score, other than if client selection is enabled. Some of these tests have false positives, so good scores don't necessarily indicate confidence in better security, but in passing compliance. Still they are useful as a guide.

Right. I'll be happy to merge this; however, you mentioned WIP. Is there something you want to complete first? If not I'll merge.

@erik-wramner I haven't finished any mentioned TODO parts, I've only tested on my docker host, and I'm still doing some more research before I'm more confident about this change.
It should mostly be fine but I don't think there is an immediate need to merge it, I'll ping you when I've moved on to other things and happy with this PR :)
I am aware this doesn't include updating the ciphers for the dovecot test config, I should probably do that before merging.

Good. I've changed the title to WIP, change it back when you're done please. That way I can see that it is pending.

Maybe adding smtpd_tls_eckey_file and smtpd_tls_eccert_file as an option would also be a good idea? I'm currently running my mailserver with an ECDSA cert as traefik has some issues with RSA for my use case and currently it doesn't seem to be possible to select the cert you want on a per-host basis (so i'm stuck with ECDSA for everything). However it seems like some mail servers don't like that and you get warning: TLS library problem: error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol:../ssl/statem/statem_srvr.c:1661: (to be fair most of these seem to come from spam servers from india, but I can't rule out that it affects any legitimate servers).
smtpd_tls_eckey_file in addition to smtpd_tls_key_file allows you to have an ECDSA and RSA cert as a fallback.

According to the Postfix docs, those two parameters are obsoleted by smtpd_tls_chain_files. I can look into that for you when I have some time to come back to this(hopefully sometime in July).

This is great work @polarathene - hearing you were thinking about wrapping this PR up. What's the current status?

@aendeavor if I don't have anything else come up over the next week, I should have time to allocate to wrapping this up finally :)
It's mostly good, just need to follow through with my review changes and do some testing. Sorry for the delay, I didn't expect to be so busy elsewhere.

Awesome. I will try getting the refactoring of start-mailserver.sh out of the way beforehand. This will probably mean resolving a conflict more, but is significantly less work on my side. Reach back to me when you're ready.
@erik-wramner I noticed the build on master, especially the elk build is not passing. What's up with that? And if these changes are in, is the code automatically built on DockerHub into the latest-image?

Yes, your changes should make it into latest automatically. As for elk, I think we dropped that one in 7.0 when we upgraded Debian version. The elk build should be from the 6.x release, so it is quite old. Ideally we should stop building it.

Good news, I'm ready to finish off this PR now :grinning:
I'll remove the WIP once ready for peer review/merge.

@polarathene FYI the tests are only failing because of the newly added shellcheck linting targets. You will just need to correct this:
# start-mailserver.sh line 1086
apply_tls_level $TLS_MODERN_SUITE $TLS_MODERN_IGNORE $TLS_MODERN_MIN
# needs to be
apply_tls_level "${TLS_MODERN_SUITE}" "${TLS_MODERN_IGNORE}" "${TLS_MODERN_MIN}"

# start-mailserver.sh line 1095
apply_tls_level $TLS_INTERMEDIATE_SUITE $TLS_INTERMEDIATE_IGNORE $TLS_INTERMEDIATE_MIN
# needs to be
apply_tls_level "${TLS_INTERMEDIATE_SUITE}" "${TLS_INTERMEDIATE_IGNORE}" "${TLS_INTERMEDIATE_MIN}"

due to
SC2248: Prefer double quoting even when variables don\'t contain special characters.
# and
SC2250: Prefer putting braces around variable references even when not strictly required.

Already had a look at the tests. The infamous 260 checking setup.sh: setup.sh email add and login. There is an issue for this - the easiest solution for this is another commit to trigger the rebuild.

Now failing with:

The command "make generate-accounts run generate-accounts-after-run fixtures tests" exited with 2.

Port 587 should be mandatory STARTTLS? testssl.sh seemed to flag it as opportunistic like port 25 (which isn't a real issue for that port) for some reason. IIRC, Postfix doesn't exchange credentials on 587 unless a secure connection is established, so should still be safe.

Sounds very good to me.

I'm pretty sure that 587 is mandatory STARTTLS in the released version as that question has come up before several times? Could be a false alarm. Anyway, good work on this.

I'm currently running my mailserver with an ECDSA cert as traefik has some issues with RSA for my use case and currently it doesn't seem to be possible to select the cert you want on a per-host basis (so i'm stuck with ECDSA for everything). However it seems like some mail servers don't like that and you get warning: TLS library problem: error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol:../ssl/statem/statem_srvr.c:1661: (to be fair most of these seem to come from spam servers from india, but I can't rule out that it affects any legitimate servers).

@Kiina I have tested ECDSA certs and confirmed that Postfix is presently configured to exclude all ECDSA ciphers. testssl.sh was unable to produce any results for Postfix SMTP ports (25, 587, 465) when using an ECDSA certificate instead of RSA.
Unless the mail servers sending mail to your instance are using TLS 1.3, they'd all be opting for plain text as they likely couldn't establish any TLS negotiation otherwise as your Postfix is configured to exclude ECDSA ciphers and you're not serving an RSA cert... hence the error I believe? (I get that plenty with testssl.sh when testing all ciphers, not just the ones the server reports as available)
Good news is I've implemented dual cert support so you can provide the RSA fallback, Dovecot will use it too(Dovecot isn't excluding ECDSA ciphers so any MUA should already be fine securely retrieving mail). You'll still need this PR as well for ECDSA to be supported with Postfix (or remove the ECDSA  from smtpd_tls_exclude_ciphers in main.cf).

I am still actively working on this, apologies for the delay. I've been collecting and documenting a rather large amount of information regarding TLS version history, usage and cipher suite compatibility.
I'll be sharing it here once it's ready as a rather comprehensive resource on the topic. It should be useful to anyone else that needs to make such a decision with confidence. If there's a need to get this PR wrapped up sooner, do let me know, otherwise I'm estimating 1-2 weeks at the latest (depending on free time I can allocate).

We may want to discuss deprecating intermediate / legacy client support. There's an RFC in progress since 2018 and expected to be completed next year that officially advise no longer supporting these protocol versions with any usage of TLS.
OpenSSL 3.0 (expected to be released early 2021H1) is said to be removing TLS 1.0 & 1.1 cipher suites by default, we would need to explicitly lower the security level with :@SECLEVEL=0 appended to the end of our cipherlists (assuming Postfix/Dovecot continue to release with support for using these TLS versions once that RFC is published).
Perhaps something to consider for 8.0. My full report will provide more details to support the decision, the only client that seems to be a valid concern is macOS/iOS releases prior to Sep 2016 which use Apple Mail instead of an alternative like Mozilla ThunderBird. Servers (for which it matters less with port 25 deliveries) should be good from 2014 onwards (as-in the majority of deployments should have no issues for TLS 1.2 support by then).

I've marked it for 7.3, so there is no need to hurry, since the next release, i.e. 7.2, comes in two weeks. Therefore, 7.3 has a big amount of time, and if you need more, I'll mark it for later :D

We may want to discuss deprecating intermediate / legacy client support. There's an RFC in progress since 2018 and expected to be completed next year that officially advise no longer supporting these protocol versions with any usage of TLS.

If you ask me, I'm all for it. I know there is a never-ending debate on supporting old clients and versions, I'm a supporter of having servers and clients up-to-date. I'd go for it.

OpenSSL 3.0 (expected to be released early 2021H1) is said to be removing TLS 1.0 & 1.1 cipher suites by default, we would need to explicitly lower the security level with :@SECLEVEL=0 appended to the end of our cipherlists (assuming Postfix/Dovecot continue to release with support for using these TLS versions once that RFC is published).

That's one of the reasons I'd like to be up-to-date withut support for outdated versions.

I know there is a never-ending debate on supporting old clients and versions

Yes, this is usually due to mail clients (MUA and MTA) lagging behind browsers TLS support.
Often I only ever saw the concern expressed for legacy clients without much knowledge as to what fell under that umbrella and when, so I've looked into that with references for the majority of cases where that'd have been a concern, be that support for TLS version, cipher suites or ECC (ECDSA / ECDHE).
I'm rather confident we can issue a deprecation notice to drop legacy support from 8.0 release and require anyone wanting to support legacy to use 7.x or a user-patches script (we could document an example in the wiki). Should I add a deprecation notice in this PR when intermediate is used? (in addition to a changelog mention)

I'd strongly recommend doing so, yes.

Sorry for barging in late, but I feel we should keep support for old clients using an option, but use best practice ciphers by default. I tried to change this and there were users having problems. You can control the server, but it is not always possible to control the clients. Relegating installations to old versions that won't be patched and supported because they need to support mail clients they cannot upgrade seems less than ideal.
It would be another thing if this stopped us from adopting something, but as far as I can see it doesn't. We can use whatever we like with the default modern setting and still support old clients with the legacy setting, right?

I feel we should keep support for old clients using an option, but use best practice ciphers by default. I tried to change this and there were users having problems.

I'm aware. When I've completed the report it'll detail everything needed to make such a decision. The only likely clients that come to mind are macOS 10.11 and iOS 9 users (prior to sep 2016 release), neither receives security updates since mid 2018.

You can control the server, but it is not always possible to control the clients. Relegating installations to old versions that won't be patched and supported because they need to support mail clients they cannot upgrade seems less than ideal.

That's understandable. I'm not going to push for it other than provide the information and references towards making that decision. TLS 1.1 for most updates came with TLS 1.2 support. So you're supporting TLS 1.0 from 1999 with a few CBC cipher suites.
A new RFC advising against supporting such an old TLS version is expected to be published out of draft status sometime next year. OpenSSL 3.0 will also drop support by default, so when upgrading Debian base image at some point when it adopts the new version expect TLS 1.0 support to require additional effort to support.

It would be another thing if this stopped us from adopting something, but as far as I can see it doesn't. We can use whatever we like with the default modern setting and still support old clients with the legacy setting, right?

Yes, I don't see any issues supporting a legacy profile (might be better to call it "legacy" rather than "intermediate").
OpenSSL 3.0 just requires us to lower the security level via cipher list string, we might have to explicitly define the cipher suites for port 25 though (presently we use postfix defaults for that port and exclude some ciphers). Unless a future update of Postfix/Dovecot drops support.

macOS users that for whatever reason are unable to upgrade (systems older than year 2010) are only affected by mail apps that use Secure Transport (Apple TLS), such as Apple Mail. They could use web mail clients or alternatives that use a different TLS implementation like Thunderbird and be ok.
iOS users unable to upgrade to iOS 10 seems to be devices from 2012 and earlier? Support for iOS 9 ended in 2019.
It really depends how long you want to officially support that vs direct admins towards a wiki entry for legacy deployments to handle themselves (user patch). Since both macOS and iOS versions affected no longer receive updates, they've reached EOL status? I already know of one mail host that's only offering TLS 1.2 as a minimum since this year, I've not checked support on popular services yet.

@erik-wramner So are we leaving "intermediate" as-is and not logging a deprecation notice about TLS 1.0/1.1 being dropped in a future release?

@polarathene as long as it works I don't mind the warning. As you say it will be removed by upstream eventually and I certainly don't plan on supporting it at that point. However, as long as it is supported by upstream I would ideally not want to rock the boat for those admins who have to manage old clients.
When you mention it I think iOS devices were the main source of problems last time.

@polarathene I will close this PR in this repository as we just migrated to docker-mailserver/docker-mailserver. Please re-open your PR over there. This PR is very much awaited :D

@polarathene I will close this PR in this repository as we just migrated to docker-mailserver/docker-mailserver. Please re-open your PR over there. This PR is very much awaited :D

Yes sorry, been a busy month, I still intend to complete this.
Heads up that very soon default renewals of LetsEncrypt will be on a new root certificate that older devices without will fail to trust it. May possibly affect some older macOS / iOS clients from connecting to a mailserver.
Just in case you get reports all of a sudden of breakage, it's possibly due to that switch over. They can use Thunderbird or install a new root certificate directly into their affected devices.

Yes sorry, been a busy month, I still intend to complete this.
Heads up that very soon default renewals of LetsEncrypt will be on a new root certificate that older devices without will fail to trust it. May possibly affect some older macOS / iOS clients from connecting to a mailserver.
Just in case you get reports all of a sudden of breakage, it's possibly due to that switch over. They can use Thunderbird or install a new root certificate directly into their affected devices.

Just FYI: Please do not re-open this PR, but open a new one. That's because of (ugly) merge conflicts that would appear as some files have moved.

Just FYI: Please do not re-open this PR, but open a new one. That's because of (ugly) merge conflicts that would appear as some files have moved.


Implement RFC-001: Composable Pipelines

Description
This issue is about implementing RFC-001: Composable Pipelines. In the spirit of continuing the discussion out in the open, we want to lay out our plan for the implementation after having done a feasibility analysis on some designs we considered. Here's what we ended up with:
Element Type
The element type defines how operators can be combined. For starters, we want to support the element types Void and Arrow, but have the code written with extensibility in mind. Element types define whether they require a schema.
Logical and Physical Operators
A logical operator has a known input and output element type, and is able to instantiate a physical operator with an interface depending on the input and output type for a given input schema. The input schema may be null if the input type does not require a schema.
A physical operator is instantiated from a logical operator and an input schema, and maps an input element generator to an output element generator.
Pipelines and Plans
A pipeline is an ordered sequence of logical operators with matching input and output types along the sequence. The pipeline's input type is the input type of the first operator in the sequence, and the pipeline's output type is the output type of the last operator in the sequence. A pipeline is by definition a logical operator.
A plan is a pipeline whose input and output types are Void.
Logical Operator Plugin
A logical operator plugin creates a logical operator from its textual representation.
When parsing a pipeline op1 [args...] | op2 [args...] | ... VAST looks for the op1 and op2 logical operator plugins and passes them their respective argument string to instantiate a logical operator. For example, where '#type == "suricata.flow"' could instantiate the logical where operator with the expression #type == "suricata.flow".
The exact syntax of the individual logical operators is considered an implementation detail that we will discuss further in review.
Executor
An executor takes ownership of a plan and executes it in its (potentially asynchronous) execution model. The executor must execute a logical operator on only one thread at a time, but may concurrently access physical operator instantiations from one logical operator. The executor is responsible for transferring ownership of elements between logical operators by using thread-safe queues, e.g., an actor mailbox when using the actor runtime as an execution model.
In the future, we may consider relaxing the requirement not to execute physical operators instantiations from one logical operator in parallel if the logical operator allows for it.
Definition of Done

[ ] We have implemented and tested the proposed pipeline executor.
[ ] All existing pipeline operators are converted to the new plugin API.
[ ] We support executing pipelines with input and output type Arrow by specifying them on the command-line for exports, e.g., vast export <format> '<expr> | <pipeline>'.
[ ] Optional: We support executing plans via vast exec <plan> locally.

Anything else?
No response

I've made some implementation progress on this, and there were four key learnings during development so far that I would like to share with reasoning.

It turns out that the most practical backend readily available in VAST for the executor is CAF streaming, as it makes the executor scheduling very natural. With regular CAF messaging I quickly had overrunning buffers, as the actor scheduler schedules keeps an actor scheduled for as long as its mailbox is not empty. This does not play well with data pipelines because it effectively caused them to be executed sequentially, so it quickly switched to a streaming-based implementation.

In terms of control flow, the vast exec command needs to understand when an IPC boundary is required, e.g., in a read - csv | summarize sum(bytes) group by community_id | ingest pipeline, it needs to understand that read wants to run locally, summarize doesn't care where it runs, and ingest wants to run remotely. Every operator must categorize itself like this, and it is the executor's responsibility to ensure that the plan is split at the latest point possible.

There is a very natural mapping from logical operator to execution node, which corresponds to one green thread per actor. We may want to consider adding a mechanism for an operator to say that it wants to run detached, i.e., on a thread not owned by the actor scheduler. This is especially important for operators that interact with the filesystem.

Some logical operators are just pipelines, e.g., read - csv can internally be represented as load - | parse csv, which works automatically because I've implemented pipelines in a way that they are always also a logical operator. Turning a pipeline into a plan flattens the nested pipeline sequence. This implies that the read - csv operator maps onto two execution nodes.


Documentation needed about how to display Account selection page during every OAuth (or what option to use in OAuth request)

Code of Conduct

[X] I have read and agree to the GitHub Docs project's Code of Conduct

What article on docs.github.com is affected?
https://github.com/github/docs/blob/main/content/apps/oauth-apps/building-oauth-apps/authorizing-oauth-apps.md
What part(s) of the article would you like to see updated?
https://github.com/github/docs/blob/main/content/apps/oauth-apps/building-oauth-apps/authorizing-oauth-apps.md#1-request-a-users-github-identity
Docs do not mention how to force a new login during OAuth. Currently, if LoginId1 is already logged in to the browser, and OAuth is triggered for user1 always. This happens even on repeated OAuth flows. LoginId1 just repeats the authentication step for OAuth flow automatically (and receive a different access token).
Expected behavior should be that GitHub displays the "account selection" page during every OAuth authentication (or at least provide a way to specify this option as part of the OAuth request). This way, the OAuth will not be auto-assumed to be for Login1(which is the only user logged in to the browser), and the user has the option to login to a different id in this selection page.
Since GitHub now supports multiple accounts [https://docs.github.com/en/account-and-profile/setting-up-and-managing-your-personal-account-on-github/managing-your-personal-account/managing-multiple-accounts], there should be an option to trigger OAuth for a new user (not for a specific user by using the "login" field in step 1).
I imagine this is a missing documentation problem, if not, please redirect this issue to the right repository.
Thanks!
Additional information
No response

Hello @vineeththomasalex 👋 ! The best place to send this feedback regarding GitHub is to https://github.com/github/feedback/discussions ✨ I'll go ahead and close this issue 💛


typeError: ImagePickerManager.showImagePicker is not a function

Hi dear support. Your package depends on react-native-image-picker.
This package react-native-image-picker does not support more a method .showImagePicker
For now you have to use method .launchImageLibrary instead of .showImagePicker
For all who have the same issue, I found solution. Just decrease image picker version to<EMAIL_ADDRESS>

Hi, Maybe was it too obvious, but @chukhlov did not provide the "fix command" :-)
npm install<EMAIL_ADDRESS>--save
And I confirm that this fix fixed the crash. Thank you @chukhlov


Parser: Union validation should enforce at-most-one default value

Per the Apache compiler: https://github.com/apache/thrift/blob/042580f53441efe1bc5c80c89351fcb30740659e/compiler/cpp/src/thrift/parse/t_struct.h#L82
While this isn't documented anywhere outside the compiler code, the Thrift ecosystem supports unions with a single default value, e.g.
union Example {
  1: i32 A;
  2: i32 B = 2;
  3: string C;
}

We don't currently validate anything of the sort, but I guess we should.  Unions are little-used in my experience, but as long as we're advertising Apache Thrift compatibility, we may as well deliver the goods.
We'll need to augment StructType#validate for this, and add test cases to LoaderTest.

Huh, looks like we validate this already within our antlr listener.  On reflection, it's a weird place to do this specific "semantic" validation, since it doesn't interfere with antlr parsing.  Let's move it to StructType#validate, so that we can report more errors in a single pass.


device connection string should be unique

Preventing users from inadvertently sharing device connection string

@anhashia : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@anhashia : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@anhashia : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@kgremban - I applied some of your suggested revisions. Could you take a look? Should "IoT Hub" in lines 68 and 69 be "IoT Edge hub" or is it correct as is?
Thanks.

Thanks @anhashia

Thank you @anhashia and @ktoliver for making these changes. Looks good!
#sign-off

Thanks!


Code of Conduct

The contributor covenant is pretty good.

This is perfect. Looks like we just need a project e-mail address.


How can I import font-awesome

Sorry if my question is silly, but how can I import font-awesome using this template? I know I need to give a require in it somewhere, but where would it be the most appropriate place?
Thank you!

I would like to know this too. Found this: https://github.com/vuejs-templates/pwa/issues/91
but can't say I understand it completely.


"Poiana Stampei" is listed in the drop-down menu

Description:
"Poiana Stampei" is listed in the drop-down menu alongside its corresponding county according to the law
Precondition:
The website is up an running.

Step 1
Type "Poiana Stampei" in the search bar.

Expected results
The location is listed in the drop-down menu as "Comuna Poiana Stampei, SUCEAVA".


Test PSNR not improving on custom dataset

Nice work! I'm trying to train a custom face dataset (FaceScape) using the one-view reconstruction setting. Since the cameras in this dataset is already in OpenCV format, there's no conversion needed. I have tuned the radius parameter to reflect the change in the object size. However, the test PSNR never improves, although training PSNR does.

It also seems the rendered faces are not correctly localized. However I double checked the camera parameters, and made sure they are in the same convention as SRN car dataset and the poses loaded are c2w matrices. The only difference is that the intrinsics for each camera are different, but I've modified the dataloader correspondingly.
I'm a bit lost on what could be wrong. Do you make other assumptions on the camera parameters? Are there any other parameters that I should tune?

Thank you for your interest in our work!
How is the training PSNR actually? The screenshot only shows very early results so it's hard to tell if anything goes wrong. Evaluating the model at this early stage is not really helpful.
If the image resolution has been changed, then you probably need to tweak some parameters like guidance scale. But if it's just the camera intrinsics then it should be ok. Also starting from the SRN car config for face reconstruction is a good choice.

Thank you for your fast response! It turned out it was indeed just too early in the training stage. After training overnight, now the evaluation results look reasonable.


<EMAIL_ADDRESS>

Another Electron was just released. Let's bump from 1.2.8 to 1.3.0.
Changelog:

Upgrade to Chrome 52
Update to Node.js 6.3.0

Nice, LGTM 👍


vault: Add Version 1.9.0

What type of Apps related PR is this? (check all applicable)

[x] ✨ Add Manifest
[ ] ♻ Refactor
[ ] 🐛 Bug Fix
[ ] 👷 Optimization
[ ] 🚩 Other

Description
vault.exe ia a tool for secrets management, encryption as a service, and privileged access management
Related Tickets & Documents
none
Did you verify the followings
Add/Modify a description in the manifest?

[x] 👍 yes
[ ] 🙋 yes, but I need help
[ ] 🙅 no, because they aren't needed

Check URLs?

[x] 👍 yes
[ ] 🙋 yes, but I need help
[ ] 🙅 no, because they aren't needed

Update to the latest version?

[x] 👍 yes
[ ] 🙋 yes, but I need help
[ ] 🙅 no, because they aren't needed

Update Hashes?

[x] 👍 yes
[ ] 🙋 yes, but I need help
[ ] 🙅 no, because they aren't needed

Format the json manifest?

[x] 👍 yes
[ ] 🙋 yes, but I need help
[ ] 🙅 no, because they aren't needed

Did you identify extra steps
Notes in the manifest?

[ ] 👍 yes
[ ] 🙋 yes, but I need help
[x] 🙅 no, because they aren't needed

Application Dependencies?

[ ] 👍 yes
[ ] 🙋 yes, but I need help
[x] 🙅 no, because they aren't needed

Application Suggestions?

[ ] 👍 yes
[ ] 🙋 yes, but I need help
[x] 🙅 no, because they aren't needed

Pre-installation steps to perform?

[ ] 👍 yes
[ ] 🙋 yes, but I need help
[x] 🙅 no, because they aren't needed

Post-installation steps to perform?

[ ] 👍 yes
[ ] 🙋 yes, but I need help
[x] 🙅 no, because they aren't needed

File persistence to perform?

[ ] 👍 yes
[ ] 🙋 yes, but I need help
[x] 🙅 no, because they aren't needed

[optional] What gif/emoji best describes this PR or how it makes you feel?

All changes look good.
Wait for review from human collaborators.
vault

[x] Description
[x] License
[x] Hashes
[x] Checkver
[x] Autoupdate
[x] Autoupdate Hash Extraction


Revamp Internal Horizon Documentation

PR Checklist
PR Structure

[ ] This PR has reasonably narrow scope (if not, break it down into smaller PRs).
[ ] This PR avoids mixing refactoring changes with feature changes (split into two PRs
otherwise).
[ ] This PR's title starts with name of package that is most changed in the PR, ex.
services/friendbot, or all or doc if the changes are broad or impact many
packages.

Thoroughness

[ ] This PR adds tests for the most critical parts of the new functionality or fixes.
[ ] I've updated any docs (developer docs, .md
files, etc... affected by this change). Take a look in the docs folder for a given service,
like this one.

Release planning

[ ] I've updated the relevant CHANGELOG (here for Horizon) if
needed with deprecations, added features, breaking changes, and DB schema changes.
[ ] I've decided if this PR requires a new major/minor version according to
semver, or if it's mainly a patch change. The PR is targeted at the next
release branch if it's not a patch change.

What
This PR rearranges and adds internal documentation:

Creating an updated development guide that is easier to understand.
Adding a new readme on Horizon SDK and API.
Adding references to all these files on the main README page for easier access.

Why
There was a lot of redundant readme files and documentation that made it harder for someone new in the team to understand the codebase. These changes remove the duplicate documentation, adds some new READMEs and brings all important resources under a single roof.
Known limitations
None.

looking really good, great work on consolidating from various points. I think would be good to also reach out for any feedback on this from others on team with extended experience in this repo such as @tamirms , @paulbellamy, @2opremio

@aditya1702 , can you update the description on pr, it effectively closes #4936 and #4939 correct?


Time Scale: How could one provide a custom irregular spacing of ticks?

As weekends are less interesting in one of our use cases, we are looking for a way to a have a time scale (and axis) in which the weekend hours are spaced more densly then the workday hours. How could the be achieved? Since there is alreay support for "irregular spacing of ticks because time intervals have varying length" this should somehow be possible, or?

This candlestick chart follows a similar requirement, by removing the week-ends. However note that it's an ordinal chart (day by day), not continuous time.
If you want to make time continuous but non-linear you can certainly write your own function from the time domain to floats, and then put back the correct ticks on the axis. I'm doing that (although with a different time warper) in this renko chart notebook.
I'm closing the issue since this is not a bug or a thing we'd want to develop inside of d3-scale, but happy to continue the discussion.


Sign In existing user without 2FA

in case 2FA factor is_active = false user should log in with DS to Cabinet. At the moment Error is shown.
Thw way to fix it:

add new step to app_authorize
Send additional client_id in token request

not actual anymore


html: Add Unicode 11 xterm.js addon

Add Unicode 11 addon to xterm.js to enable support for newer Unicode standards. This addon significantly improves rendering of emojis.

@tsl0922 Will need some advice on how I activated the feature. Not sure I activated it in the preferred location. Feel free to recommend any changes or improvements.
For reference this is the line I am uncertain about:
terminal.unicode.activeVersion = '11';

For reference this is the line I am uncertain about:
terminal.unicode.activeVersion = '11';

Maybe it can be a Client Option, if we want to extend it to support unicode-graphemes addon later.

I think once the unicode-graphemes addon becomes stable it would be worth switching over.
The reason I didn't make Unicode 11 an option was I can't think of why you wouldn't want this always enabled. What benefit do we gain from this being an option? Without this being set, most emoji are just rendered incorrectly.

What benefit do we gain from this being an option?

You can enable / disable this addon, or switch to unicode-graphemes later.

You can enable / disable this addon, or switch to unicode-graphemes later.

Does this mean we want something that has a value of unicode-11 or unicode-graphemes so you select which support you want?

You actually have me thinking, should we add unicode-graphemes and mark is experimental, while setting the default to unicode11. Can do all the work now and later on just switch the default.

Does this mean we want something that has a value of unicode-11 or unicode-graphemes so you select which support you want?

The value should be valid for term.unicode.activeVersion, for example: 11 / 15-graphemes  (6 if both disabled)
Ref: https://github.com/xtermjs/xterm.js/blob/1da0db435f93879a60b9badbb663852ff3223063/demo/client.ts#L659-L679

should we add unicode-graphemes and mark it experimental

No, unicode-graphemes is not published to npm yet.

So let me clarify this before I get started:

We add a new client side variable: unicodeVersion.
If unset default is 11.
Can also be set to 6 or 15-graphemes.
If set to 11 we enable the unicode11 addon.
If set to 15-graphemes we fallback to default 11 and enable unicode11 addon.
If set to 6 we do nothing (not enable the unicode11 addon).

This is slightly different to your earlier post but I think it makes more sense to have Unicode 11 as the default. Unicode 6 is just way too old and renders very differently.
Happy to adjust this logic though based on your feedback.

We add a new client side variable: unicodeVersion.
If unset or unknown value, use the default 11.
If set to 6 we do nothing (not enable any unicode addon).

Thanks for merging this! 🤞 when this is released there are no issues.


bug in luv_new_tcp

static int luv_new_tcp(lua_State* L) {
  lua_settop(L, 1); // FIXED: or lua_isnoneornil(L, 1) will test the userdata!
  uv_tcp_t* handle = (uv_tcp_t*)luv_newuserdata(L, sizeof(*handle));
  int ret;
  if (lua_isnoneornil(L, 1)) {
    ret = uv_tcp_init(luv_loop(L), handle);
  }

Thanks.  I wonder what kind of undefined behavior we've been causing.  I guess if the userdata always converts to 0 as an integer it was at least harmless.

How did you find this bug?  Was it causing bad behavior or did you just see it reviewing the code?  I'd like to know what the impact was.

I found this bug occasionally when debugging a mysterious crash.
But the crash is not related to luv_new_tcp. It is bug in LuaJIT. After I upgrade to new LuaJIT version, the crash disappear.
I think the bug in luv_new_tcp has little impact, it only causes no flags is passed to uv_tcp_init_ex(uv_loop_t* loop, uv_tcp_t* handle, unsigned int flags).

Thanks.  I just made a new luvi release yesterday that uses the latest luajit 2.1 as well as this fix.  Thanks for reporting it even it probably wasn't causing any trouble.


[Packaging] apt install with explicit version fails due to dependency version mismatch

Describe the bug, including details regarding any error messages, version, and platform.
Installing, on Ubuntu, with explicit version specified, e.g. apt install -y -V libarrow-dataset-dev=14.0.1-1 fails, with errors like:
root@cd210974a97c:/# apt install -y -V libarrow-dataset-dev=14.0.1-1
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 libarrow-dataset-dev : Depends: libarrow-acero-dev (= 14.0.1-1) but 15.0.0-1 is to be installed
                        Depends: libparquet-dev (= 14.0.1-1) but 15.0.0-1 is to be installed
E: Unable to correct problems, you have held broken packages.

If I, manually, first install libparquet-dev=14.0.1-1 and libarrow-acero-dev=14.0.1-1 (in that order), the installation of libarrow-dataset-dev=14.0.1-1 succeeds.
So, it seems the apt packages does not contain precise enough version information in their dependency information.
This is mostly following the "Debian GNU/Linux and Ubuntu" instructions for version 14 but the instructions for the current version 15 is similar:
$ docker run --interactive --tty --rm ubuntu:22.04
root@cd210974a97c:/# uname -a
Linux cd210974a97c 6.6.12-linuxkit #1 SMP PREEMPT_DYNAMIC Fri Jan 19 12:50:23 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
root@cd210974a97c:/# apt update
...
root@cd210974a97c:/# apt install -y -V ca-certificates lsb-release wget
...
root@cd210974a97c:/# lsb_release --id --short
Ubuntu
root@cd210974a97c:/# lsb_release --codename --short
jammy
root@cd210974a97c:/# wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
...
root@cd210974a97c:/# apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
...
root@cd210974a97c:/# apt update
...
root@cd210974a97c:/# apt install -y -V libarrow-dev=14.0.1-1
...
root@cd210974a97c:/# apt install -y -V libarrow-dataset-dev=14.0.1-1
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 libarrow-dataset-dev : Depends: libarrow-acero-dev (= 14.0.1-1) but 15.0.0-1 is to be installed
                        Depends: libparquet-dev (= 14.0.1-1) but 15.0.0-1 is to be installed
E: Unable to correct problems, you have held broken packages.
root@cd210974a97c:/# apt install -y -V libarrow-acero-dev=14.0.1-1
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 libarrow-acero-dev : Depends: libparquet-dev (= 14.0.1-1) but 15.0.0-1 is to be installed
E: Unable to correct problems, you have held broken packages.
root@cd210974a97c:/# apt install -y -V libparquet-dev=14.0.1-1
...
root@cd210974a97c:/# apt install -y -V libarrow-acero-dev=14.0.1-1
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
   libarrow-acero1400 (14.0.1-1)
The following NEW packages will be installed:
   libarrow-acero-dev (14.0.1-1)
   libarrow-acero1400 (14.0.1-1)
0 upgraded, 2 newly installed, 0 to remove and 10 not upgraded.
Need to get 1089 kB of archives.
After this operation, 6101 kB of additional disk space will be used.
Get:1 https://apache.jfrog.io/artifactory/arrow/ubuntu jammy/main amd64 libarrow-acero1400 amd64 14.0.1-1 [423 kB]
Get:2 https://apache.jfrog.io/artifactory/arrow/ubuntu jammy/main amd64 libarrow-acero-dev amd64 14.0.1-1 [665 kB]
Fetched 1089 kB in 4s (253 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libarrow-acero1400:amd64.
(Reading database ... 15259 files and directories currently installed.)
Preparing to unpack .../libarrow-acero1400_14.0.1-1_amd64.deb ...
Unpacking libarrow-acero1400:amd64 (14.0.1-1) ...
Selecting previously unselected package libarrow-acero-dev:amd64.
Preparing to unpack .../libarrow-acero-dev_14.0.1-1_amd64.deb ...
Unpacking libarrow-acero-dev:amd64 (14.0.1-1) ...
Setting up libarrow-acero1400:amd64 (14.0.1-1) ...
Setting up libarrow-acero-dev:amd64 (14.0.1-1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.6) ...
root@cd210974a97c:/# apt install -y -V libarrow-dataset-dev=14.0.1-1
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
   libarrow-dataset1400 (14.0.1-1)
The following NEW packages will be installed:
   libarrow-dataset-dev (14.0.1-1)
   libarrow-dataset1400 (14.0.1-1)
0 upgraded, 2 newly installed, 0 to remove and 11 not upgraded.
Need to get 1125 kB of archives.
After this operation, 7496 kB of additional disk space will be used.
Get:1 https://apache.jfrog.io/artifactory/arrow/ubuntu jammy/main amd64 libarrow-dataset1400 amd64 14.0.1-1 [441 kB]
Get:2 https://apache.jfrog.io/artifactory/arrow/ubuntu jammy/main amd64 libarrow-dataset-dev amd64 14.0.1-1 [683 kB]
Fetched 1125 kB in 3s (330 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libarrow-dataset1400:amd64.
(Reading database ... 15275 files and directories currently installed.)
Preparing to unpack .../libarrow-dataset1400_14.0.1-1_amd64.deb ...
Unpacking libarrow-dataset1400:amd64 (14.0.1-1) ...
Selecting previously unselected package libarrow-dataset-dev:amd64.
Preparing to unpack .../libarrow-dataset-dev_14.0.1-1_amd64.deb ...
Unpacking libarrow-dataset-dev:amd64 (14.0.1-1) ...
Setting up libarrow-dataset1400:amd64 (14.0.1-1) ...
Setting up libarrow-dataset-dev:amd64 (14.0.1-1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.6) ...
root@cd210974a97c:/#

See the attached full transcript.
arrow_14_install_problems.txt
Component(s)
Packaging

You must specify versions to all related packages explicitly when you want to install specific version:
apt install -y -V libarrow-dataset-dev=14.0.1-1libarrow-acero-dev=14.0.1-1 libparquet-dev=14.0.1-1

The problem is not that I do not specify versions, the problems are:

Even if I follow the installation instructions, and add explicit versions to all packages, the install will fail because, in the example shell commands, the required packages are not installed before the dependent packages in the Ubuntu installation instructions.
Presumably a package is supposed to tell apt which other packages it depends on. So, you should not need to know everything that needs to be installed, apt should be able to figure this out from the package meta information. The problem seems to be that the meta information, e.g. for libarrow-dataset-dev, does not seem to provide the necessary version constraints for the dependencies.

Really? Do you know an official Debian/Ubuntu package set that we don't need to specify to a specific version to all related packages?


Ulriks packages

Hitting this error when building;
File "expander/sexp_grammar_lifter.ml", line 31, characters 44-69:
31 | let lift_string ~loc s = pexp_constant ~loc (Pconst_string (s, None))
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^
Error: The constructor Pconst_string expects 3 argument(s),
       but is applied here to 2 argument(s)

It now builds after https://github.com/anmonteiro/nix-overlays/commit/bd8b986de7f1065c21c570ff5d1b23ade17a5fbd

It now builds after https://github.com/anmonteiro/nix-overlays/commit/bd8b986de7f1065c21c570ff5d1b23ade17a5fbd


Remove veDelegationProxy

Fixes #689
Changes proposed in this PR:

Remove veDelegationProxy

@alexcos20 approved. He will merge this.


AnnData split-apply-combine

anndata should get a split apply combine framework. This is something we’ve wanted for a while, but this issue is just establishing a concrete place to track it.
Questions
API
There are a number of split-apply-combine frameworks/ API's out there. Which do we want to emulate, and what are the most important features?

Access to intermediate elements (e.g. for k, sub_adata in adata.groupby(obs="celltype"))
Efficient reductions and combination
Nested groupby, group over multiple dimensions
Lagged/ windowed grouping (e.g. groupby neighbors of a cell, moving average over time)

What is the correct default return type of a reduction? Can we be flexible about this?

AnnData would be nice for consistency, but do we want to keep all that metadata?

Also, how would we handle reductions over non-X data? E.g. mean point in PCA space for a cluster

DataFrames would let us keep labels, but should they be used as "labelled arrays"?
xarray.DataArrays make sense for just labelled arrays

Reductions and sparse data
How can we make reductions efficient for sparse matrices? One of the most annoying problems when writing code for scanpy and anndata is getting operations to work on both dense and sparse arrays, since sparse arrays have a different API. If reductions and combinatons are included here, then we'll have to be sure sparse arrays work, which could be a lot of extra code. My personal preference for addressing this is to get better support for sparse data with an array API in upstream libraries like pydata/sparse.
Current and previous attempts/ discussion

https://github.com/ivirshup/mantis#group-by
https://github.com/rafaqz/DimensionalData.jl/issues/3#issuecomment-590061903
https://github.com/theislab/anndata/pull/350#issuecomment-607033414
https://github.com/theislab/scanpy/pull/1390 (@gokceneraslan)
#527 (@vals)

Ping @jbloom22, who has also done some work on this.
Other implementations

xarray

Modeled on pandas, but has multidimensional groupby

pandas
hail

MatrixTable groupby
Aggregations

SplitApplyCombine.jl

Another (IMO interesting) implementation: siuba's group_by: https://siuba.readthedocs.io/en/latest/intro.html#Basic-use.
Behavior for other data types can be added via singledispatch.

https://github.com/theislab/anndata/pull/564


Error in progeny evidence

The first line of http://storage.googleapis.com/ot-releases/18.08/progeny-23-07-2018.json.gz is "Parsed all HGNC genes - source: http://ftp.ebi.ac.uk/pub/databases/genenames/new/json/hgnc_complete_set.json" this is not a valid evidence string and should be removed and prevented from being added to future evidence generation runs.

There are 196 lines in the progeny_normalVStumor_opentargets.txt file which are converted into a JSON file with 407 lines. 392 of these lines are in the validated-data-progeny ES index. Of the 15 lines that do not make it into the ES index, 14 are duplicates and 1 is the first line mentioned by @afaulconbridge above.
Of the 392 lines, only 308 are deemed valid meaning that 84 evidence strings do not pass validation. All 84 invalid strings have a disease error because their disease ID is ID EFO_0000000:
GET 18.08_validated-data-progeny/_search
{
  "query": {
    "bool": {
      "must": [
        {"match": {
          "disease_id": "http://www.ebi.ac.uk/efo/EFO_0000000"
        }},
        {"match": {
          "explanation.disease_error": "true"
        }}
      ]
    }
  }
}

Checking the JSON file reveals that EFO_0000000 appears in 87 evidence strings (29 times each for the 3 cancer types below) which means that of the 84 dropped evidence strings one each is a duplicate indicating that the 14 dropped evidence strings that do not make it into the ES index correspond to one replicate per cancer type. Looking at the cancer type mappings, it appears that for this pathway 'PI3K'     : ['PIK3CA', 'PI3K(Class1)'], both genes are mapped to PIK3CA resulting in replicate entries for each cancer. I will find out which protein PI3K(Class1) should refer to.

Affected evidence strings
JSON data

29
"disease": {"id": "http://www.ebi.ac.uk/efo/EFO_0000000", "name": "colorectoral adenoma"}

29
"disease": {"id": "http://www.ebi.ac.uk/efo/EFO_0000000", "name": "kidney renal papillary cell carcinoma"}

29
"disease": {"id": "http://www.ebi.ac.uk/efo/EFO_0000000", "name": "kidney chromophobe"}

These three cancer types have been added after the publication came out. I will check whether it is ok to include these three cancer types now.

@afaulconbridge I cannot see where the first line is written into the PROGENy JSON file. The code for SLAPEnrich is nearly identical and the SLAPEnrich file does not have that line.

I think it came from the HGNCParser indirectly, but I agree as long as the 18.08 file isn't used directly and this line isn't in a newly generated file, this ticket can be closed.

Will be addressed post EFO 3 release.  Moved to 19.04.

The line from the HGNC parser is not written into the JSON file anymore and the three missing cancer type mappings have been updated:
'KICH': {'uri': 'http://www.ebi.ac.uk/efo/EFO_0000335', 'label': 'kidney chromophobe'}, # EFO label: chromophobe renal cell carcinoma
'KIRP': {'uri': 'http://www.ebi.ac.uk/efo/EFO_0000640', 'label': 'kidney renal papillary cell carcinoma'}, # EFO label: papillary renal cell carcinoma
'COREAD': {'uri': 'http://www.ebi.ac.uk/efo/EFO_0005406', 'label': 'colorectoral adenoma'} # EFO label: colorectal adenoma

The 'PI3K' pathway question will be addressed tomorrow.

Update: The extra line at the beginning of the PROGENy evidence file Parsed all HGNC genes - source: http://ftp.ebi.ac.uk/pub/databases/genenames/new/json/hgnc_complete_set.json was there because the last PROGENy evidence file has been generated by saving print output to the screen as a JSON file because the proper writing method does something funky - for all genes of a pathway-disease combo the evidence string for the last gene in the pathway gene set is repeated for each member of the pathway even though the evidence strings are generated correctly.

Closing this ticket because the 'PI3K' pathway question is covered in this ticket #713


How to remove the default Typescript tooltip hover

Thanks for making such a nice library, I know my question is not specifically related to this repo, but I'm guessing you guys might know the answer.
I would like to remove the duplicate .ts error on hover, Do you have any idea around it?

As you know the fist line of the error is almost redundant.

+1

+1
that's the first thing i thought about.

I'm pretty sure this is not supported by the VS Code API yet 🤮.

I would post this issue in the VSCode repository :)

+1 !!
(thanks for the extension 🔥)

It would already be a big improvement if we could change the order. Does anyone know if that's possible? I wouldn't mind if the gibberish errors are at the bottom and the readable one is at the top.

i have this by default

It would already be a big improvement if we could change the order. Does anyone know if that's possible? I wouldn't mind if the gibberish errors are at the bottom and the readable one is at the top.

i also at the top by default, it might be an issue caused by other plugins in your vscode?

Thank you for opening this issue!
It's currently not supported by VSCode APIs, but @johnsoncodehk got a break through there and made a POC of service language plugin that could do it.
Let's continue to talk about the order of the messages and the option to hide the original errors there:
https://github.com/yoavbls/pretty-ts-errors/issues/3


Support Namespaced Schemas

First off, thanks for such a wonderful tool! It really helps speed up the prototyping process.
I am using a schema other than public in Postgres. Running diesel print-schema produces a schema.rs similar to the following:
pub mod my_schema {
    table! {
        my_schema.my_table (my_pk) {
            my_pk -> Int4,
            some_field -> Text,
        }
    }
// ...
}

Trying to dump the models from this produces the following panic:
thread 'main' panicked at ' is not recognized. Please free feel to expand the HashMap. This could provide good hints: https://kotiri.com/2018/01/31/postgresql-diesel-rust-types.html', /Users/aaron/.cargo/registry/src/github.com-1ecc6299db9ec823/diesel_cli_ext-0.1.13/src/parse.rs:104:24
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

Moving the table declaration in schema.rs outside of the module produces what I'd expect. As a workaround, I can move all of the tables outside of the my_schema module, remove the my_schema prefix, outdent all table declarations to the left margin, generate the models, then do any other updates by hand.

What is the expected output?

It would probably be best to preserve the schema for the created modules:
pub mod my_schema {
    table! {
        my_schema.my_table (my_pk) {
            my_pk -> Int4,
            some_field -> Text,
        }
    }
}

pub mod my_other_schema {
    table! {
        my_other_schema.my_table (my_pk) {
            my_pk -> Int4,
            some_field -> Text,
        }
    }
}

would produce
pub mod my_schema {
    #[derive(Queryable,Debug)]
    pub struct MyTable {
        pub my_pk: i32,
        pub some_field: String,
    }
}

pub mod my_other_schema {
    #[derive(Queryable,Debug)]
    pub struct MyTable {
        pub my_pk: i32,
        pub some_field: String,
    }
}

updated.
indentation was not done though

indentation added. 💃

Thank you so much for working on this!


RT 2.1 - base_adjacencies_test.go

Hi,
Check.Equal function checks for values in ordered fashion.
Added code to verify string of slices in un ordered fashion.
Issue Ref:  https://github.com/openconfig/featureprofiles/issues/1103
Thanks,
Prabha

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 21 (0.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.6%) to 56.643%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

internal/check/check.go
0
21
0.0%

Totals

Change from base Build<PHONE_NUMBER>:
-0.6%

Covered Lines:
1181

Relevant Lines:
2085

💛  - Coveralls

@xw-g , please could review the changes for un ordered list verification. thanks

/gcbrun


Update README with python example

@kusti8 Oops, I didn't realise that README.md had a copy of the python example. (I assumed it would slurp in the .py file).
This PR updates the code inside the README.md too.
Sorry about not catching it before.

I've fixed it a bit more elegantly in 9a30aa4.


Do the check at compile time in case of integral_constant

When someone write this:
void f (positive i);
//somewhere in the code
f (integral_constant<int, 0>{});

In this case there is no need to do the check at runtime.
In theory, this can beat the equivalent code performance wise.
However, that's not readable enough.
A better solution would be to do hana style user defined literals, like this:
void f (0_i) //0_i is an integral_constant<int, 0>
That's still not ideal because the users have to think about it, but much better than the previous.
Unfortunately that requires an extension and it's not yet in the standard (works well with clang/gcc), so i might not rush to do the literal stuff.

basic scenario is done, literal is not


Update paper_movement.rst

Change '$1B $65  n   m   t' to '$1D $65  n   m   t' in Ejector Commands.

Thanks for the fix!


I meet Segmentation fault when I run your code on linux

I use gdb to debug
in line 184 sum += fabs(entityVec[last2 + ii] - entityVec[last1 + ii] - relationVec[lastr + ii]);
once last2 is too big(for examble:<PHONE_NUMBER>),
I print entityVec[1775403720]
there is a tip: Cannot access memory at address 0x80019e18db30
I want to know how to fix this bug

I have fixed bugs. The problem is not in the main body of the program but in the processing of data input. Please accept my apologies for my oversight.


Reverting to 1.0 meta layers and removed verified boot

Reverting to 1.0 meta layers as we do not have support to maintain the mbl meta layers.

recipes-kernel/linux/linux-raspberrypi_%.bbappend

This will also need to be rebased with the base branch before it can continue.

Sorry, merging this PR in order to release 2.0.0-RC1 over the weekend.


Update Advanced Hunting Query

I have updated the Advanced Hunting query to be more specific and also detect events that aren't being collected by the current query.
Missed Events
As per https://github.com/MicrosoftDocs/WDAC-Toolkit/blob/fd87c779d1cc06d7207e5c0502a810f2c08d89cc/WDAC-Policy-Wizard/app/src/AdvancedHunting.cs#L16-17 there are events starting with "AppControlCIScript" which wouldn't be picked up as the current query specifies "startswith 'AppControlCodeIntegrity' "
Extra Events
As per the the below link, there are extra events being picked up which the WDAC wizard doesn't need

AppControlCodeIntegrityImageRevoked
AppControlCodeIntegrityOriginAllowed
AppControlCodeIntegrityPolicyLoaded

https://learn.microsoft.com/en-us/windows/security/application-security/application-control/app-control-for-business/operations/querying-application-control-events-centrally-using-advanced-hunting#action-types
Misc Notes
If this PR is accepted, the docs at MS should also be updated - https://learn.microsoft.com/en-us/windows/security/application-security/application-control/app-control-for-business/design/appcontrol-wizard-parsing-event-logs#mde-advanced-hunting-app-control-event-parsing

@microsoft-github-policy-service agree

Reordering the columns wasn't supposed to be in this commit

This looks great. Thanks for your help on this. I will update the MS Learn doc to reflect these changes


Assert all snapshots have been checked

This is a feature request for testing, that all snapshots have been checked for a given test case. We have a large code base with dynamic snapshot tests and frequent refactors. This helps us better maintain our snapshots and find more mistakes. It is only for mac/iOS currently.
I hope you find it usefull (it would be nice not to maintain a fork to use your project 🙂) and we can start a discussion on it.
Example usage:
class MyViewControllerTests: XCTestCase {
  override func tearDown() {
    assertAllSnapshotsChecked(for: MyViewControllerTests.self)
    super.tearDown()
  }
  // ...
}

I gave it a shot for the "platform as folder", you can check it out at: https://github.com/szotyi/swift-snapshot-testing/commit/6e1bea20553d9500b78e633613c0a4fc330be646
(My questions from the previous comment are still up)

Good catch. Platforms make this difficult. To annotate snapshots in meaningful and storable manner, I can only see creating folders for each platform or using naming schemes (eg. postfix). This would create duplicated snapshots and would have to create some kind of migration functionality from old versions of the library. For the first problem maybe some kind of way to create "combined platforms" or if there is only one than no annotation (the current way).
Do you have other ideas?

A global configuration feature around annotation would be helpful in general, though I'm not sure what it would look like quite yet.

What other edge cases do you see (beside the Linux compatibility)?

None that come to mind right now, but I'll think on it some more.

What’s your strategy of new features? Do you only put fully implemented ones in or can there be ones with limited functionality (with clear documentation) and than approve them over time (when demand arrives or fresh ideas pop up)?

Nothing formalized yet, but I think the thoughts are:

New features should be non-breaking (till a new major release, e.g., 2.0)
Additive new features should be rolled out in minor releases (e.g., 1.1) and should be stable/documented by the time of the release
New features are welcome into master without having been fully stabilized/documented for those that want to test the feature out

We'd also like to keep the core library small, I think, but extensible. Ideally your helper could be written as another library and somehow hook into our public API. Then over time, popular external libraries could be folded into this main one.

Was thinking about your “requirements” and couldn’t come up with a nice way, that would fit the robustness and easiness of your current state.
I will keep eye on your future enhancements and features (and of course will watch you videos) and if any new idea comes to mind will let you know.
Feel free to close this pr (I will maintain my fork).
Merry Christmas!


Update languages: Portuguese (Brazil) key

I saw supported languages response as below, the key changed to "pt"
        "pt": {
            "name": "Portuguese (Brazil)",
            "nativeName": "Português (Brasil)",
            "dir": "ltr"
        }

@huybuidac : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@swmachan Could you review the proposed change to your article? Thanks.

@krishna-doss-mohan - please review

@krishna-doss-mohan - please review


Studio doesn't show index

SQL Operations Studio Version: sqlops 0.23.6 (8a3011756318a2ded1e6e84332230a2b2bf9c544, 2017-11-06T21:51:43.232Z)
OS Version: Windows_NT x64 10.0.16299

A table in the database has primary key (non-clustered) and clustered index. However, the editor doesn't show primary key. Moreover, it would be welcome to have distinct icons for keys and indexes like in SSMS.

@JohnyL Can you please elaborate 'the editor doesn't show primary key' part? I am trying to understand this issue better with more information :)

@erickangMSFT A picture is worth a thousand words :)

I did some digging and this looks like a bug we ported over from SSDT (which we based our Object Explorer implementation on since it's cleaner than SSMS). In SSDT, foreign keys show as entries under Keys and Indexes, but PKs only show under Keys. In SSMS they show in both.
Given FKs show in both it seems like this must be a bug where PKs don't show rather than an intentional difference. @llali we can update the XML file to get this working?

this bug is fixed and will be in the next release


Made ensure_connected fail properly

Requires https://github.com/bluesky/bluesky/pull/1758

I'll add tests for ensure_connected passing/failing


chore: update resource cache to be atomic + cleanup process

Summary

I've moved these suggestions onto a separate branch given they are quite substantial. They are recommendations for https://github.com/FuelLabs/fuels-ts/pull/2872

I noticed we only remove the expired cached UTXOs when getActiveData is called, but that is only called when getResourcesToSpend is. Correct me if I am wrong but a consumer could submit a transaction without calling getResourcesToSpend and thus expired UTXOs wouldn't have been cleared and the transaction submitted. So I wrote a cleanup process that runs in the background to clear the cache.

Because that process could be trying to delete a cached UTXO that could be potentially be trying to be accessed or the cache itself could be accessed during a delete, I made the operations atomic through a semaphore.

Checklist

[x] I added — tests to prove my changes
[ ] I updated — all the necessary docs
[x] I reviewed — the entire PR myself, using the GitHub UI
[ ] I described — all breaking changes and the Migration Guide

Closing as these changes are no longer relevant (see https://github.com/FuelLabs/fuels-ts/pull/2872#issuecomment-2271748130)