Increase MySQL 8 speed

Preliminary checklist

[X] I am using the latest stable version of DDEV (see upgrade guide)
[X] I have searched existing issues
[X] I have checked the troubleshooting guide
[X] I have run ddev debug test to include output below

Output of ddev debug test
Expand `ddev debug test` diagnostic information

======== Please make sure that you have already looked at troubleshooting guide ========
Troubleshooting guide: https://ddev.readthedocs.io/en/stable/users/usage/troubleshooting/
Simple things to check:
* ddev poweroff
* Restart Docker Provider
* Reboot computer
* Temporarily disable VPN and firewall
Press any key to continue:

======== Output file will be in /tmp/ddev-debug-test.txt ========

======== Existing project config ========
ddev installation alternate locations:
/usr/bin/ddev
/bin/ddev

These config files were loaded for project example.com: [/home/hannes/Sites/example/.ddev/config.yaml /home/hannes/Sites/example/.ddev/config.wunderio.yaml]
name: example.com
type: drupal10
docroot: web
php_version: 8.1
webserver_type: nginx-fpm
webimage: ddev/ddev-webserver:v1.23.1
database: {mysql 8.0}
hooks: map[post-import-db:[map[exec:/var/www/html/.ddev/wunderio/core/_run-scripts.sh hooks-db-post-import.sh]] post-start:[map[exec-host:.ddev/wunderio/core/_run-scripts.sh hooks-host-post-start-2.sh] map[exec:[ ! -d "/var/www/html/vendor" ] && /var/www/html/.ddev/wunderio/core/_run-scripts.sh hooks-web-post-start-once.sh || true] map[exec-host:.ddev/wunderio/core/_run-scripts.sh hooks-host-post-start.sh] map[exec:/var/www/html/.ddev/wunderio/core/_run-scripts.sh hooks-web-post-start.sh] map[exec:UPDATE_SCRIPT="/var/www/html/vendor/wunderio/ddev-drupal/scripts/update_check.sh"; [ -f "$UPDATE_SCRIPT" ] && $UPDATE_SCRIPT || true]]]
project_tld: ddev.site
use_dns_when_possible: true
composer_version: 2.7.2
disable_settings_management: true
nodejs_version: 18
default_container_timeout: 180

======== existing project customizations ========
.ddev/docker-compose.mounts.yaml
.ddev/php/php-example.ini.example
.ddev/php/php.ini
.ddev/nginx_full/nginx-site.conf

======== installed DDEV add-ons ========
No registered add-ons were found. Add-ons installed before DDEV v1.22.0 will not be listed.
Update them with `ddev get` so they'll be shown.

======== mutagen situation ========
looking for #ddev-generated in mutagen.yml in project /home/hannes/Sites
.ddev/mutagen/mutagen.yml not found

======== Creating dummy project named tryddevproject-28389 in ../tryddevproject-28389 ========
Creating a new DDEV project config in the current directory (/home/hannes/Sites/tryddevproject-28389)
Once completed, your configuration will be written to /home/hannes/Sites/tryddevproject-28389/.ddev/config.yaml

Configuring a 'php' project with docroot 'web' at /home/hannes/Sites/tryddevproject-28389/web
Configuration complete. You may now run 'ddev start'.

======== OS Information ========
Linux hannes-ThinkPad-X1-Carbon-Gen-9 6.5.0-35-generic #35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue May  7 09:00:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

======== User information ========
uid=1000(hannes) gid=1000(hannes) groups=1000(hannes),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),120(lpadmin),132(lxd),133(sambashare),996(microk8s),998(docker)

======== DDEV version ========
 ITEM             VALUE
 DDEV version     v1.23.1
 architecture     amd64
 cgo_enabled      0
 db               ddev/ddev-dbserver-mariadb-10.11:v1.23.1
 ddev-ssh-agent   ddev/ddev-ssh-agent:v1.23.1
 docker           26.1.3
 docker-api       1.45
 docker-compose   v2.27.0
 docker-platform  linux-docker
 mutagen          0.17.2
 os               linux
 router           ddev/ddev-traefik-router:v1.23.1
 web              ddev/ddev-webserver:v1.23.1

======== proxy settings ========

 HTTP_PROXY=''
 HTTPS_PROXY=''
 http_proxy=''
 NO_PROXY=''

======== DDEV global info ========
developer-mode=false
disable-http2=false
fail-on-hook-fail=false
instrumentation-opt-in=true
instrumentation-queue-size=0
instrumentation-reporting-interval=0s
instrumentation-user=
internet-detection-timeout-ms=3000
last-started-version=v1.23.1
letsencrypt-email=
mailpit-http-port=8025
mailpit-https-port=8026
mkcert-caroot=/home/hannes/.local/share/mkcert
no-bind-mounts=false
omit-containers=[]
performance-mode=none
project-tld=ddev.site
required-docker-compose-version=v2.27.0
router=traefik
router-bind-all-interfaces=false
router-http-port=80
router-https-port=443
simple-formatting=false
table-style=default
traefik-monitor-port=10999
use-docker-compose-from-path=false
use-hardened-images=false
use-letsencrypt=false
wsl2-no-windows-hosts-mgt=false
xdebug-ide-location=

======== DOCKER provider info ========
docker client location: -rwxr-xr-x 1 root root 39732136 mai   16 11:33 /usr/bin/docker

docker client alternate locations:
/usr/bin/docker
/bin/docker

Docker provider: linux-docker
docker version:
Client: Docker Engine - Community
 Version:           26.1.3
 API version:       1.45
 Go version:        go1.21.10
 Git commit:        b72abbb
 Built:             Thu May 16 08:33:49 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.1.3
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.10
  Git commit:       8e96db1
  Built:            Thu May 16 08:33:49 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.32
  GitCommit:        8b3b7ca2e5ce38e8f31a34f35b2b68ceb8470d89
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

DOCKER_DEFAULT_PLATFORM=notset
Docker platform: linux-docker
Using Docker context: default (unix:///var/run/docker.sock)
docker-compose: v2.27.0
Using DOCKER_HOST=unix:///var/run/docker.sock
Docker version: 26.1.3
Able to run simple container that mounts a volume.
Able to use internet inside container.
Docker disk space:Filesystem                Size      Used Available Use% Mounted on
overlay                 469.8G    353.0G     92.8G  79% /

======== Existing docker containers ========
CONTAINER ID   IMAGE                                        COMMAND                  CREATED        STATUS                     PORTS     NAMES
562b8a82216e   bitnami/nginx:1.25.3-debian-11-r1            "/lando-entrypoint.s…"   33 hours ago   Exited (137) 8 hours ago             examplecom_appserver_nginx_1
2ebc3125a7a5   mailhog/mailhog:v1.0.1                       "/lando-entrypoint.s…"   33 hours ago   Exited (137) 8 hours ago             examplecom_mailhog_1
ff6756b722e2   phpmyadmin/phpmyadmin:5.1.1                  "/lando-entrypoint.s…"   33 hours ago   Exited (137) 8 hours ago             examplecom_phpmyadmin_1
6b358ce81c39   dehy/adminer                                 "/lando-entrypoint.s…"   33 hours ago   Exited (137) 8 hours ago             examplecom_adminer_1
bfbde9e56e48   drupalci/webdriver-chromedriver:production   "/lando-entrypoint.s…"   33 hours ago   Exited (137) 8 hours ago             examplecom_chrome_1
7b7a7e9ce5bb   bitnami/memcached:1.6.1-debian-10-r4         "/lando-entrypoint.s…"   33 hours ago   Exited (137) 8 hours ago             examplecom_memcached_1
41e26d64fa12   devwithlando/php:8.1-fpm-4                   "/lando-entrypoint.s…"   33 hours ago   Exited (137) 8 hours ago             examplecom_appserver_1
c41a45e8a74a   bitnami/mysql:8.0.19-debian-10-r57           "/lando-entrypoint.s…"   33 hours ago   Exited (137) 8 hours ago             examplecom_database_1
67cdf3c7b7be   backstopjs/backstopjs:6.2.2                  "/lando-entrypoint.s…"   33 hours ago   Exited (137) 8 hours ago             examplecom_node_1
7760de656b79   bitnami/mysql:5.7.29-debian-10-r51           "/lando-entrypoint.s…"   3 days ago     Exited (137) 2 days ago              codequality_database_1
3c28155b3896   devwithlando/php:8.1-apache-4                "/lando-entrypoint.s…"   3 days ago     Exited (137) 2 days ago              codequality_appserver_1
7d0f982bada4   devwithlando/php:8.2-fpm-4                   "/lando-entrypoint.s…"   8 days ago     Exited (137) 8 days ago              downloadexamplecom_appserver_1
f80ab4746a0c   ppodgorsek/robot-framework                   "run-tests-in-virtua…"   9 days ago     Exited (1) 9 days ago                agitated_bohr
f3450615fdf3   ppodgorsek/robot-framework                   "run-tests-in-virtua…"   9 days ago     Exited (1) 9 days ago                crazy_bell
144b7552b0d4   ppodgorsek/robot-framework                   "run-tests-in-virtua…"   9 days ago     Exited (252) 9 days ago              optimistic_hopper
65b16674d1ec   ppodgorsek/robot-framework                   "run-tests-in-virtua…"   9 days ago     Exited (252) 9 days ago              xenodochial_galileo
2d6bc4846c00   ppodgorsek/robot-framework                   "run-tests-in-virtua…"   9 days ago     Exited (252) 9 days ago              adoring_wozniak
95832e099d33   robot-test                                   "./entrypoint.sh tes…"   9 days ago     Exited (1) 9 days ago                blissful_shockley
931c9358b2f9   fc3ed48e6e1d                                 "./entrypoint.sh tes…"   9 days ago     Created                              eloquent_goldstine
9714c34dff4f   traefik:2.2.0                                "/lando-entrypoint.s…"   10 days ago    Exited (137) 8 hours ago             landoproxyhyperion5000gandalfedition_proxy_1
ab74b3b4e128   bitnami/mysql:5.7.29-debian-10-r51           "/lando-entrypoint.s…"   11 days ago    Exited (137) 11 days ago             drupal10_database_1
a9e8dd1273b7   bitnami/nginx:1.18.0-debian-10-r363          "/lando-entrypoint.s…"   11 days ago    Exited (137) 11 days ago             drupal10_appserver_nginx_1
4f32c6aad7e7   devwithlando/php:8.2-fpm-4                   "/lando-entrypoint.s…"   11 days ago    Exited (137) 11 days ago             drupal10_appserver_1
2e967b0a1c08   devwithlando/php:8.1-fpm-4                   "/lando-entrypoint.s…"   11 days ago    Exited (137) 8 days ago              developerexamplecom_appserver_1
4a5977a6cf3e   bitnami/mysql:8.0.19-debian-10-r57           "/lando-entrypoint.s…"   11 days ago    Exited (137) 11 days ago             developerexamplecom_database_1
89d70feeec5a   node:16                                      "/lando-entrypoint.s…"   11 days ago    Exited (137) 11 days ago             developerexamplecom_node_1
e82faa70338d   localstack/localstack:latest                 "/lando-entrypoint.s…"   2 weeks ago    Exited (137) 2 weeks ago             examplecom_mock-aws_1
37e3b2a07afd   mailhog/mailhog:v1.0.1                       "/lando-entrypoint.s…"   2 weeks ago    Exited (137) 2 weeks ago             examplecom_mailhog_1
357654012d9b   bitnami/nginx:1.25.3-debian-11-r1            "/lando-entrypoint.s…"   2 weeks ago    Exited (137) 2 weeks ago             examplecom_appserver_nginx_1
cdf637ef5777   cars10/elasticvue:1.0.7                      "/lando-entrypoint.s…"   2 weeks ago    Exited (137) 2 weeks ago             examplecom_elasticvue_1
8bab3db19c29   bitnami/mysql:8.0.19-debian-10-r57           "/lando-entrypoint.s…"   2 weeks ago    Exited (137) 2 weeks ago             examplecom_database_1
a59b61cee7a8   opensearchproject/opensearch:2.11.1          "/lando-entrypoint.s…"   2 weeks ago    Exited (137) 2 weeks ago             examplecom_elasticsearch_1
3b897d80c15c   devwithlando/php:8.2-fpm-4                   "/lando-entrypoint.s…"   2 weeks ago    Exited (137) 2 weeks ago             examplecom_appserver_1
0fe15c00cd74   node:18                                      "/lando-entrypoint.s…"   2 weeks ago    Exited (137) 2 weeks ago             examplecom_node_1

======== mkcert information ========
/usr/bin/mkcert
/bin/mkcert
/home/hannes/.local/share/mkcert
total 8
-r-------- 1 hannes hannes 2484 dets   7 10:42 rootCA-key.pem
-rw-r--r-- 1 hannes hannes 1805 dets   7 10:42 rootCA.pem

======== ping attempt on ddev.site ========
PING dkdkd.ddev.site (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from localhost (<IP_ADDRESS>): icmp_seq=1 ttl=64 time=0.016 ms

--- dkdkd.ddev.site ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.016/0.016/0.016/0.000 ms

======== ddev debug refresh ========
Rebuilding project images...
Rebuilding web image with `/home/hannes/.ddev/bin/docker-compose -f /home/hannes/Sites/tryddevproject-28389/.ddev/.ddev-docker-compose-full.yaml build web --no-cache`
#0 building with "default" instance using docker driver

#1 [web internal] load build definition from Dockerfile
#1 transferring dockerfile: 1.23kB done
#1 DONE 0.0s

#2 [web internal] load metadata for docker.io/ddev/ddev-webserver:v1.23.1
#2 DONE 0.0s

#3 [web internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s

#4 [web 1/7] FROM docker.io/ddev/ddev-webserver:v1.23.1
#4 CACHED

#5 [web 2/7] RUN (groupadd --gid 1000 "hannes" || groupadd "hannes" || true) && (useradd  -l -m -s "/bin/bash" --gid "hannes" --comment '' --uid 1000 "hannes" || useradd  -l -m -s "/bin/bash" --gid "hannes" --comment '' "hannes" || useradd  -l -m -s "/bin/bash" --gid "1000" --comment '' "hannes" || useradd -l -m -s "/bin/bash" --comment '' hannes )
#5 DONE 0.4s

#6 [web 3/7] RUN export XDEBUG_MODE=off; composer self-update --stable || composer self-update --stable || true; composer self-update --2 || composer self-update --2 || true
#6 0.544 Storing "stable" as default update channel for the next self-update run.
#6 0.764 You are already using the latest available Composer version 2.7.6 (stable channel).
#6 1.068 You are already using the latest available Composer version 2.7.6 (2.x channel).
#6 DONE 1.1s

#7 [web 4/7] RUN mkdir -p /home/hannes && chown hannes /home/hannes && chmod 600 /home/hannes/.pgpass
#7 DONE 0.4s

#8 [web 5/7] RUN apt update
#8 0.350
#8 0.350 WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
#8 0.350
#8 0.465 Get:1 http://deb.debian.org/debian bookworm InRelease [151 kB]
#8 0.501 Get:2 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB]
#8 0.520 Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
#8 0.556 Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8786 kB]
#8 0.648 Get:5 https://packages.sury.org/php bookworm InRelease [7542 B]
#8 0.677 Get:6 https://deb.nodesource.com/node_20.x nodistro InRelease [12.1 kB]
#8 0.782 Get:7 https://packages.sury.org/php bookworm/main amd64 Packages [229 kB]
#8 0.819 Get:8 http://nginx.org/packages/debian bookworm InRelease [2860 B]
#8 0.841 Get:9 https://dl.cloudsmith.io/public/platformsh/cli/deb/debian bookworm InRelease [3675 B]
#8 0.848 Get:10 https://deb.nodesource.com/node_20.x nodistro/main amd64 Packages [7455 B]
#8 0.917 Get:11 http://nginx.org/packages/debian bookworm/nginx amd64 Packages [7975 B]
#8 1.010 Get:12 https://dl.cloudsmith.io/public/platformsh/upsun-cli/deb/debian bookworm InRelease [3675 B]
#8 1.139 Get:13 http://packages.blackfire.io/debian any InRelease [28.0 kB]
#8 1.219 Get:14 https://dl.cloudsmith.io/public/symfony/stable/deb/debian bookworm InRelease [4422 B]
#8 1.442 Get:15 https://dl.cloudsmith.io/public/platformsh/cli/deb/debian bookworm/main amd64 Packages [3310 B]
#8 1.552 Get:16 https://dl.cloudsmith.io/public/platformsh/upsun-cli/deb/debian bookworm/main amd64 Packages [2193 B]
#8 1.672 Get:17 https://dl.cloudsmith.io/public/symfony/stable/deb/debian bookworm/main amd64 Packages [6917 B]
#8 2.100 Get:18 http://deb.debian.org/debian bookworm-updates/main amd64 Packages [13.8 kB]
#8 2.103 Get:19 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [157 kB]
#8 2.411 Get:20 http://packages.blackfire.io/debian any/main amd64 Packages [26.2 kB]
#8 2.740 Fetched 9557 kB in 2s (4048 kB/s)
#8 2.740 Reading package lists...
#8 3.098 Building dependency tree...
#8 3.195 Reading state information...
#8 3.210 7 packages can be upgraded. Run 'apt list --upgradable' to see them.
#8 DONE 3.3s

#9 [web 6/7] RUN curl -I https://www.google.com
#9 0.316   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
#9 0.316                                  Dload  Upload   Total   Spent    Left  Speed
#9 0.316
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0HTTP/2 200
#9 0.405 content-type: text/html; charset=ISO-8859-1
#9 0.405 content-security-policy-report-only: object-src 'none';base-uri 'self';script-src 'nonce-4JX1N7Fq41gXlyRQj8ykJg' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
#9 0.405 p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
Project images built in 2s.
 Network ddev-tryddevproject-28389_default  Created
 Container ddev-tryddevproject-28389-web  Created
 Container ddev-tryddevproject-28389-db  Created
 Container ddev-tryddevproject-28389-db  Started
 Container ddev-tryddevproject-28389-web  Started
Waiting for web/db containers to become ready: [web db]
Starting ddev-router if necessary...
 Container ddev-router  Created
 Container ddev-router  Started
Waiting for additional project containers to become ready...
All project containers are now ready.

======== Project startup ========
Starting tryddevproject-28389...
2024-05-24T22:01:34.375 Pulled image for ddev/ddev-webserver:v1.23.1
2024-05-24T22:01:34.375 Pulled image for busybox:stable
2024-05-24T22:01:34.376 Pulled image for ddev/ddev-ssh-agent:v1.23.1
2024-05-24T22:01:34.376 Pulled image for ddev/ddev-traefik-router:v1.23.1
2024-05-24T22:01:35.501 chowning /mnt/ddev-global-cache and /var/lib/mysql to 1000
2024-05-24T22:01:36.11 done chowning /mnt/ddev-global-cache and /var/lib/mysql to 1000
2024-05-24T22:01:36.115 host.docker.internal='' runtime.GOOS==linux and docker 20.10+
2024-05-24T22:01:36.157 Pulled image for ddev/ddev-dbserver-mariadb-10.11:v1.23.1
2024-05-24T22:01:36.157 Pulled image for ddev/ddev-webserver:v1.23.1
2024-05-24T22:01:36.158 Pulled image for ddev/ddev-ssh-agent:v1.23.1
2024-05-24T22:01:36.158 Pulled image for ddev/ddev-traefik-router:v1.23.1
Building project images...
2024-05-24T22:01:36.17 Executing docker-compose -f /home/hannes/Sites/tryddevproject-28389/.ddev/.ddev-docker-compose-full.yaml build --progress=plain
.Project images built in 1s.
2024-05-24T22:01:37.172 Executing docker-compose -f /home/hannes/Sites/tryddevproject-28389/.ddev/.ddev-docker-compose-full.yaml up -d
 Container ddev-tryddevproject-28389-web  Recreate
 Container ddev-tryddevproject-28389-db  Recreate
 Container ddev-tryddevproject-28389-db  Recreated
 Container ddev-tryddevproject-28389-web  Recreated
 Container ddev-tryddevproject-28389-web  Started
 Container ddev-tryddevproject-28389-db  Started
2024-05-24T22:01:48.919 VIRTUAL_HOST=tryddevproject-28389.ddev.site for web
2024-05-24T22:01:48.919 HTTP_EXPOSE=80:80,8025:8025 for web
2024-05-24T22:01:48.919 HTTPS_EXPOSE=443:80,8026:8025 for web
2024-05-24T22:01:49.446 Copied /home/hannes/Sites/tryddevproject-28389/.ddev/traefik:CopyIntoVolume_sstfhmrhqdwx into /mnt/v/traefik in 63.089021ms
2024-05-24T22:01:49.475 Exec chown -R 1000 /mnt/v/traefik stdout=, stderr=, err=<nil>
2024-05-24T22:01:49.699 Copied Traefik certs in /home/hannes/Sites/tryddevproject-28389/.ddev/traefik/certs to ddev-global-cache/traefik
2024-05-24T22:01:49.699 Running /start.sh in ddev-webserver
Waiting for web/db containers to become ready: [web db]
2024-05-24T22:01:52.763 Copied /home/hannes/.ddev/commands:CopyIntoVolume_lykmyfaocnzi into /mnt/v/global-commands in 73.191772ms
2024-05-24T22:01:52.787 Exec chown -R 1000 /mnt/v/global-commands stdout=, stderr=, err=<nil>
2024-05-24T22:01:53.093 Testing to see if /mnt/ddev_config is properly mounted
Starting ddev-router if necessary...
2024-05-24T22:01:53.653 Copied /home/hannes/.ddev/traefik:CopyIntoVolume_zulekudoqnkc into /mnt/v/traefik in 54.268116ms
2024-05-24T22:01:53.677 Exec chown -R 1000 /mnt/v/traefik stdout=, stderr=, err=<nil>
2024-05-24T22:01:53.928 Copied global Traefik config in /home/hannes/.ddev/traefik/certs to ddev-global-cache/traefik
 Container ddev-router  Running
2024-05-24T22:01:53.961 Waiting for ddev-router to become ready. docker inspect --format "{{json .State.Health }}" ddev-router
2024-05-24T22:01:54.465 ddev-router is ready
Waiting for additional project containers to become ready...
All project containers are now ready.
Successfully started tryddevproject-28389
Project can be reached at https://tryddevproject-28389.ddev.site https://<IP_ADDRESS>:32888

======== Curl of site from inside container ========
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 24 May 2024 19:01:56 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding

======== curl -I of http://<IP_ADDRESS>:32889 (web container http docker bind port) from outside ========
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 24 May 2024 19:01:56 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding

======== curl -I of http://tryddevproject-28389.ddev.site (router http URL) from outside ========
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Date: Fri, 24 May 2024 19:01:56 GMT
Server: nginx
Vary: Accept-Encoding

======== Full curl of http://tryddevproject-28389.ddev.site (router http URL) from outside ========
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   198    0   198    0     0  14213      0 --:--:-- --:--:-- --:--:-- 15230
Success accessing database... db via TCP/IP<br />
ddev is working.<br />
The output file for Discord or issue queue is in
<b>/tmp/ddev-debug-test.txt</b><br />
file:///tmp/ddev-debug-test.txt<br />

======== Full curl of https://tryddevproject-28389.ddev.site (router https URL) from outside ========
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   198    0   198    0     0   4185      0 --:--:-- --:--:-- --:--:--  4212
Success accessing database... db via TCP/IP<br />
ddev is working.<br />
The output file for Discord or issue queue is in
<b>/tmp/ddev-debug-test.txt</b><br />
file:///tmp/ddev-debug-test.txt<br />

======== Curl google.com to check internet access and VPN ========
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
HTTP/2 200
content-type: text/html; charset=ISO-8859-1
content-security-policy-report-only: object-src 'none';base-uri 'self';script-src 'nonce-BamjlQG87rSGP--gHjt_ig' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
date: Fri, 24 May 2024 19:01:56 GMT
server: gws
x-xss-protection: 0
x-frame-options: SAMEORIGIN
expires: Fri, 24 May 2024 19:01:56 GMT
cache-control: private
set-cookie: AEC=AQTF6HwpXD9KkickOLgmfaiNGPdrQRqHX3aohdz_Do_3Ao51i8x6XgH8reY; expires=Wed, 20-Nov-2024 19:01:56 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
set-cookie: __Secure-ENID=19.SE=LFnvnhO9sRWVurppLNdftix64BHSZiLhalZIPIOPdy6yfgYAQZW-S439vHj131Sdi6tyDme06hu-0CFjPA8ApGkUxc5CdKKbsYmb1DnjtqrP3AKD3J6K_60Kjh1tpebTP_58F9PXkuoPo-pILXVb1M6ocFLjt4EKQdb5C9o6rCohix1b2zEdIA; expires=Tue, 24-Jun-2025 11:20:14 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

======== host.docker.internal status ========
PING host.docker.internal (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from host.docker.internal (<IP_ADDRESS>): icmp_seq=1 ttl=64 time=0.049 ms

--- host.docker.internal ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.049/0.049/0.049/0.000 ms

======== Project ownership on host ========
drwxrwxr-x 4 hannes hannes 4096 mai   24 22:00 ../tryddevproject-28389

======== Project ownership in container ========
drwxrwxr-x 4 hannes hannes 4096 May 24 19:00 //var/www/html

======== In-container filesystem ========
Filesystem     Type 1K-blocks      Used Available Use% Mounted on
/dev/nvme0n1p3 ext4 489161968 435907112  28333512  94% /var/www/html

======== Thanks for running the diagnostic! ========
Running ddev launch in 3 seconds
Running ddev launch
Waiting for ddev launch to complete before deleting project
Opening in existing browser session.

Cleanup: deleting test project tryddevproject-28389
 Container ddev-tryddevproject-28389-db  Stopped

Expected Behavior
I expect MySQL 8 to import as fast as MariaDB or at least as close to it. Right now the difference is too big. I imported 3x times 14GB mysql file and the import speed with Maria DB was:

18m 24s
15m 53s
15m 56s

Actual Behavior
I didn't have time to do 3 tests atm, but the first test with MySQL 8 was:

59m 16s

This is something I have felt that's wrong somewhere, but I wasn't sure what. We changed to MySQL 1-2 months ago and previously I was used to 10-15 minutes import time for the big SQL with MariaDB.
Steps To Reproduce

Nothing special in my env:

name: example.com
type: drupal10
docroot: web
php_version: "8.1"
webserver_type: nginx-fpm
xdebug_enabled: false
additional_hostnames: []
additional_fqdns: []
database:
    type: mysql
    version: "8.0"
use_dns_when_possible: true
composer_version: "2.7.2"
disable_settings_management: true
web_environment: []
nodejs_version: "18"
# Fix "Failed waiting for web/db containers to become ready" when restoring snapshot.
# @todo Still still getting 2 minute warning so let's deal with this later.
default_container_timeout: "180"

This is script that I tested with:

#!/bin/bash

# Testing import speed

# Test 1
ddev delete --omit-snapshot --yes && ddev start
time ddev import-db --file=./db.sql.gz

# Test 2
ddev delete --omit-snapshot --yes && ddev start
time ddev import-db --file=./db.sql.gz

# Test 3
ddev delete --omit-snapshot --yes && ddev start
time ddev import-db --file=./db.sql.gz

Anything else?
No response

I think you should probably open an issue with MySQL rather than DDEV :)
It also makes a lot of difference what Docker provider you're using, as that is key here, as @douggreen has demonstrated many times.
Also, MariaDB 10.4 is nearly obsolete, you should be comparing with MariaDB 10.11 if anything. But MySQL 8.0 is more complex than older MariaDB and I would expect some difference.
You also seem to have a post-import-db hook. I would expect that to impact your tests.
And you have lots of customizations. Please do exploration like this on a plain vanilla project
.ddev/docker-compose.mounts.yaml
.ddev/php/php-example.ini.example
.ddev/php/php.ini
.ddev/nginx_full/nginx-site.conf

and you have .ddev/config.wunderio.yaml
You don't mention how much memory you have on your thinkpad, which can also be a factor.
Closing this for now, please continue your exploration, I'd love to hear more, and I invite you to do a guest blog post on ddev.com with your results.

I know in Lando they do some overrides for database to speed things up - at least for the writing part. One was this https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_flush_log_at_trx_commit and that speeds up things like 2x times if you change it from default 1 to either 0 or 2. So you're saying you just get default database image and don't adjust settings for MariaDB for example?
I did have db post hook, but I emptied the script. But I do understand why testing this in vanilla would be better.
The slowness can be seen already from the import status. With MySQL 8 it jumps to 2+ hours and stays there for long time:

With MariaDB it's more optimistic:

DDEV's configuration is at https://github.com/ddev/ddev/blob/master/containers/ddev-dbserver/files/etc/my.cnf
You can also see it by ddev ssh -s db and view /etc/.cnf (use cat /etc/my.cnf or vi /etc/my.cnf
You can override it, see https://ddev.readthedocs.io/en/stable/users/extend/customization-extendibility/#custom-mysqlmariadb-configuration-mycnf, and anything you learn will be welcome.
Sometimes when you import a database dump like this, the indexes get built as it loads, which is quite costly, especially on certain types of database. That is why DDEV has snapshots, which are in binary form and can be loaded without index creation.
Note that the actual structure of the database you're loading may be related to the behavior you're seeing, and of course the priorities of one database vendor over the other.
If you haven't already read https://ddev.readthedocs.io/en/stable/users/usage/database-management/ please do, thanks!

I did another benchmark on Vanilla DDEV and latest Maria DB. I also unpacked the db to only count importing time.
MariaDB
Test 1: 10m 27s
Test 2: 9m 54s
Test 3:  9m 49s
MySQL 8
Test 1: 60m 54s
Test 2: 59m 18s
Test 3: 59m 35s
So the difference is not 4x times now, but 6x.
I created setup with 2 branches in this repo: https://github.com/hkirsman/ddev-db-import-bench
I guess one reason I'm here because snapshot tool is also broken atm as it times out on large db's. I understand it's in roadmap for next minor version.

Snapshot doesn’t actually timeout. You should tail the log withddev logs -fs dbAnd wait for it to complete.What’s broken is setting the timeout that would allow this to run in the shell and not return until it is complete.My imports take over an hour (with mariadb). I turn off indexing during the import and turn it back on afterwards. Snapshot saves me hours every day.DougSent from my iPhonePlease excuse brevity and typosOn May 25, 2024, at 3:32 PM, Hannes Kirsman @.***> wrote:
I did another benchmark on Vanilla DDEV and latest Maria DB. I also unpacked the db to only count importing time.
MariaDB
Test 1: 10m 27s
Test 2: 9m 54s
Test 3:  9m 49s
MySQL 8
Test 1: 60m 54s
Test 2: 59m 18s
Test 3: 59m 35s
So the difference is not 4x times now, but 6x.
I created setup with 2 branches in this repo: https://github.com/hkirsman/ddev-db-import-bench
I guess one reason I'm here because snapshot tool is also broken atm as it times out on large db's. I understand it's in roadmap for next minor version.
Tx for the info!
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

As Doug says, the problem is just that the timeout isn't working correctly,

https://github.com/ddev/ddev/issues/5133

If you ddev logs -s db -f you'll see that ddev snapshot restore continues on just fine.
I'm glad you're exploring this territory, and you'll probably want to look more broadly.

Are there issues in mariadb or mysql bug trackers about the difference?
Is this a problem just for your particular database, or does this happen to any database? (For example, if a flat set of rows in one table with no index, what happens?)
Maybe Doug's technique "I turn off indexing during the import and turn it back on afterwards" will help you as well.

I have these hooks on my sites with large databases. tbh, I added these
some time ago, and have not re-evaluated if they are helpful.
hooks:
pre-import-db:

exec: mysql -uroot -proot -e "ALTER INSTANCE DISABLE INNODB REDO_LOG;"
post-import-db:
exec: mysql -uroot -proot -e "ALTER INSTANCE ENABLE INNODB REDO_LOG;"

On Sat, May 25, 2024 at 8:29 PM Randy Fay @.***> wrote:

As Doug says, the problem is just that the timeout isn't working correctly,

#5133 https://github.com/ddev/ddev/issues/5133

If you ddev logs -s db -f you'll see that ddev snapshot restore continues
on just fine.
I'm glad you're exploring this territory, and you'll probably want to look
more broadly.

Are there issues in mariadb or mysql bug trackers about the
difference?
Is this a problem just for your particular database, or does this
happen to any database? (For example, if a flat set of rows in one table
with no index, what happens?)
Maybe Doug's technique "I turn off indexing during the import and
turn it back on afterwards" will help you as well.

—
Reply to this email directly, view it on GitHub
https://github.com/ddev/ddev/issues/6244#issuecomment-2131793026, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJPZNMAWTTCKS3TVAP5HTLZEEUHHAVCNFSM6AAAAABIIDURROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZRG44TGMBSGY
.
You are receiving this because you were mentioned.Message ID:
@.***>

I hope you can figure out the difference! It will be great. You can obviously use the lando mysql config in your .ddev/mysql/ to test this. If you would add a link to the actual lando mysql config here, in your repo or a gist or whatever, or if it's short right in the issue or as an attached file.

In another site lot smaller db I got this result when trying Landos mysql conf (it's lot faster):

Default
ddev import-db --file=./down-prod-2024-05-16.sql.gz
mysql: [Warning] Using a password on the command line interface can be insecure.
2.44GiB 0:14:01 [2.97MiB/s] [====================================================>] 100%

dev with https://github.com/lando/drupal/blob/main/config/drupal10/mysql8.cnf

✗ ddev import-db --file=./down-prod-2024-05-16.sql.gz
mysql: [Warning] Using a password on the command line interface can be insecure.
2.44GiB 0:10:56 [3.81MiB/s] [====================================================>] 100%
But still with Lando it's like 5 minutes or less... only difference I can think of is that Lando is using bitnami. Should I try to create custom service and try something there? Or should I try come up with dockerfile that directly spins up the containers and test there? What's the mysql 8 image DDEV is using?

I think I got it. Listed image in my system is this:
ddev/ddev-dbserver-mysql-8.0
Dockerhub https://hub.docker.com/r/ddev/ddev-dbserver-mysql-8.0
Github https://github.com/ddev/ddev/tree/master/containers/ddev-dbserver

Would love to have you try it with bitnami, it's pretty easy to do. Related:

https://github.com/ddev/ddev/pull/5640

Bitnami only recently started providing arm64 images, and that made it so we might be able to use it as an upstream in the future.
Currently, we use for amd64 (your use) the official mysql image. For arm64 we have to build our own using Ubuntu base.
My prediction would be that the difference will be the my.cnf/mysql.cnf, but your explorations are excellent.

So somewhere you check out that branch and then you have bitnami... but where in your system? How do you develop DDEV?

That PR didn't work out and never matured enough for you to try out. But you can use the bitnami/mysql image for testing (and compare it directly against the mysql:8.0 official image as well), see https://hub.docker.com/r/bitnami/mysql

For more about DDEV development, see https://ddev.readthedocs.io/en/stable/developers/
The image management stuff is in https://ddev.readthedocs.io/en/stable/developers/release-management/

I'm going to reopen this since you're doing such a good job exploring it. Thanks for the concerted effort!

Tx!
Got these results now:
13m 9s
12m 16s
11m 56s
The config is a bit of mess, put together from 3 different locations (mainly from Land), but it's good start:
#
# The MySQL database server configuration file for Lando
#
# LANDODRUPALMYSQL8CNF

[mysqld]

#from bitnami container
skip-name-resolve
explicit_defaults_for_timestamp
#basedir=/opt/bitnami/mysql
port=3306
#tmpdir=/opt/bitnami/mysql/tmp
#socket=/opt/bitnami/mysql/tmp/mysql.sock
#pid-file=/opt/bitnami/mysql/tmp/mysqld.pid
max_allowed_packet=16M
#bind-address=<IP_ADDRESS>
#log-error=/opt/bitnami/mysql/logs/mysqld.log
character-set-server=UTF8
collation-server=utf8_general_ci
#plugin_dir=/opt/bitnami/mysql/plugin

# from github

#
# * Basic Settings
#
# Data is stored in a volume on the db container /sql
default-storage-engine = innodb

#
# * Fine Tuning
#
key_buffer_size    = 384M
max_allowed_packet  = 32M
thread_stack    = 400K
thread_cache_size       = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
#max_connections        = 100
#table_cache            = 64
#thread_concurrency     = 10
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 64M
table_open_cache = 512
sort_buffer_size = 2M
read_buffer_size = 2M

#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file        = /src/.lando/log/mysql.log
#general_log             = 1
#
# Error logging goes to syslog due to /etc/mysql/conf.d/mysqld_safe_syslog.cnf.
#
# Here you can see queries with especially long duration
#log_slow_queries = /var/log/mysql/mysql-slow.log
#long_query_time = 2
#log-queries-not-using-indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
#       other settings you may need to change.
#server-id    = 1
#log_bin      = /src/.lando/log/mysql-bin.log
expire_logs_days = 10
max_binlog_size = 101M
#binlog_do_db   = include_database_name
#binlog_ignore_db = include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# Uncomment the following if you are using InnoDB tables
#innodb_data_file_path = ibdata1:2000M;ibdata2:10M:autoextend
#innodb_log_group_home_dir = C:\mysql\data/
# You can set .._buffer_pool_size up to 50 - 80 %
# of RAM but beware of setting memory usage too high
#innodb_buffer_pool_size = 384M
#innodb_additional_mem_pool_size = 20M
# Set .._log_file_size to 25 % of buffer pool size
innodb_log_file_size = 100M
#innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 0
#innodb_lock_wait_timeout = 50
innodb_buffer_pool_size = 384M
innodb_log_buffer_size  = 4M
innodb_file_per_table   = 1
innodb_open_files       = 256
innodb_io_capacity      = 512
innodb_flush_method     = O_DIRECT
innodb_thread_concurrency = 8
innodb_lock_wait_timeout = 127
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem

# https://dba.stackexchange.com/questions/83125/mysql-any-way-to-import-a-huge-32-gb-sql-dump-faster

innodb_buffer_pool_size = 4G
innodb_log_buffer_size = 256M
innodb_log_file_size = 1G
innodb_write_io_threads = 16
innodb_flush_log_at_trx_commit = 0
innodb_doublewrite = 0

[mysqldump]
quick
quote-names
max_allowed_packet  = 32M

[mysql]
#no-auto-rehash # faster start of mysql but no tab completion

[isamchk]
key_buffer_size = 384M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

Looking forward to a .ddev/mysql/import.cnf that demonstrates the fix! And then a PR.
I don't understand isamchk here as ISAM is long obsolete and shouldn't be a part of anything... We're using Innodb...

I've done lot of testing and this is what I've come up with:
innodb_write_io_threads = 16
innodb_doublewrite = 0
innodb_log_file_size = 100MB

Originally the import takes 60 minutes. With the above config it takes 13.5 minutes.
Most impact gives innodb_doublewrite = 0. From 60 minutes to 15.3 minutes.
With innodb_log_file_size I tested with 1MB, 10MB, 20MB etc until 90MB because originally I had it at 1000 MB:

At 70MB the import speed did not increase. According to MariaDB https://mariadb.com/kb/en/innodb-system-variables/#innodb_log_file_size that value is:
100663296 (96MB) (>= MariaDB 10.5), 50331648 (48MB) (<= MariaDB 10.4)
So that's why I've rounded it 100MB, but maybe 96MB would better?
Checking my report, I think I need to test this again without setting innodb_write_io_threads at all.

Super interesting! Thanks for the good work. Could you please update the title and OP of the issue to reflect your current comparisons, and consider a PR when you're ready.

I am starting a test with these settings. Isthe innodb log size is helpful if you disable logging during port.I don’t have a good history of my tests. I believe that switching to orb drastically slowed down my imports. My last one took almost 3 hours and I perceive that they used to be closer to an hour.I’ll let you know what it is with these settings…Sent from my iPhonePlease excuse brevity and typosOn Jun 16, 2024, at 8:44 AM, Randy Fay @.***> wrote:
Super interesting! Thanks for the good work. Could you please update the title and OP of the issue to reflect your current comparisons, and consider a PR when you're ready.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

This seems to have radically improved the actual time from start to finish, but I'm confused that the user/system time didn't change much. Is this real?
Before 112.55s user 12.05s system 1% cpu 2:44:06.85 total
After 108.91s user 10.44s system 1% cpu 1:59:04.68 total

So it's 2 hours and 44 minutes initially and then 2 hours, right? I would have hoped even more speed increase. I'm not sure if you did add all 3 settings or not?
The user is basically 2 minutes and system only around 10 seconds. There's not much to change to in my oppionion. Also I've asked my colleague and he said it's how long 'client' (mysql or ddev) took cpu cycles, but stops counting after the job is sent off to import. The thing to focus on is cpu.

And on a second (supposedly identical) run, that 2:44 was now 2:15, so it was not consistent. I'll try with the log file size tonight after work.

Just a note: It's only the elapsed time that matters. User is user-level CPU, System is system-level CPU, but almost all of an import is waiting for I/O, so that's where the rest of the time is. DB imports aren't about CPU.

I always destroyed the instance before starting new test douggreen

I may have missed that step on the first test. I didn't on the second.
ddev delete --omit-snapshot
orb delete -fa
ddev restart
time ddev import-db --file=
On Mon, Jun 17, 2024 at 9:30 AM Hannes Kirsman @.***>
wrote:

I always destroyed the setup before starting new test douggreen
https://github.com/douggreen
—
Reply to this email directly, view it on GitHub
https://github.com/ddev/ddev/issues/6244#issuecomment-2173403160, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJPZNJAA2ICENTL3QXRULLZH3QGBAVCNFSM6AAAAABIIDURROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZTGQYDGMJWGA
.
You are receiving this because you were mentioned.Message ID:
@.***>

It would really be worthwhile to create a dummy large db dump and a standardized test for this. How about creating a repository specifically for this class of testing?

Pretty much no change with the 100MB log file

Before 112.55s user 12.05s system 1% cpu 2:44:06.85 total
After 108.91s user 10.44s system 1% cpu 1:59:04.68 total * I think this was a bad test *, I did another run of this which showed 2:15-ish
100MB log file 109.22s user 11.21s system 1% cpu 2:18:28.08 total

And there are no extra hooks or other settings anymore with the above tests? I did create totally new environment for myself to be sure and as suggested above (wondering if also something else could affect the db settings eg type drupal 10):
name: example.com
type: drupal10
docroot: web
php_version: "8.2"
webserver_type: nginx-fpm
xdebug_enabled: false
additional_hostnames: []
additional_fqdns: []
database:
    type: mysql
    version: "8.0"
use_dns_when_possible: true
composer_version: "2"
web_environment: []
corepack_enable: false

Also, my OS is Linux, running on ThinkPad X1 gen 9.
rfay I do agree on the dummy db.

innodb_write_io_threads=32

109.10s user 11.20s system 1% cpu 2:11:09.49 total
so threads makes almost no difference, remove innodb_write_io_threads, so using the default innodb_write_io_threads=4
109.86s user 11.40s system 1% cpu 2:16:13.53 total

@rfay I can optimize the default MySQL 8.0 config DDEV ships with so it provides better performance for general use cases (php apps - primarily drupal and WP)

Clear improvements are welcome. And the folks in this thread can test it. Mostly complaints are usually about import speed.

The only thing I've heard that's possibly conclusive here is that @hkirsman believe this is a significant improvement:
innodb_write_io_threads=16
innodb_doublewrite=0
``

Is that true? Any followup? I was hoping for a PR.

Here is the config which provides an immense boost.
#
# * Keys and buffers
#
key_buffer_size         = 0
sort_buffer_size        = 2M
join_buffer_size        = 128M

#
# * InnoDB
#
innodb_fast_shutdown            = 1
innodb_max_dirty_pages_pct      = 90
innodb_max_dirty_pages_pct_lwm  = 10
innodb_doublewrite              = 0
innodb_log_compressed_pages     = 0
innodb_lru_scan_depth           = 1024
innodb_page_cleaners            = 8
innodb_flush_neighbors          = 0
innodb_adaptive_flushing        = 1
innodb_flush_log_at_trx_commit  = 2
innodb_write_io_threads         = 8
innodb_read_io_threads          = 8
innodb_flush_method             = O_DIRECT_NO_FSYNC
innodb_use_native_aio           = 1
innodb_io_capacity              = 40000
innodb_io_capacity_max          = 50000
innodb_compression_level        = 2
innodb_stats_on_metadata        = 0

tl;dr; the only proposed change that had an impact on importing (ddev
import-db) was setting innodb_doublewrite=0 or setting
innodb_write_io_threads to match the number of cores. The latest
additional changes in this thread made importing slower.
Some quick testing on one of our smaller .sql files.
Standardizing the tests:

ddev delete --omit-snapshot -a && orb delete -fa
Then using CleanMyMacX to "Free Up Purgeable Space" and "Run
Maintenance Scripts"
Modify .ddev/mysql/myproject.cnf
orb start && ddev restart

I've removed our post processing hooks (drush updb -y && drush en -y
dblog stage_file_proxy && drush cc all).
hooks:
pre-import-db:

exec: mysql -uroot -proot -e "ALTER INSTANCE DISABLE INNODB REDO_LOG;"
post-import-db:
exec: mysql -uroot -proot -e "ALTER INSTANCE ENABLE INNODB REDO_LOG;"

The machine
Apple M1 Pro, 32GB RAM, 10 cores (8 performance and 2 efficiency),
Sonoma 14.5, ddev v1.23.3, about 500GB free disk space
Baseline Config
[mysqld]
max_connections=200
tmp_table_size=64M
max_heap_table_size=64M
sql-mode=NO_ENGINE_SUBSTITUTION

The tests
These tests were run with time ddev import-db --file=dbsuperdiet.sql.gz. This is a Drupal database dump that is
1.9G compressed. Time's below are total time only:
Test 0 - 8:43.18 - Baseline
Test 1 - 8:46.28 - change pre/post to not stop INNODB logging
Test 2 - 8:52.79 - add innodb_write_io_threads=16 only
Test 3 - 8:16.05 - add innodb_write_io_threads=8 only
Test 3 - 7:39.62 - add innodb_write_io_threads=10 only -
Test 5 - 7:48.83 - add innodb_doublewrite=0 only - this is surprising also,
Test 6 - 8:12.34 - add innodb_write_io_threads=16 &
innodb_doublewrite=0 - this has become our default since this thread
was first reported
Test 7 - 7:37.81 - add innodb_write_io_threads=10 &
innodb_doublewrite=0 - not much better than 3 or 5
Test 8 - 8:34.94 - add all the changes from below
Test 9 - 7:50.18 - changes from below but matching my number of cores of 10
Conclusion

My prior strategy of disabling and re-enabling INNODB had little impact.
The additional proposed changes make it slower.
If you do set any of the thread counts, don't exceed the number of cores.
Setting innodb_doublewrite=0 by itself is almost helpful as
anything and doesn't rely on the number of cores, so this is something
we can do in git shared configs.

I'll test the change to just remove setting innodb_write_io threads
with our larger database this weekend (44G, Test 3 above, The baseline
of 2:21:40.86 run just a few days ago is Test 6 above).
Doug
On Jul 19, 2024, at 3:48 AM, Sibin Grašić @.***> wrote:

Here is the config which provides an immense boost.

* Keys and buffers

key_buffer_size         = 0
sort_buffer_size        = 2M
join_buffer_size        = 128M

* InnoDB

innodb_fast_shutdown            = 1
innodb_max_dirty_pages_pct      = 90
innodb_max_dirty_pages_pct_lwm  = 10
innodb_doublewrite              = 0
innodb_log_compressed_pages     = 0
innodb_lru_scan_depth           = 1024
innodb_page_cleaners            = 8
innodb_flush_neighbors          = 0
innodb_adaptive_flushing        = 1
innodb_flush_log_at_trx_commit  = 2
innodb_write_io_threads         = 8
innodb_read_io_threads          = 8
innodb_flush_method             = O_DIRECT_NO_FSYNC
innodb_use_native_aio           = 1
innodb_io_capacity              = 40000
innodb_io_capacity_max          = 50000
innodb_compression_level        = 2
innodb_stats_on_metadata        = 0
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID:
@.***>

It looks to me like innodb_doublewrite=0 is a safe and reasonable thing for a local development environment:

https://mariadb.com/kb/en/innodb-doublewrite-buffer/
https://dba.stackexchange.com/questions/86636/when-is-it-safe-to-disable-innodb-doublewrite-buffering

See the PR, please do test it and provide your results, artifacts and instructions at https://github.com/ddev/ddev/pull/6411#issuecomment-2239288221
Here are my results: https://docs.google.com/spreadsheets/d/1a1eZVexjQZU4jbnwQxSir01X4n7K6M0jtDkT5T8lib0/edit?usp=sharing
It gives a modest improvement in MySQL 8.0 and MySQL 5.7, about 20% improvement - 437s instead of 551s
The improvement may be different based on the type of import, especially its structure; this is a random hugedb that I keep around for this kind of testing, and may not have the same structure as your databases.

On the MariaDB 10.11 line, is this showing the innodb_doublewrite=0 has no
impact (actually a few seconds slower)?
On Fri, Jul 19, 2024 at 11:56 AM Randy Fay @.***> wrote:

See the PR, please do test it and provide your results, artifacts and
instructions at #6411 (comment)
https://github.com/ddev/ddev/pull/6411#issuecomment-2239288221
Here are my results:
https://docs.google.com/spreadsheets/d/1a1eZVexjQZU4jbnwQxSir01X4n7K6M0jtDkT5T8lib0/edit?usp=sharing
It gives a modest improvement in MySQL 8.0 and MySQL 5.7, about 20%
improvement - 437s instead of 551s
The improvement may be different based on the type of import, especially
its structure; this is a random hugedb that I keep around for this kind
of testing, and may not have the same structure as your databases.
—
Reply to this email directly, view it on GitHub
https://github.com/ddev/ddev/issues/6244#issuecomment-2239505735, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJPZNKMDZK7X3NRVGBWLKTZNEZKNAVCNFSM6AAAAABIIDURROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZZGUYDKNZTGU
.
You are receiving this because you were mentioned.Message ID:
@.***>

On the MariaDB 10.11 line, is this showing the innodb_doublewrite=0 has no

Agreed. The difference in timing is certainly not statistically significant. (Each of these was tried only one time; in general it didn't seem to affect mariadb)

We are using mysql 8.0. It seems like we should switch to mariadb 10.11, or
at least test it. Do you agree?
On Fri, Jul 19, 2024 at 12:45 PM Randy Fay @.***> wrote:

On the MariaDB 10.11 line, is this showing the innodb_doublewrite=0 has no
Agreed. The difference in timing is certainly not statistically
significant. (Each of these was tried only one time; in general it didn't
seem to affect mariadb)
—
Reply to this email directly, view it on GitHub
https://github.com/ddev/ddev/issues/6244#issuecomment-2239620352, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJPZNJARS54Z6B5VEY4Q7LZNE7BJAVCNFSM6AAAAABIIDURROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZZGYZDAMZVGI
.
You are receiving this because you were mentioned.Message ID:
@.***>

We are using mysql 8.0. It seems like we should switch to mariadb 10.11, or at least test it. Do you agree?

As far as what your project chooses (I think that's what you're asking) it is probably a choice based on the capabilities of MySQL 8.0. These databases have diverged a bit over the years, with different features and strengths (and weaknesses).
For development import time can be important, but this is unusual for production, and production requirements usually win.

I didn't test the PR. But I think this change is solid.
I wondered if any single test can really tell us anything. So I wrote
a script to run the import 10 times. The script restarts ddev between
each run.
So I ran a baseline test of 10 runs on the smaller database:
Here is the baseline,

results: 8:35.53, 8:38.97, 9:04.61, 8:16.03, 7:44.10, 7:52.91,
7:33.34, 7:42.74, 8:12.99, 8:30.87
average: 8:13.21
mean: 8:15.51
stddev: 28.11 seconds

And then deleted dde & orb, free'd up purgeable memory, and then
re-ran the test with innodb_doublewrite=0

results: 7:40.57, 7:17.47, 8:33.01, 7:52.60, 7:15.46, 7:08.89,
7:43.72, 8:01.68, 7:18.94, 8:04.33
average: 7:41.67
mean: 7:42.14
stddev: 25.57

I want to do a similar test on our larger database (6.8G compressed,
44G uncompressed), but that takes 2 hours+ per run. I did do a single
import with and without innodb_doublewrite=0. Before was 2:20:28.21
and after was 2:01:14.75.
Doug
On Fri, Jul 19, 2024 at 1:04 PM Randy Fay @.***> wrote:

We are using mysql 8.0. It seems like we should switch to mariadb 10.11, or at least test it. Do you agree?
As far as what your project chooses (I think that's what you're asking) it is probably a choice based on the capabilities of MySQL 8.0. These databases have diverged a bit over the years, with different features and strengths (and weaknesses).
For development import time can be important, but this is unusual for production, and production requirements usually win.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.***>

@hkirsman @douggreen @seebeen I pulled

https://github.com/ddev/ddev/pull/6411

Would love to have you test HEAD now, it's super easy.
If you discover specific improvements in the future please open a new issue with the specific suggestion. It seemed that innodb_doublewrite=0 was the only one discovered in this round. It definitely provided improvement on import, but more like 20% for me rather than the hoped-for 40%.

Followups on this would include

A standardized large mariadb/mysql database (or perhaps more than one size) that has no personal information in it, but does have diverse structure.
A script that can run automated timing on a database like that, with different configurations.

This issue was explicitly about mysql:8.0, as many people have to use that. Yes, the PR is already in DDEV v1.23.4.

Another discussion in

https://github.com/orgs/ddev/discussions/6591

Hello, I post here after an initial discussion on drupal ddev slack (https://drupal.slack.com/archives/C5TQRQZRR/p1731656603549439)
We have a project with a large database (the dump is 15G uncompressed). When importing the db dump into ddev using ddev/ddev-dbserver-mysql-8.0:v1.23.5 , I noticed that the resulting volume in docker was much bigger than that, expecially because I had to do the import multiple times (to test something unrelated).
The db is 15 gigabytes, but the volume ended up taking 85 gigabytes, because of quite many big binlog. files:

This was most likely caused by my tests of importing the database then deleting it, and there is this setting currently that would delete the binlog after 1 day: https://github.com/ddev/ddev/blob/master/containers/ddev-dbserver/files/etc/my.cnf#L31
Now, I can disable this in my particular project by creating a small configuration file in .ddev/mysql/myconf.cnf (For example) and adding this:
[mysqld]
disable-log-bin=1

and after that:

my import operation got faster (6m 30s before, 5m 04s afterwards)
the disk space stayed at 20gigs

So my question is: are there disadvantages in disabling the binary log in mysql? Reading about it on the web, it seems that it's used for multi instance databases, recovery etc.. things that in local development don't sound very important.
Also in my test ddev snapshot seems to work happily without it.
So maybe we can save some CPU cycles and disk space by disabling it altogether?
Or maybe we can set a lower time for the binlog to be cleaned up with the newer binlog_expire_logs_seconds setting advertised here? https://dev.mysql.com/worklog/task/?id=10924

I see that binlog setting was used here a long time ago:

#1515

So my question is: are there disadvantages in disabling the binary log in mysql? Reading about it on the web, it seems that it's used for multi instance databases, recovery etc.. things that in local development don't sound very important.

Agreed, or if these logs are needed, they should be removed when doing database import (use RESET MASTER; ?):

https://dba.stackexchange.com/a/47048

I think the binlogs are mostly intended for replication and crash recovery. I'm pretty sure we could turn them off or limit them more. A PR is welcome. Go ahead and test with your own system, and when happy, suggest an improved approach.

I use this in my config.yaml, which probably does the same thing.
hooks:
pre-import-db:

exec: mysql -uroot -proot -e "ALTER INSTANCE DISABLE INNODB REDO_LOG;"
post-import-db:
exec: mysql -uroot -proot -e "ALTER INSTANCE ENABLE INNODB REDO_LOG;"

On Fri, Nov 15, 2024 at 9:07 AM Randy Fay @.***> wrote:

I think the binlogs are mostly intended for replication and crash
recovery. I'm pretty sure we could turn them off or limit them more. A PR
is welcome. Go ahead and test with your own system, and when happy, suggest
an improved approach.
—
Reply to this email directly, view it on GitHub
https://github.com/ddev/ddev/issues/6244#issuecomment-2478937528, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJPZNL7LICUHPSLZNCCAPD2AX5ZDAVCNFSM6AAAAABIIDURROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINZYHEZTONJSHA
.
You are receiving this because you were mentioned.Message ID:
@.***>

... which does the same thing, but only during the import.
On Fri, Nov 15, 2024 at 12:36 PM Doug Green
@.***> wrote:

I use this in my config.yaml, which probably does the same thing.
hooks:
pre-import-db:

exec: mysql -uroot -proot -e "ALTER INSTANCE DISABLE INNODB REDO_LOG;"
post-import-db:
exec: mysql -uroot -proot -e "ALTER INSTANCE ENABLE INNODB REDO_LOG;"

On Fri, Nov 15, 2024 at 9:07 AM Randy Fay @.***> wrote:

I think the binlogs are mostly intended for replication and crash recovery. I'm pretty sure we could turn them off or limit them more. A PR is welcome. Go ahead and test with your own system, and when happy, suggest an improved approach.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.***>

I use this in my config.yaml, which probably does the same thing.

I tried to format the reply into a readable format, but without success (Sent via email doesn't allow formatting?):
Copying it here:
hooks:
    pre-import-db:
        - exec: mysql -uroot -proot -e "ALTER INSTANCE DISABLE INNODB REDO_LOG;"
    post-import-db:
        - exec: mysql -uroot -proot -e "ALTER INSTANCE ENABLE INNODB REDO_LOG;"


Replace feature

Needed a way to leave a message up until another message was requested. Added a config replace property to achieve this. A title option was also added to the messages.

Thanks for the PR! Tested it out locally and it looks good.


Fix example for configurable data stores

I was trying to set up an ImmortalDB instance with different storages and got this error:
Uncaught TypeError: Class constructor ImmortalStorage cannot be invoked without 'new'

I think the example should be updated :)

Awesome catch. Merged.
Thank you, Mikhail. Don't hesitate to let me know if there's anything else I can do for you.


avoid marketplace action

Replaced the first marketplace doxygen action with simple commands.
The second marketplace action is still there.
Let's see if it works.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 96.853%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
4894

Relevant Lines:
5053

💛  - Coveralls

Tested the workflow and it ran successfully. Yay!


Adding detail to the effects of adding a folder to "protected folders" in ransomware protections settings

The information imparted in the page doesn't explicitly indicate whether or not adding a folder also includes sub-folders or not. This small detail has important downstream consequences depending upon the answer.
Please add whether or not sub-folders are included in protection when you add a new folder to the list.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 14fb0d02-ceff-61d8-3041-13df9513769a
Version Independent ID: 7c98a316-de58-520a-a6ad-06ddaf0d71b3
Content: Protect important folders from ransomware from encrypting your files with controlled folder access
Content Source: microsoft-365/security/defender-endpoint/controlled-folders.md
Service: microsoft-365-security
Sub-service: mde
GitHub Login: @denisebmsft
Microsoft Alias: deniseb

@Reynolds-Beta Thank you for your feedback. We have made changes to the document via PR#https://github.com/MicrosoftDocs/microsoft-365-docs/pull/10611
Once the author approves, the changes will be live.
Thanks
Sri

@msbemba Thank you very much for the contribution and updating the documentation with PR. @Reynolds-Beta Hope this update is helpful for you. Thanks for taking out some time to open the issue. Appreciate and encourage you to do the same in future also.


[Question] Is notification received when app is closed

Is it possible to receive notification with this lib even when app is closed, not in background mode?
Thank you!

yes, it's possible


[Bug]: Headless Browser sizing is incorrect (in Linux)

Minimal, reproducible example
import Puppeteer from "puppeteer";

const browser = await Puppeteer.launch({
  // Puppeteer default viewport is 800x600. By setting to null, the viewport will be full size of the browser window.
  defaultViewport: null,
  headless: true,
  args: ["--window-size=1280,1024", "--headless=new"],
});

const page = await browser.newPage();

// Navigate the page to a URL.
await page.goto("https://whatismyviewport.com/");

await page.screenshot({
  path: "what-is-my-viewport.png",
});

await browser.close();

Error string
no error
Bug behavior

[x] Consistent
[ ] Flaky
[ ] PDF

Background
Running the example script under Mac and Linux gives different results
Expectation
Running chrome headless, setting the window size should mean that both the browser window / viewport & screen dimensions will be set to the same provided value
Reality
The browser window is sized correctly, however the screen size itself is not.
This also means that window.screen.width & window.screen.height are also incorrect.
The issue is not present on Mac, but IS present on Linux.

Mac (correct screen size)
Linux (incorrect screen size)

Puppeteer configuration file (if used)
No response
Puppeteer version
22.12.0, likely others
Node version
Any
Package manager
npm
Package manager version
Latest
Operating system
Linux

This issue appears to have been present in #11457.
I suspect this should be a crbug for Chrome, but verification from Puppeteer team would be appreciated.

Worth noting that Google's PageSpeed Insights service (also using Puppeteer & Chrome under Linux) also displays 800x600 for whatismyviewport.com
cc @paulirish

Running --headless=old yields a screen size the same as the window/viewport size. This appears to be a Chrome bug.
Refs:

https://issues.chromium.org/issues/40256833

Adding Chrome argument --ozone-override-screen-size=1280,1024 appears to correct the issue under Linux:
const browser = await Puppeteer.launch({
  // Puppeteer default viewport is 800x600. By setting to null, the viewport will be full size of the browser window.
  defaultViewport: null,
  headless: true,
  args: [
    "--window-size=1280,1024",
    "--ozone-override-screen-size=1280,1024",
    "--headless=new",
  ],
});

After discovering ozone-override-screen-size on Peter Beverloo's Chrome switches page, I found #10243, which also provides an expected browser size.
@ Puppeteer maintainers - ozone-override-screen-size seems like something that should be documented in Puppeteer docs, as it's a pretty unexpected argument to set.

We do not document --window-size either and, in general, the args are up to the user to set and understand what they do. Currently, Puppeteer only supports the viewport emulation via the defaultViewport or page.setViewport(). Where would you expect to find the documentation for the Chrome args like --window-size and --ozone-override-screen-size? Perhaps we could link to https://github.com/GoogleChrome/chrome-launcher/blob/main/docs/chrome-flags-for-tools.md#chrome-flags-for-tooling but ozone-override-screen-size is not documented there as well.

@OrKoN It makes sense that Puppeteer doesn't want to be responsible for documenting Chrome args.
However, given it's so difficult (requiring several flags and or options) to correctly size a window, it feels like a worthy inclusion to help people in this scenario. During my searches I saw numerous examples of issues with window/viewport sizing dating back years and covering numerous forums. It seems clear that people have issues in this area!
Perhaps a good step would be to add a section to Chrome flags for tooling that generally covers browser window sizing, viewport sizing, fullscreen and ozone?


[Android] JS objects not supported as metadata, JSON metadata is incorrectly escaped

Setting a metadata param to a stringified JS object results in an incorrectly-escaped link.
On iOS, metadata params can be directly set to JS object and can be retrieved with no issues.
However, objects are not directly supported on android and must be stringified
(there is no case for objects here and a JS object will receive a value of "unsupported type")
Originally reported in Cordova SDK: https://github.com/BranchMetrics/cordova-ionic-phonegap-branch-deep-linking/issues/133#issuecomment-327849377
code:

    const user = {
    name: 'computerjazz'
}

    let branchUniversalObject = await branch.createBranchUniversalObject(
        'content/12345', // canonical identifier
        {
          contentDescription: 'Testing!',
          metadata: {
            testParam: JSON.stringify(user),
          }
        }
      )

    let linkProperties = {
      channel: 'internal',
    }

    let controlParams = {
      '$desktop_url': 'http://www.example.com',
      '$ios_url': 'example://',
      '$android_url': 'example://',
    }
    let { url } = await branchUniversalObject.generateShortUrl(linkProperties, controlParams)
    return url

result:

The short answer here is that only strings are supported for metadata values, and the fact that other types work on iOS is not entirely intentional. That will stop working in the near future, with some planned changes in the Branch system and the iOS SDK.
What you're doing for Android, with JSON.stringify, is correct. I can see that something is being serialized in an odd way though. The string returned by the SDK should not be escaped. I'll fix that and release a patch.

@jdee I ran into this problem as well, also tried JSON.stringify and discovered that weird serializing issue. So I dove into Android source to try and come up with a fix. How quickly do you think you'll have a fix.. should I continue to search one myself? I'm looking to supply an update to my app asap and this is the outstanding issue. Thanks

@calebfoster I'm using this workaround and it's doing the trick for now: https://github.com/BranchMetrics/cordova-ionic-phonegap-branch-deep-linking/issues/133#issuecomment-215752681

@computerjazz what exactly are you doing to fix it?

So far I can only tell you that this happens during serialization on the Android side. You can pass stringified JSON when creating links on iOS and open those links just fine on Android. I think I verified that everything was coming through just fine from RN (the ReadableMap object that is passed to Java). So it's likely a question of the Android SDK doing something to the input. The workaround recommended by @computerjazz looks like the right thing to do. I'll see if I can get that released quickly.

@computerjazz as we embed the data portion of the link payload inside another json, it's nested and therefore double encoded.
Do you think either of the following  can help you to fix the issue

Using JSonObject
Convert the escaped string to  a JsonObject and covering to back to string
String orginalvalue = new JSONObject("{\"key\":\"value\"}").toString();

Using an apache lib
From your build.gradle, include compile 'org.apache.commons:commons-lang3:3.4' -- in activity just call currentStringWithSlashes = StringEscapeUtils.unescapeJson(currentStringWithSlashes) and you'll have the String as you sent it off.

@calebfoster
branch.subscribe(bundle => {
    if (bundle && bundle.params && !bundle.error) {
      // grab deep link data and route appropriately.
      if (bundle.params.yourCustomParam) {
        // Fix branch issue on android that double-escapes strings
        if (typeof bundle.params.yourCustomParam == 'string') {
          const __unescaped = bundle.params.yourCustomParam.replace(/\\/g, "");
          bundle.params.yourCustomParam = JSON.parse(__unescaped);
        }
       // now you can do stuff with bundle.params.yourCustomParam
      }
    }
  })

We haven't forgotten about this, but a workaround in the RN SDK is not that easy. As @computerjazz showed in the original post, the problem seems to happen at link creation on Android. Links like that will not return the correct metadata in any Branch SDK. If we work around this in this SDK, it also has to be done in the iOS SDK, the web SDK, Cordova, Titanium, Unity, etc. Not to mention that metadata keys have to be identified separately. Just replacing every backslash in every field is probably not safe. It would have to be something along these lines:
for (key in params) {
  const param = params[key]
  if (/^~/.test(param) || /^\+/.test(param) || /^\$/.test(param)) continue

  const unescaped = param.replace(/\\/g, '')

  try {
    // only do this if the unescaped param is parseable as JSON
    JSON.parse(unescaped)
    params[key] = unescaped
  }
  catch (error) {
    // not JSON, do nothing
  }
}

I believe this is an issue in the Android SDK, and the right answer is probably to patch the Android SDK and include the updated SDK here. This is under review with the Android team and will happen quite soon.
For the moment, if you can live with working around this in app code as indicated above, that's probably best. It will improve in the next release. Apologies for the delay.

@computerjazz  Hello, Hope it is clear now. Can we close the issue ?


[Feature Request] Search Google, return first result

Pretty much title. Would be nice to have [p]g or [p]google to search Google and return the first result as a link.

nah, at least not for now

Ok, as a less good option, how about just this, @Kwoth?
User: ~g some words here
Bot: https://www.google.com/search?q=some+words+here

Ok i'll do it

Since i actually liked ~g i will add another command with your original request - to post the actual first result link


Fix typo in comment (UTC -> UTS)

@bors r+

:pushpin: Commit de011ea has been approved by BurntSushi

Nice catch, thanks!

:hourglass: Testing commit de011ea with merge 4bbff94...

:sunny: Test successful - status-appveyor, status-travis
Approved by: BurntSushi
Pushing 4bbff943a457841dbb0ec7579747d0985612efbe to master...


Bug: NBNS packets are malformed

RE: https://github.com/secdev/scapy/issues/4426

This was fixed in ea86cb7afdd94de5c5454d82983badfdf3049bf0
See the above linked issue for details.


[New translation] Chinese

我翻译了这个计时器，添加了简体中文。
I translated this timer, added Simplified Chinese.
能够把其加入到翻译列表吗？
Can add it to the translation list?
谢谢SCP Wiki团队为翻译人员提供了良好的环境。
Thank you to the SCP Wiki team for providing a great environment for translators.

Hello! Thank you for the submission. I asked a question in the comments. I hope I can help provide a good experience for translators.
Also I am curious, are you a staff member for the SCP-CN (Simplified Chinese) branch?

I'm not a staff member, but I want to participate in these affairs, OK?
我不是工作人员，但是我想参与这些事务，可以吗？

I see, thank you.

Then I'll go and revise it first
那么我先去修改啦。

Please continue to explain any other questions. In a word, thank you for your guidance!
还有什么问题请继续说明，总而言之，谢谢你的指导！

And then, how is it now?

I see the advanced line is still English?

It's alright now

Of course, I'll take a look at how it runs... Earlier I was wondering why there is no Chinese version of this timer... Now there is, thank you!

Thank you for the translations!

Of course, I would also like to thank you for your guidance!


popover-wrapper a styles

Is there any particular reason for having this style applied for all child elements in popover-wrapper?
.popover-wrapper a {
    display: inline !important;
}

I have a custom template with a elements and maintaining styles for these because of added inline styles to override becomes a concern.
Can we just have
.popover-wrapper > a {
    display: inline !important;
}

instead?

There was no specific reason why the css was setup like that.  I can add a fix for that in my next pull request.

@ckosloski that would be great thanks.

Is there a reason in the first place why that has to be !important? I understand the need for it to be inline for functionality reasons, but right now I'm having a hell of a time juggling display values because of this.
I'd appreciate it if you removed the !important and added a note in the documentation that this needs to be inline.

You can always overwrite the style in your style sheet.
PR's are welcome :)

Perhaps I am missing something. In order for me to override or overwrite this locally don't I either have to:
a) Edit the css file from your library locally
b) Use more !important tags in my own css files
Is there a third way?

Would if work if you defined
.popover-wrapper > a {
    display: inline;
}

In your own css file that is included after the xeditable file so that it will override it?


Remove sidekick button is broken

Version: Master 0506
**Steps to Reproduce: **

Create a service and Add a sidekick

Results: The remove sidekick button is broken

Expected:  Should not be broken

i18n


Negative Power Bug

The Chrome Extension does not show negative power readings correctly for current power and for the demand hour history nothing is displayed.

Hey there, Thanks for the issue. I'll fix it and post an update onto the chrome extension.


Missing tables with "fast translation"

When a stix-shifter connector's "to_stix_map" doesn't use an object name in a mapping, those objects could be silently dropped by async translate/ingest.  This happens e.g. with qradar's software:name mapping.
You can tell by inspecting the DB (in this case PostgreSQL but also happens with sqlite3):
  otype   | path | shortname | dtype
----------+------+-----------+-------
 software | name | name      | str
(1 row)
# \d
              List of relations
 Schema  |      Name       | Type  |  Owner
---------+-----------------+-------+----------
 flat-id | __columns       | table | postgres
 flat-id | __contains      | table | postgres
 flat-id | __metadata      | table | postgres
 flat-id | __queries       | table | postgres
 flat-id | __symtable      | table | postgres
 flat-id | artifact        | table | postgres
 flat-id | domain-name     | table | postgres
 flat-id | email-message   | table | postgres
 flat-id | file            | table | postgres
 flat-id | identity        | table | postgres
 flat-id | ipv4-addr       | table | postgres
 flat-id | ipv6-addr       | table | postgres
 flat-id | network-traffic | table | postgres
 flat-id | observed-data   | table | postgres
 flat-id | url             | table | postgres
 flat-id | x-ibm-finding   | table | postgres
 flat-id | x-oca-event     | table | postgres
 flat-id | x-qradar        | table | postgres
(18 rows)

From this example you can see that firepit recorded a software:name column (meaning it was in the native qradar data passed into the translate function) which means there should be a software table with id and name columns, but listing the tables in the database shows that software is missing.

Wrong repo!


Let-Else is not formated correctly after else

fn main() {
    let position:      Option<usize> = None;
    let mut normal: Option<usize> =  None;
    let mut tex_coords: Option<usize>    = None;

    let (Some( position), Some(    normal), Some(tex_coords  )) = (position, normal, tex_coords) else {
        return;
 };
 }

Does not format the let-else at all, resulting in:
fn main() {
    let position: Option<usize> = None;
    let mut normal: Option<usize> = None;
    let mut tex_coords: Option<usize> = None;

    let (Some( position), Some(    normal), Some(tex_coords  )) = (position, normal, tex_coords) else {
        return;
 };

}

# How to reproduce
$ rustfmt .\src\main.rs

Version:
$ rustfmt --version
rustfmt 1.5.1-stable (897e3755 2022-11-02)

Thanks for reaching out!
rustfmt doesn't support formatting let-else as the rules for how they should be formatted were not decided until about 2 weeks ago. You can check out https://github.com/rust-lang/fmt-rfcs/pull/165#issuecomment-1371258783 for more details. support for let-else formatting will be added in a future release.
To be notified when let-else support is added you can follow tracking issue #4914

Alright, thanks :)


Set ConnectionMonitor-thread to daemon to avoid blocking shutdown.

We cannot rely on a shutdown-signal and there is no support for explicit stop(). Making into daemon-thread

Manually tested this solves the problem with app-shutdown

Slightly cold feet on removing the shutdown hook. It might still be useful to avoid this thread running after the http-client has been shutdown in normal shutdown scenarios. Verified with Even.

So it is the shutdown hook itself that may never run? Even though it has been added to the runtime? The invocation to ConnectionMonitor.shutdown() never takes place? Or is it invoked, but not effective? Is the JVM shut down in a way circumvent the shutdown hooks?

It appears HttpClientBuilder now has built-in support for idle connection evictor, using that instead!

So it is the shutdown hook itself that may never run? Even though it has been added to the runtime? The invocation to ConnectionMonitor.shutdown() never takes place? Or is it invoked, but not effective? Is the JVM shut down in a way circumventing the shutdown hooks?

Shutdown hooks will not run unless a normal shutdown signal is received or we explicitly call System.exit(). May happen when the main method for starting applications throws an exception before successful start of the app

Maybe we should upgrade to v5.1.3 aswell 🤔

Makes sense. If there are any non-daemon threads running when the main thread ends (normally or by throwing an exception), and there has been no signal received by the JVM to shut down, the JVM should continue running. Anyway, a monitor thread should probably be a daemon thread.
But I am guessing that if the JVM is in a state where there is no non-daemon "user threads" left, i.e. the main thread has ended normally or by exception, and any forked non-daemon threads have also ended, the JVM is shutting down even though it may not have received a signal to shut down, and shutdown hooks would be run.
Again, it probably makes sense to have the monitoring thread as a daemon thread. But it may also be relevant to leave an option for an explicit shutdown of the monitoring thread, as I guess it is tied to one instance of an HTTP client? It may not be relevant for our use to instantiate and shutdown HTTP clients which are not tied to the startup and shutdown of the application, but it may be a use case, so it may be relevant to also tie a controlled shutdown of the monitor thread to shutdown (i.e. .close()) of a HTTP client.

haha, great minds!

But I am guessing that if the JVM is in a state where there is no non-daemon "user threads" left, i.e. the main thread has ended normally or by exception, and any forked non-daemon threads have also ended, the JVM is shutting down even though it may not have received a signal to shut down, and shutdown hooks would be run.

Yes! :)

Again, it probably makes sense to have the monitoring thread as a daemon thread. But it may also be relevant to leave an option for an explicit shutdown of the monitoring thread, as I guess it is tied to one instance of an HTTP client? It may not be relevant for our use to instantiate and shutdown HTTP clients which are not tied to the startup and shutdown of the application, but it may be a use case, so it may be relevant to also tie a controlled shutdown of the monitor thread to shutdown (i.e. .close()) of a HTTP client.

Yeah, agreed

Verified the new build solves the problem we experienced.

Releaser i tag 4.1.0


MAINT: Printing Error Msg to STDERR and Removing iterate.dat

Printing Error Msg to STDERR in place of STDOUT
Removed the iterate.dat from LBFGS
See #5037

scipy/optimize/lbfgsb.py should be updated as well.

@larsmans Sorry wasn't in town. Would make the required changes asap.

@larsmans Not able to grasp the changes to be made in scipy/optimize/lbfgsb.py. Kindly guide.

The docstring for fmin_l_bfgs_b still describes the old logging behavior.

Thanks @larsmans.
Made the change in documentation of fmin_l_bfgs_b to account for new logging behavior.

@larsmans @ev-br
Does the changes made in PR meet the requirements? Kindly let me know if any additional changes are required.

LGTM, but I'll let someone who can actually read Fortran do the merging :)

I read Fortran, and the code seems OK. But the docstring is not very consistent: it only mentions iprint < 0 and iprint==0, while the code recognizes positive values of iprint. To check, try with
In 10]: from scipy.optimize import fmin_l_bfgs_b as lbfgs

In [11]: def f(x): return (x - 2.)**2, 2.*(x - 2.)

In [12]: lbfgs(f, 1., iprint=101)

and change iprint to 1 or 99. I'm not sure many people use this (partly because I don't think many people need this or are aware of this); still I'd rather avoid giving misleading information. Ideally, the Fortran level description of the iprint variable (e.g., https://github.com/scipy/scipy/pull/5340/files#diff-b68fd6e59113bfe219e7099876e38b34L130) makes its way to the python level docstring.

Have updated python docs to reflect fortran code. Is it okay, @ev-br?

Right. I think it's because | symbol has a special meaning in reST, so it needs to be shielded by double backicks, like so |stuff|.
You can check the doc build locally by `python runtests.py --refguide-check``

Wrapping |proj g| into double backicks solved the reST issue. Thanks @ev-br :-)
Let me know if any additional changes are required.

Great! Merging, thank you @sumitbinnani and congratulations with what I believe is your first SciPy PR. Keep them coming!

Thanks @sumitbinnani!

Happy to make my first contribution to SciPy :-)
Thanks @ev-br and @larsmans :-)


Event markers per channel

Describe the problem
Events and annotations can only be specified for all channels.
Describe your solution
A way to assign an event to a specific channel, along with plotting and analysis routines that handle that would be extremely helpful for removing specific segments of specific bad channels, or for any decoding/classification of channel-wise EEG features. Maybe an extra column in the events array which specifies the channel would be a good way to input this information.
Additional context
I understand this is a large request, but id like to start a discussion here and see if anyone else is interested in this feature.

@bdyetton this feature has been requested in the past by Boris Burle in Marseille. He had goor arguments. I think it's doable in the near future with the Annotations object. Can I ask what is your use case?

We are interested in sleep features such as spindles, k-complexes, and slow oscillations. These short (~0.5-2s) generally occur in each channel independently (although sometimes they appear in many at the same time).
It would be great to:
a) Mark these events, by hand, in the plotting tool (much like you can highlight segments currently)
b) Be able to add the output of sleep feature detection algorithms (start, duration, channel) to a mne.io.RawArray so we can then plot and view them
c) Be able to save these annotation locations back to disk with the EEG data (as .fif or preferable, as .edf).
How these single channel annotations would work with the mne.Epochs object is not something I've thought about as we don't use the epochs object that much.
Marking per-channel artifacts would be nice also, as sometimes an electrode can go bad halfway through a night of sleep, but we don't want to throw away the whole channel (by marking electrode as bad), or throw away half the night (by marking a segment of all channels as bad).
p.s. @agramfort Great work on that sleep staging algorithm, we are hoping to give it a go soon ;)

O wow! I had this same idea and was tempted to try it out this year.
Glad it worked out! Will the code be made available after the conference? I'd love to try it out.
You might be interested in this: https://osf.io/8bma7/ if you want higher quality spindles for the MASS dataset.

cool ! I did not know about these data.
ping @slasnista
Yes we should make the code available at some point. We're about to send the journal paper.

Annotations currently has onset, duration, description. I suggest we add ch_names which is a tuple-of-tuples of channel names that the annotations apply to (like our epochs.drop_log). If the tuple-of-tuples is all empty, we don't write it to disk (I/O the same). If it's not empty, we write it to disk, probably however we do with drop_log. I think this gets people what they want in a backward-compatible way. Then we "just" have to figure out how to draw this in raw.plot, which I think means drawing either a  single fill_between like we do now when it applies to all channels, or M fill_betweens for the M visible channels out of the N channels the annotation currently applies to.

yes changing annotations is not too hard but it means the plotting code
needs to deal with it.
I feel it's much more painful... you need to be able to annotate a subset
of channels that may not be
below eachother

Having worked on the plotting code I actually don't think it's going to be very difficult actually. So if that's all that's stopping us I could give it a shot

Give it a try

Visual channel-specific annotations is one thing, but are you planning to support creating them as well?

I am less worried about the plotting and more worried about downstream processing. See recent discussion here: https://mne.discourse.group/t/annotations-per-channels/2693/4

I say we start by treating them like any other annotation -- i.e., ignore the ch_names attribute -- and things will proceed as normal, so it's just an aesthetic and bookkeeping change for people to make if they want to. For example, if some marks manually a BAD_flux_jump just for MEG0111, that entire segment would still be ignored in epoching, and treated like an acquisition skip in filtering, etc. just as it is now depending on the skip_by_annotation parameter.
However, in future PRs things like this become possible:

We could modify find_bad_channels_maxwell and annotate_flat to mark badness channel-by channel over time (easy)
Modify maxwell_filter to reconstruct BAD_-marked MEG channels just during their bad times (hard)
Update https://github.com/mne-tools/mne-python/pull/1869 repair the channels on a per-epoch basis (easy to get working, medium to make computationally efficient)

In other words, we implement the marking and storage, then make things annotations-channels-aware gradually as we can. This is analogous to how we implemented annotations in the first place, then gradually made downstream functions and methods annotation-aware as we found it beneficial or necessary to do so. Most places we have a skip_by_annotation or similar argument came into being this way, and it seemed to work as both a creator of some of those PRs and as a consumer.

Sounds good!

Support creating channel-specific annotations in raw.plot (medium to hard)

Are any of these tasks suitable for the upcoming sprint @larsoner ?  If so let's wait until then to implement.

The first PR, namely to add ch_names to Annotations, is not new-contributor friendly, and is a prerequisite for everything else enumerated above. So I think I should make first make a PR to add ch_names as tuple of tuples of ch_names as above. Then someone (probably @drammock) makes the plotting code handle it properly using fill_betweens or rectangle patches or whatever is necessary in a follow-up PR.
Then the rest of the items can be saved for the sprint, but really the preprocessing enhancements in (1) above are probably the only ones suitable for new contributors.

At the risk of sounding repetitive, I wouldn't implement showing channel-specific annotations in our plotting code without the option to create them. Or do you think people will not want to create/edit them (as opposed to just view) if they happen to have channel-specific annotations?

At the risk of sounding repetitive, I wouldn't implement showing channel-specific annotations in our plotting code without the option to create them

Agreed.  Displaying them is the easy part.  I've been scratching my head a bit over how to do the UI/UX for creating them (I have some ideas, but there's no obvious "right" way IMO)

At the risk of sounding repetitive, I wouldn't implement showing channel-specific annotations in our plotting code without the option to create them

Agreed

I disagree with this. We're about to have a PR that implements the ability to make these manually via Annotations(..., ch_names=...), we might as well show them / this information properly.
Moreover, the PRs to make them via find_* or mark_* functions and the one(s) to make them interactively in the GUI would benefit from actually showing them properly for testing/iteration purposes, and also be easier to review.

I can see the benefits you mention @larsoner.  In that case I think we need a (temporary) clear signal somewhere (docstring?  Annotation UI window?) making it clear that per-channel annotations cannot (yet) be created through the raw.plot() GUI.

Indeed I would just add a note somewhere that the functionality has not been implemented / annotations created interactively apply to all channels, probably in small text at the bottom of the interactive annotations window (with the buttons and such)


memory allocation of<PHONE_NUMBER> bytes failed (that's 2GB!)

I tried running it on a directory posted on reddit yesterday:
$ suckit https://icsarchive.org/icsarchive-org/paperback/cookbooks/

memory allocation of<PHONE_NUMBER> bytes failed
zsh: abort      suckit https://icsarchive.org/icsarchive-org/paperback/cookbooks/

I don't understand how it would ever come to try to allocate 2GB of memory. None of the files is that big (and even if they were, they shouldn't be buffered to memory).
I left it running for sometime so it didn't happen until quite late in the game when I was afk. I didn't run with RUST_BACKTRACE so don't have more details at this time, but I can try fetching again to see if I can replicate

Ah actually I do have one other bit of information, because my .zshrc outputs timing for any command over 2 seconds:
25:52.17s total, 130.18s user, 196.37s system, 21% cpu

I can see from re-running with --versbose that it ascended to a parent directory so I suppose it may have encountered a larger file or has some data structure that is growing with each link and hit some upper bound.

Thanks a lot for the report. We're investigating it

Re-running reproduced it at the ~28 minute mark. No backtrace shown even with RUST_BACKTRACE=1 (haven't tried RUST_BACKTRACE=full), but I haven't checked if cargo install strips debug symbols or something. Hopefully you're able to reproduce.

cargo install does strip debug symbols since it's building in release mode. You can compile the project with cargo build if you want them, or simply run cargo run -- <your suckit arguments>. Thanks again, I'll let you know how we're progressing.

I had the exact same problem down to the byte, but I'm not working with large files.
memory allocation of<PHONE_NUMBER> bytes failed
Aborted (core dumped)


Implement a ServiceMock that adheres to the Service Upgrade specifica…

…tion.

Nice, dense implementation especially considering that you included your own routing ;)
I am mostly concerned about implementing the correct protocol.

It would be cool if the operations requiring a block ID, would fail if the block does not exist. This could help the UI guys to find errors.

I only reviewed the request processing. Since you squashed all commits, it was difficult to see what was changed. What else did you change? Can you paste a diff?

@kolloch
-added a json parser for the BlockPosition and handle exceptions specifically
-interrupt now takes all done and inProgress
-I did not touch the implementation for force_complete. Will talk to Gabriel about this. But this should not prevent us from merging this PR.

I still think that it would be cool if you could indicate an error if the block is unknown. It would take you 3min and potentially catch bugs. Can you do that, please?
Then, let's merge it.

Squash and actually target the feature branch (not master).

@kolloch the git branch was already rebased on the feature branch - but  obviously created the PR on master. Closed and reopened here: #3625


HIVE-24311: Rowcontainer should reset readBlocks when we clear rows to prevent OOM.

What changes were proposed in this pull request?

Hi, We found that Rowcontainer.clearRows() only reset some index, such as: addCursor, itrCursor, etc, without resetting read blocks.
As we all know, currentReadBlock and currentWriteBlock is the main memory usage of a Rowcontainer and might be very large, depending on the data pattern.
currentReadBlock and currentWriteBlock won't be the same object after rowcontainer flushed data to disk.
Resetting currentReadBlock and currentWriteBlock while clearing rows will prevent OOM.
Therefore, I submit a patch to reset read blocks for Rowcontainer, just like PTFRowcontainer does.
Why are the changes needed?

Resetting currentReadBlock and currentWriteBlock while clearing rows will prevent OOM.
Does this PR introduce any user-facing change?

No
How was this patch tested?

Unit tests .

Failed Unit tests are not related with this patch. Anybody help to review the code?  Thanks!

@vineetgarg02  Could you please review this patch?  Thank you very much!


Nested virtualization

Describe the solution you'd like
macOS Sequoia added support for nested virtualization using the Virtualization.framework. It would be a nice to be able to run VMs inside of the a VM. In particular this would help when testing Linux based VMMs
Additional context
UTM added support for this in their latest beta release https://github.com/utmapp/UTM/releases/tag/v4.6.0

This is planned.


unable to run go build

Describe the bug
A clear and concise description of what the bug is.
I am unable to run go build command it throws error, I have checked go env in my systems  , I moved scorecard repo to $GOPATH/src folder but still no luck.
I am not good in go langague, so it would be better if you update proper installation method in README  about setting up or checking go env before run go build.
My error message as below : -
go build
../go/src/github.com/bradleyfalzon/ghinstallation/appsTransport.go:11:2: cannot find package "github.com/dgrijalva/jwt-go/v4" in any of:
/usr/src/github.com/dgrijalva/jwt-go/v4 (from $GOROOT)
/home/ubuntu/go/src/github.com/dgrijalva/jwt-go/v4 (from $GOPATH)
../go/src/github.com/ossf/scorecard/cmd/root.go:30:2: cannot find package "github.com/google/go-github/v32/github" in any of:
/usr/src/github.com/google/go-github/v32/github (from $GOROOT)
/home/ubuntu/go/src/github.com/google/go-github/v32/github (from $GOPATH)
../go/src/github.com/bradleyfalzon/ghinstallation/transport.go:14:2: cannot find package "github.com/google/go-github/v35/github" in any of:
/usr/src/github.com/google/go-github/v35/github (from $GOROOT)
/home/ubuntu/go/src/github.com/google/go-github/v35/github (from $GOPATH)
ubuntu@jumpserver-nik ~/scorecard (main) $ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ubuntu/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/ubuntu/go"
GORACE=""
GOROOT="/usr"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/gcc/x86_64-linux-gnu/8"
GCCGO="/usr/bin/x86_64-linux-gnu-gccgo-8"
CC="x86_64-linux-gnu-gcc-8"
CXX="x86_64-linux-gnu-g++-8"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build883208298=/tmp/go-build -gno-record-gcc-switches -funwind-tables"
Reproduction steps
Steps to reproduce the behavior:

installed go, set gopath and go root
download scorecard repo
run - go build

Expected behavior
A clear and concise description of what you expected to happen.
it should run as given in README file
Additional context
Add any other context about the problem here.
It would be great if you include more detailed installation guilde in README

Can you run the make build-scorecard?  The https://github.com/ossf/scorecard/blob/main/Makefile has all the make targets to build and run.

Closing this issue due to inactivity. Please re-open if make build-scorecard didn't fix your issue.


Correctly expand absolute paths for link_lib

Otherwise absolute paths will be appended to the project path.

heh, we encountered this bug at positron!