Cannot deploy on on-premise k8s

Hi all!
I'm running into issues installing the helm chart on my on-prem/local kubernetes cluster. I'll first dump all relevant manifests, and then describe the issue I'm running into.

Helm values
persistence:
  enabled: true
  existingClaim: galaxy-data
metrics:
  enabled: false
s3csi:
  deploy: false
cvmfs:
  deploy: false
setupJob:
  createDatabase: true
  ttlSecondsAfterFinished: 3600
ingress:
  enabled: true
serviceAccount:
  create: true
postgresql:
  enabled: true
    #galaxyDatabasePassword: "potato"
    #postgresqlPassword: "potato"
    #postgresqlPostgresPassword: "potato"
  persistence:
    enabled: true
    storageClass: openebs-jiva-csi-sc
    size: 5Gi
refdata:
  enabled: false
influxdb:
  enabled: false

galaxy-data PVC definition
kind: Namespace
apiVersion: v1
metadata:
  name: galaxy
  labels:
    kubernetes.io/metadata.name: galaxy
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: galaxy-data-nfs
  namespace: galaxy
spec:
  accessModes: [ "ReadWriteOnce" ]
  storageClassName: openebs-jiva-csi-sc
  resources:
    requests:
      storage: 50Gi
---
kind: Service
apiVersion: v1
metadata:
  name: nfs-server
  namespace: galaxy
spec:
  clusterIP: <IP_ADDRESS>
  ports:
    - name: nfs
      port: 2049
    - name: mountd
      port: 20048
    - name: rpcbind
      port: 111
  selector:
    role: nfs-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-server
  namespace: galaxy
spec:
  replicas: 1
  selector:
    matchLabels:
      role: nfs-server
  template:
    metadata:
      labels:
        role: nfs-server
    spec:
      securityContext:
        fsGroup: 101
      containers:
      - name: nfs-server
        image: registry.k8s.io/volume-nfs:0.8
        ports:
          - name: nfs
            containerPort: 2049
          - name: mountd
            containerPort: 20048
          - name: rpcbind
            containerPort: 111
        securityContext:
          privileged: true
        volumeMounts:
          - mountPath: /exports
            name: mypvc
      volumes:
        - name: mypvc
          persistentVolumeClaim:
            claimName: galaxy-data-nfs
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: galaxy-data
  namespace: galaxy
spec:
  accessModes:
    - ReadWriteMany
  capacity:
    storage: 50Gi
  nfs:
    #server: nfs-server.galaxy.svc.cluster.local
    server: <IP_ADDRESS>
    path: "/"
  mountOptions:
    - nfsvers=4.2
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: galaxy-data
  namespace: galaxy
spec:
  accessModes:
    - ReadWriteMany
  volumeName: galaxy-data
  resources:
    requests:
      storage: 50Gi

I use the NFS backed PVC since my "standard" sc (jiva) doesn't support RWX.
Running helm install -n galaxy my-galaxy -f galaxy-values.yaml galaxy-helm/galaxy/ results in
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
NAME: my-galaxy
LAST DEPLOYED: Fri Oct 14 16:59:25 2022
NAMESPACE: galaxy
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  http:///galaxy

Looks reasonable to me. However, the job my-galaxy-init-db fails in container galaxy-init-postgres, related to permission denied to chown 101:101 /galaxy/server/database. Easy enough to fix: wipe the namespace, recreate the NFS share, create that folder with correct ownership in the NFS share, and reinstall the helm chart.
Next issue has me stumped though kubectl logs -n galaxy jobs/my-galaxy-init-db-hb4hl -c galaxy-init-postgres:
nc: bad address 'galaxy-my-galaxy-galaxy-postgres'
waiting for galaxy-postgres service
(repeating)

And kubectl logs -n galaxy jobs/my-galaxy-init-db-hb4hl has a python traceback resulting in sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL:  password authentication failed for user "galaxydbuser". On the NFS share, the folder server/database remains empty.
The postgres service pod is running:
NAME                                            READY   STATUS      RESTARTS   AGE
pod/galaxy-my-galaxy-galaxy-postgres-0          1/1     Running     0          17m
pod/my-galaxy-galaxy-postgres-7c897bf6c-dd7tb   1/1     Running     0          17m
pod/my-galaxy-init-mounts-monlp-mvwsp           0/4     Completed   0          17m
pod/my-galaxy-job-0-7459f74d56-c6fmb            0/1     Init:0/1    0          17m
pod/my-galaxy-nginx-8474bd9bd-qf58r             1/1     Running     0          17m
pod/my-galaxy-web-dfbddc76b-f6qf7               0/1     Init:0/1    0          17m
pod/my-galaxy-workflow-77fd776ccd-jqtfq         0/1     Init:0/1    0          17m
pod/nfs-server-69cd8d9cb9-tbjs6                 1/1     Running     0          20m

NAME                                              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
service/galaxy-my-galaxy-galaxy-postgres          ClusterIP   <IP_ADDRESS>    <none>        5432/TCP                     17m
service/galaxy-my-galaxy-galaxy-postgres-config   ClusterIP   None           <none>        <none>                       16m
service/galaxy-my-galaxy-galaxy-postgres-repl     ClusterIP   <IP_ADDRESS>    <none>        5432/TCP                     17m
service/my-galaxy-galaxy-postgres                 ClusterIP   <IP_ADDRESS>   <none>        8080/TCP                     17m
service/my-galaxy-nginx                           ClusterIP   <IP_ADDRESS>   <none>        8000/TCP                     17m
service/my-galaxy-uwsgi                           ClusterIP   <IP_ADDRESS>   <none>        4001/TCP                     17m
service/nfs-server                                ClusterIP   <IP_ADDRESS>      <none>        2049/TCP,20048/TCP,111/TCP   20m

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/my-galaxy-galaxy-postgres   1/1     1            1           17m
deployment.apps/my-galaxy-job-0             0/1     1            0           17m
deployment.apps/my-galaxy-nginx             1/1     1            1           17m
deployment.apps/my-galaxy-web               0/1     1            0           17m
deployment.apps/my-galaxy-workflow          0/1     1            0           17m
deployment.apps/nfs-server                  1/1     1            1           20m

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/my-galaxy-galaxy-postgres-7c897bf6c   1         1         1       17m
replicaset.apps/my-galaxy-job-0-7459f74d56            1         1         0       17m
replicaset.apps/my-galaxy-nginx-8474bd9bd             1         1         1       17m
replicaset.apps/my-galaxy-web-dfbddc76b               1         1         0       17m
replicaset.apps/my-galaxy-workflow-77fd776ccd         1         1         0       17m
replicaset.apps/nfs-server-69cd8d9cb9                 1         1         1       20m

NAME                                                READY   AGE
statefulset.apps/galaxy-my-galaxy-galaxy-postgres   1/1     17m

NAME                                  SCHEDULE    SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cronjob.batch/my-galaxy-maintenance   0 2 * * *   False     0        <none>          17m

NAME                                    COMPLETIONS   DURATION   AGE
job.batch/my-galaxy-init-db-hb4hl       0/1           17m        17m
job.batch/my-galaxy-init-mounts-monlp   1/1           5m25s      17m

NAME                                                            IMAGE                                               CLUSTER-LABEL   SERVICE-ACCOUNT   MIN-INSTANCES   AGE
operatorconfiguration.acid.zalan.do/my-galaxy-galaxy-postgres   registry.opensource.zalan.do/acid/spilo-13:2.1-p1   cluster-name    postgres-pod      -1              17m

NAME                                                        TEAM     VERSION   PODS   VOLUME   CPU-REQUEST   MEMORY-REQUEST   AGE   STATUS
postgresql.acid.zalan.do/galaxy-my-galaxy-galaxy-postgres   galaxy   13        1      5Gi                                     17m   SyncFailed

kubectl logs -n galaxy pods/galaxy-my-galaxy-galaxy-postgres-0
2022-10-14 15:00:01,209 - bootstrapping - INFO - Figuring out my environment (Google? AWS? Openstack? Local?)
2022-10-14 15:00:03,219 - bootstrapping - INFO - Could not connect to <IP_ADDRESS>, assuming local Docker setup
2022-10-14 15:00:03,222 - bootstrapping - INFO - No meta-data available for this provider
2022-10-14 15:00:03,223 - bootstrapping - INFO - Looks like your running local
2022-10-14 15:00:03,320 - bootstrapping - INFO - Configuring pgbouncer
2022-10-14 15:00:03,320 - bootstrapping - INFO - No PGBOUNCER_CONFIGURATION was specified, skipping
2022-10-14 15:00:03,321 - bootstrapping - INFO - Configuring log
2022-10-14 15:00:03,321 - bootstrapping - INFO - Configuring pgqd
2022-10-14 15:00:03,322 - bootstrapping - INFO - Configuring bootstrap
2022-10-14 15:00:03,322 - bootstrapping - INFO - Configuring pam-oauth2
2022-10-14 15:00:03,323 - bootstrapping - INFO - Writing to file /etc/pam.d/postgresql
2022-10-14 15:00:03,323 - bootstrapping - INFO - Configuring crontab
2022-10-14 15:00:03,324 - bootstrapping - INFO - Skipping creation of renice cron job due to lack of SYS_NICE capability
2022-10-14 15:00:03,325 - bootstrapping - INFO - Configuring patroni
2022-10-14 15:00:03,362 - bootstrapping - INFO - Writing to file /run/postgres.yml
2022-10-14 15:00:03,365 - bootstrapping - INFO - Configuring certificate
2022-10-14 15:00:03,365 - bootstrapping - INFO - Generating ssl certificate
2022-10-14 15:00:03,532 - bootstrapping - INFO - Configuring wal-e
2022-10-14 15:00:03,532 - bootstrapping - INFO - Configuring standby-cluster
2022-10-14 15:00:05,136 WARNING: Kubernetes RBAC doesn't allow GET access to the 'kubernetes' endpoint in the 'default' namespace. Disabling 'bypass_api_service'.
2022-10-14 15:00:05,162 INFO: No PostgreSQL configuration items changed, nothing to reload.
2022-10-14 15:00:05,167 INFO: Lock owner: None; I am galaxy-my-galaxy-galaxy-postgres-0
2022-10-14 15:00:05,349 INFO: trying to bootstrap a new cluster
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /home/postgres/pgdata/pgroot/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/13/bin/pg_ctl -D /home/postgres/pgdata/pgroot/data -l logfile start

2022-10-14 15:00:08 UTC [79]: [1-1] 63497978.4f 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-10-14 15:00:08 UTC [79]: [2-1] 63497978.4f 0     LOG:  pg_stat_kcache.linux_hz is set to 500000
2022-10-14 15:00:08,662 INFO: postmaster pid=79
/var/run/postgresql:5432 - no response
2022-10-14 15:00:08 UTC [79]: [3-1] 63497978.4f 0     LOG:  redirecting log output to logging collector process
2022-10-14 15:00:08 UTC [79]: [4-1] 63497978.4f 0     HINT:  Future log output will appear in directory "../pg_log".
/var/run/postgresql:5432 - accepting connections
/var/run/postgresql:5432 - accepting connections
2022-10-14 15:00:09,709 INFO: establishing a new patroni connection to the postgres cluster
2022-10-14 15:00:09,772 INFO: running post_bootstrap
DO
DO
DO
CREATE EXTENSION
NOTICE:  version "1.1" of extension "pg_auth_mon" is already installed
ALTER EXTENSION
GRANT
CREATE EXTENSION
NOTICE:  version "1.3" of extension "pg_cron" is already installed
ALTER EXTENSION
ALTER POLICY
REVOKE
GRANT
GRANT
CREATE FUNCTION
REVOKE
GRANT
REVOKE
GRANT
REVOKE
GRANT
GRANT
CREATE EXTENSION
DO
CREATE TABLE
GRANT
ALTER TABLE
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
RESET
SET
NOTICE:  schema "zmon_utils" does not exist, skipping
DROP SCHEMA
DO
NOTICE:  language "plpythonu" does not exist, skipping
DROP LANGUAGE
NOTICE:  function plpython_call_handler() does not exist, skipping
DROP FUNCTION
NOTICE:  function plpython_inline_handler(internal) does not exist, skipping
DROP FUNCTION
NOTICE:  function plpython_validator(oid) does not exist, skipping
DROP FUNCTION
CREATE SCHEMA
GRANT
SET
CREATE TYPE
CREATE FUNCTION
CREATE FUNCTION
GRANT
You are now connected to database "postgres" as user "postgres".
CREATE SCHEMA
GRANT
SET
CREATE FUNCTION
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
GRANT
RESET
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
NOTICE:  version "2.0" of extension "set_user" is already installed
ALTER EXTENSION
GRANT
GRANT
GRANT
CREATE SCHEMA
GRANT
GRANT
SET
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
RESET
You are now connected to database "template1" as user "postgres".
CREATE SCHEMA
GRANT
SET
CREATE FUNCTION
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
GRANT
RESET
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
NOTICE:  version "2.0" of extension "set_user" is already installed
ALTER EXTENSION
GRANT
GRANT
GRANT
CREATE SCHEMA
GRANT
GRANT
SET
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
RESET
2022-10-14 15:00:10,961 WARNING: Could not activate Linux watchdog device: "Can't open watchdog device: [Errno 2] No such file or directory: '/dev/watchdog'"
2022-10-14 15:00:11,177 INFO: initialized a new cluster
2022-10-14 15:00:11,233 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:00:11,742 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:00:11,760 INFO: Changed tcp_keepalives_idle from 0 to 900
2022-10-14 15:00:11,760 INFO: Changed tcp_keepalives_interval from 0 to 100
2022-10-14 15:00:11,767 INFO: Reloading PostgreSQL configuration.
server signaled
2022-10-14 15:00:12,888 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:00:23,334 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:00:33,385 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:00:43,384 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:00:53,329 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:01:03,387 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:01:04.675 35 LOG Starting pgqd 3.3
2022-10-14 15:01:04.676 35 LOG auto-detecting dbs ...
2022-10-14 15:00:01,209 - bootstrapping - INFO - Figuring out my environment (Google? AWS? Openstack? Local?)
2022-10-14 15:00:03,219 - bootstrapping - INFO - Could not connect to <IP_ADDRESS>, assuming local Docker setup
2022-10-14 15:00:03,222 - bootstrapping - INFO - No meta-data available for this provider
2022-10-14 15:00:03,223 - bootstrapping - INFO - Looks like your running local
2022-10-14 15:00:03,320 - bootstrapping - INFO - Configuring pgbouncer
2022-10-14 15:00:03,320 - bootstrapping - INFO - No PGBOUNCER_CONFIGURATION was specified, skipping
2022-10-14 15:00:03,321 - bootstrapping - INFO - Configuring log
2022-10-14 15:00:03,321 - bootstrapping - INFO - Configuring pgqd
2022-10-14 15:00:03,322 - bootstrapping - INFO - Configuring bootstrap
2022-10-14 15:00:03,322 - bootstrapping - INFO - Configuring pam-oauth2
2022-10-14 15:00:03,323 - bootstrapping - INFO - Writing to file /etc/pam.d/postgresql
2022-10-14 15:00:03,323 - bootstrapping - INFO - Configuring crontab
2022-10-14 15:00:03,324 - bootstrapping - INFO - Skipping creation of renice cron job due to lack of SYS_NICE capability
2022-10-14 15:00:03,325 - bootstrapping - INFO - Configuring patroni
2022-10-14 15:00:03,362 - bootstrapping - INFO - Writing to file /run/postgres.yml
2022-10-14 15:00:03,365 - bootstrapping - INFO - Configuring certificate
2022-10-14 15:00:03,365 - bootstrapping - INFO - Generating ssl certificate
2022-10-14 15:00:03,532 - bootstrapping - INFO - Configuring wal-e
2022-10-14 15:00:03,532 - bootstrapping - INFO - Configuring standby-cluster
2022-10-14 15:00:05,136 WARNING: Kubernetes RBAC doesn't allow GET access to the 'kubernetes' endpoint in the 'default' namespace. Disabling 'bypass_api_service'.
2022-10-14 15:00:05,162 INFO: No PostgreSQL configuration items changed, nothing to reload.
2022-10-14 15:00:05,167 INFO: Lock owner: None; I am galaxy-my-galaxy-galaxy-postgres-0
2022-10-14 15:00:05,349 INFO: trying to bootstrap a new cluster
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /home/postgres/pgdata/pgroot/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/13/bin/pg_ctl -D /home/postgres/pgdata/pgroot/data -l logfile start

2022-10-14 15:00:08 UTC [79]: [1-1] 63497978.4f 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-10-14 15:00:08 UTC [79]: [2-1] 63497978.4f 0     LOG:  pg_stat_kcache.linux_hz is set to 500000
2022-10-14 15:00:08,662 INFO: postmaster pid=79
/var/run/postgresql:5432 - no response
2022-10-14 15:00:08 UTC [79]: [3-1] 63497978.4f 0     LOG:  redirecting log output to logging collector process
2022-10-14 15:00:08 UTC [79]: [4-1] 63497978.4f 0     HINT:  Future log output will appear in directory "../pg_log".
/var/run/postgresql:5432 - accepting connections
/var/run/postgresql:5432 - accepting connections
2022-10-14 15:00:09,709 INFO: establishing a new patroni connection to the postgres cluster
2022-10-14 15:00:09,772 INFO: running post_bootstrap
DO
DO
DO
CREATE EXTENSION
NOTICE:  version "1.1" of extension "pg_auth_mon" is already installed
ALTER EXTENSION
GRANT
CREATE EXTENSION
NOTICE:  version "1.3" of extension "pg_cron" is already installed
ALTER EXTENSION
ALTER POLICY
REVOKE
GRANT
GRANT
CREATE FUNCTION
REVOKE
GRANT
REVOKE
GRANT
REVOKE
GRANT
GRANT
CREATE EXTENSION
DO
CREATE TABLE
GRANT
ALTER TABLE
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
RESET
SET
NOTICE:  schema "zmon_utils" does not exist, skipping
DROP SCHEMA
DO
NOTICE:  language "plpythonu" does not exist, skipping
DROP LANGUAGE
NOTICE:  function plpython_call_handler() does not exist, skipping
DROP FUNCTION
NOTICE:  function plpython_inline_handler(internal) does not exist, skipping
DROP FUNCTION
NOTICE:  function plpython_validator(oid) does not exist, skipping
DROP FUNCTION
CREATE SCHEMA
GRANT
SET
CREATE TYPE
CREATE FUNCTION
CREATE FUNCTION
GRANT
You are now connected to database "postgres" as user "postgres".
CREATE SCHEMA
GRANT
SET
CREATE FUNCTION
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
GRANT
RESET
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
NOTICE:  version "2.0" of extension "set_user" is already installed
ALTER EXTENSION
GRANT
GRANT
GRANT
CREATE SCHEMA
GRANT
GRANT
SET
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
RESET
You are now connected to database "template1" as user "postgres".
CREATE SCHEMA
GRANT
SET
CREATE FUNCTION
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
GRANT
RESET
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
NOTICE:  version "2.0" of extension "set_user" is already installed
ALTER EXTENSION
GRANT
GRANT
GRANT
CREATE SCHEMA
GRANT
GRANT
SET
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
CREATE FUNCTION
REVOKE
GRANT
GRANT
CREATE VIEW
REVOKE
GRANT
GRANT
RESET
2022-10-14 15:00:10,961 WARNING: Could not activate Linux watchdog device: "Can't open watchdog device: [Errno 2] No such file or directory: '/dev/watchdog'"
2022-10-14 15:00:11,177 INFO: initialized a new cluster
2022-10-14 15:00:11,233 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:00:11,742 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:00:11,760 INFO: Changed tcp_keepalives_idle from 0 to 900
2022-10-14 15:00:11,760 INFO: Changed tcp_keepalives_interval from 0 to 100
2022-10-14 15:00:11,767 INFO: Reloading PostgreSQL configuration.
server signaled
2022-10-14 15:00:12,888 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:00:23,334 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:00:33,385 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:00:43,384 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:00:53,329 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:01:03,387 INFO: no action. I am (galaxy-my-galaxy-galaxy-postgres-0) the leader with the lock
2022-10-14 15:01:04.675 35 LOG Starting pgqd 3.3
2022-10-14 15:01:04.676 35 LOG auto-detecting dbs ...

kubectl logs -n galaxy pods/my-galaxy-galaxy-postgres-7c897bf6c-dd7tb
time="2022-10-14T14:59:34Z" level=info msg="Spilo operator v1.7.0\n"
time="2022-10-14T14:59:34Z" level=error msg="could not create customResourceDefinition \"operatorconfigurations.acid.zalan.do\": CustomResourceDefinition.apiextensions.k8s.io \"operatorconfigurations.acid.zalan.do\" is invalid: spec.versions[0].schema.openAPIV3Schema: Required value: schemas are required" pkg=controller
time="2022-10-14T14:59:38Z" level=info msg="Parse role bindings" pkg=controller
time="2022-10-14T14:59:38Z" level=info msg="successfully parsed" pkg=controller
time="2022-10-14T14:59:38Z" level=info msg="Listening to all namespaces" pkg=controller
time="2022-10-14T14:59:38Z" level=info msg="customResourceDefinition \"postgresqls.acid.zalan.do\" is already registered and will only be updated" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="{" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ReadyWaitInterval\":<PHONE_NUMBER>," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ReadyWaitTimeout\":<PHONE_NUMBER>0," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ResyncPeriod\":<PHONE_NUMBER>000," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"RepairPeriod\":<PHONE_NUMBER>00," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableCRDValidation\": true," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ResourceCheckInterval\":<PHONE_NUMBER>," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ResourceCheckTimeout\":<PHONE_NUMBER>00," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PodLabelWaitTimeout\":<PHONE_NUMBER>00," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PodDeletionWaitTimeout\":<PHONE_NUMBER>00," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"SpiloRunAsUser\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"SpiloRunAsGroup\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"SpiloFSGroup\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PodPriorityClassName\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ClusterDomain\": \"cluster.local\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"SpiloPrivileged\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"SpiloAllowPrivilegeEscalation\": true," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"AdditionalPodCapabilities\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ClusterLabels\": {" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="      \"application\": \"spilo\"" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   }," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"InheritedLabels\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"InheritedAnnotations\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"DownscalerAnnotations\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ClusterNameLabel\": \"cluster-name\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"DeleteAnnotationDateKey\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"DeleteAnnotationNameKey\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PodRoleLabel\": \"spilo-role\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PodToleration\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"DefaultCPURequest\": \"100m\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"DefaultMemoryRequest\": \"100Mi\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"DefaultCPULimit\": \"1\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"DefaultMemoryLimit\": \"500Mi\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"MinCPULimit\": \"250m\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"MinMemoryLimit\": \"250Mi\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PodEnvironmentConfigMap\": \"/\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PodEnvironmentSecret\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"NodeReadinessLabel\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"MaxInstances\": -1," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"MinInstances\": -1," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ShmVolume\": true," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"SecretNameTemplate\": \"{username}.{cluster}.credentials.{tprkind}.{tprgroup}\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PamRoleName\": \"zalandos\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PamConfiguration\": \"https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"TeamsAPIUrl\": \"https://teams.example.com/api/\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"OAuthTokenSecretName\": \"galaxy/my-galaxy-galaxy-postgres\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"InfrastructureRolesSecretName\": \"/\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"InfrastructureRoles\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"InfrastructureRolesDefs\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"SuperUsername\": \"postgres\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ReplicationUsername\": \"standby\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ScalyrAPIKey\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ScalyrImage\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ScalyrServerURL\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ScalyrCPURequest\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ScalyrMemoryRequest\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ScalyrCPULimit\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ScalyrMemoryLimit\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"LogicalBackupSchedule\": \"30 00 * * *\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"LogicalBackupDockerImage\": \"registry.opensource.zalan.do/acid/logical-backup:v1.7.0\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"LogicalBackupProvider\": \"s3\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"LogicalBackupS3Bucket\": \"my-bucket-url\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"LogicalBackupS3Region\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"LogicalBackupS3Endpoint\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"LogicalBackupS3AccessKeyID\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"LogicalBackupS3SecretAccessKey\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"LogicalBackupS3SSE\": \"AES256\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"LogicalBackupGoogleApplicationCredentials\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"LogicalBackupJobPrefix\": \"logical-backup-\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"NumberOfInstances\": 2," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"Schema\": \"pooler\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"User\": \"pooler\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"Image\": \"registry.opensource.zalan.do/acid/pgbouncer:master-18\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"Mode\": \"transaction\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"MaxDBConnections\": 60," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ConnectionPoolerDefaultCPURequest\": \"500m\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ConnectionPoolerDefaultMemoryRequest\": \"100Mi\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ConnectionPoolerDefaultCPULimit\": \"1\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ConnectionPoolerDefaultMemoryLimit\": \"100Mi\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"WatchedNamespace\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"KubernetesUseConfigMaps\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EtcdHost\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"DockerImage\": \"registry.opensource.zalan.do/acid/spilo-13:2.1-p1\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"SidecarImages\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"SidecarContainers\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PodServiceAccountName\": \"postgres-pod\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PodServiceAccountDefinition\": \"{\\\"apiVersion\\\":\\\"v1\\\",\\\"kind\\\":\\\"ServiceAccount\\\",\\\"metadata\\\":{\\\"name\\\":\\\"postgres-pod\\\"}}\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PodServiceAccountRoleBindingDefinition\": \"{\\\"apiVersion\\\":\\\"rbac.authorization.k8s.io/v1\\\",\\\"kind\\\":\\\"RoleBinding\\\",\\\"metadata\\\":{\\\"name\\\":\\\"postgres-pod\\\"},\\\"roleRef\\\":{\\\"apiGroup\\\":\\\"rbac.authorization.k8s.io\\\",\\\"kind\\\":\\\"ClusterRole\\\",\\\"name\\\":\\\"postgres-pod\\\"},\\\"subjects\\\":[{\\\"kind\\\":\\\"ServiceAccount\\\",\\\"name\\\":\\\"postgres-pod\\\"}]}\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"MasterPodMoveTimeout\":<PHONE_NUMBER>000," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"DbHostedZone\": \"db.example.com\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"AWSRegion\": \"eu-central-1\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"WALES3Bucket\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"LogS3Bucket\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"KubeIAMRole\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"WALGSBucket\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"GCPCredentials\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"WALAZStorageAccount\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"AdditionalSecretMount\": \"\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"AdditionalSecretMountPath\": \"/meta/credentials\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableEBSGp3Migration\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableEBSGp3MigrationMaxSize\": 1000," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"DebugLogging\": true," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableDBAccess\": true," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableTeamsAPI\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableTeamSuperuser\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"TeamAdminRole\": \"admin\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"RoleDeletionSuffix\": \"_deleted\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableTeamMemberDeprecation\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableAdminRoleForUsers\": true," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnablePostgresTeamCRD\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnablePostgresTeamCRDSuperusers\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableMasterLoadBalancer\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableReplicaLoadBalancer\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"CustomServiceAnnotations\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"CustomPodAnnotations\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnablePodAntiAffinity\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PodAntiAffinityTopologyKey\": \"kubernetes.io/hostname\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"StorageResizeMode\": \"pvc\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableLoadBalancer\": null," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ExternalTrafficPolicy\": \"Cluster\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"MasterDNSNameFormat\": \"{cluster}.{team}.{hostedzone}\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ReplicaDNSNameFormat\": \"{cluster}-repl.{team}.{hostedzone}\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PDBNameFormat\": \"postgres-{cluster}-pdb\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnablePodDisruptionBudget\": true," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableInitContainers\": true," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableSidecars\": true," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"Workers\": 8," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"APIPort\": 8080," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"RingLogLines\": 100," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ClusterHistoryEntries\": 1000," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"TeamAPIRoleConfiguration\": {" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="      \"log_statement\": \"all\"" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   }," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PodTerminateGracePeriod\":<PHONE_NUMBER>00," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PodManagementPolicy\": \"ordered_ready\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"ProtectedRoles\": [" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="      \"admin\"" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   ]," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"PostgresSuperuserTeams\": [" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="      \"postgres_superusers\"" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   ]," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"SetMemoryRequestToLimit\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableLazySpiloUpgrade\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableCrossNamespaceSecret\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnablePgVersionEnvVar\": true," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"EnableSpiloWalPathCompat\": false," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"MajorVersionUpgradeMode\": \"off\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"MinimalMajorVersion\": \"9.5\"," pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="   \"TargetMajorVersion\": \"13\"" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="}" pkg=controller
time="2022-10-14T14:59:42Z" level=debug msg="acquiring initial list of clusters" pkg=controller
time="2022-10-14T14:59:42Z" level=debug msg="added new cluster: \"galaxy/galaxy-my-galaxy-galaxy-postgres\"" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="SYNC event has been queued" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=controller worker=0
time="2022-10-14T14:59:42Z" level=info msg="there are 1 clusters running" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="started working in background" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="listening on :8080" pkg=apiserver
time="2022-10-14T14:59:42Z" level=info msg="ADD event has been queued" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=controller worker=0
time="2022-10-14T14:59:42Z" level=info msg="creating pod service account \"postgres-pod\" in the \"galaxy\" namespace" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="successfully deployed the pod service account \"postgres-pod\" to the \"galaxy\" namespace" pkg=controller
time="2022-10-14T14:59:42Z" level=debug msg="new node has been added: /gondor2013.bin.bioinf.nl ()" pkg=controller
time="2022-10-14T14:59:42Z" level=debug msg="new node has been added: /rohan2013 ()" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="Creating the role binding \"postgres-pod\" in the \"galaxy\" namespace" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="successfully deployed the role binding for the pod service account \"postgres-pod\" to the \"galaxy\" namespace" pkg=controller
time="2022-10-14T14:59:42Z" level=info msg="syncing of the cluster started" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=controller worker=0
time="2022-10-14T14:59:42Z" level=debug msg="team API is disabled" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:42Z" level=debug msg="team API is disabled" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:42Z" level=info msg="syncing secrets" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:42Z" level=debug msg="secret galaxy/galaxydbuser.galaxy-my-galaxy-galaxy-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:42Z" level=debug msg="secret galaxy/postgres.galaxy-my-galaxy-galaxy-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:42Z" level=debug msg="created new secret galaxy/standby.galaxy-my-galaxy-galaxy-postgres.credentials.postgresql.acid.zalan.do, namespace: galaxy, uid: df0ac207-f80e-45e5-8802-b98462095cf3" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:42Z" level=debug msg="syncing master service" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:42Z" level=info msg="could not find the cluster's master endpoint" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:42Z" level=warning msg="master is not running, generated master endpoint does not contain any addresses" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:42Z" level=info msg="created missing master endpoint \"galaxy/galaxy-my-galaxy-galaxy-postgres\"" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:42Z" level=info msg="could not find the cluster's master service" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:43Z" level=info msg="created missing master service \"galaxy/galaxy-my-galaxy-galaxy-postgres\"" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:43Z" level=debug msg="syncing replica service" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:43Z" level=info msg="could not find the cluster's replica endpoint" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:43Z" level=info msg="created missing replica endpoint \"galaxy/galaxy-my-galaxy-galaxy-postgres-repl\"" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:43Z" level=info msg="could not find the cluster's replica service" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:43Z" level=debug msg="No load balancer created for the replica service" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:44Z" level=info msg="created missing replica service \"galaxy/galaxy-my-galaxy-galaxy-postgres-repl\"" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:44Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:44Z" level=info msg="volume claims do not require changes" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:44Z" level=debug msg="syncing statefulsets" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:44Z" level=info msg="cluster's statefulset does not exist" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:44Z" level=debug msg="created new statefulset \"galaxy/galaxy-my-galaxy-galaxy-postgres\", uid: \"eb26faf0-1c3f-44bb-8af7-dbe2406a9368\"" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T14:59:47Z" level=debug msg="Waiting for 1 pods to become ready" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T15:00:11Z" level=info msg="created missing statefulset \"galaxy/galaxy-my-galaxy-galaxy-postgres\"" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T15:00:11Z" level=debug msg="making GET http request: http://<IP_ADDRESS>:8008/config" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T15:00:11Z" level=debug msg="patching Postgres config via Patroni API on pod galaxy/galaxy-my-galaxy-galaxy-postgres-0 with following options: {\"synchronous_mode\":false,\"synchronous_mode_strict\":false}" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T15:00:11Z" level=debug msg="making PATCH http request: http://<IP_ADDRESS>:8008/config" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T15:00:11Z" level=debug msg="restarting Postgres server within pods" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T15:00:11Z" level=debug msg="making GET http request: http://<IP_ADDRESS>:8008/patroni" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T15:00:11Z" level=debug msg="Postgres server successfuly restarted in master pod galaxy/galaxy-my-galaxy-galaxy-postgres-0" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T15:00:11Z" level=debug msg="syncing pod disruption budgets" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T15:00:11Z" level=info msg="could not find the cluster's pod disruption budget" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T15:00:11Z" level=warning msg="error while syncing cluster state: could not sync pod disruption budget: could not create pod disruption budget: the server could not find the requested resource" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=cluster
time="2022-10-14T15:00:11Z" level=error msg="could not sync cluster: could not sync pod disruption budget: could not create pod disruption budget: the server could not find the requested resource" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=controller worker=0
time="2022-10-14T15:00:11Z" level=info msg="recieved add event for already existing Postgres cluster" cluster-name=galaxy/galaxy-my-galaxy-galaxy-postgres pkg=controller worker=0

Any help would be much appreciated :)

Are you using k8s 1.25 by chance? Looks like there's an issue with it: https://github.com/galaxyproject/galaxy-helm/pull/385
ping @afgane

Thanks for the quick reply. I'm indeed running k8s 1.25.1:
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.1", GitCommit:"e4d4e1ab7cf1bf15273ef97303551b279f0920a9", GitTreeState:"clean", BuildDate:"2022-09-14T19:49:27Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.1", GitCommit:"e4d4e1ab7cf1bf15273ef97303551b279f0920a9", GitTreeState:"clean", BuildDate:"2022-09-14T19:42:30Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"linux/amd64"}

And in case it matters, the helm version:
version.BuildInfo{Version:"v3.10.0", GitCommit:"ce66412a723e4d89555dc67217607c6579ffcb21", GitTreeState:"clean", GoVersion:"go1.18.6"}

It looks like it's exactly the issue described in #385.
Do you have any idea about the release cadence of the pgsql operator? Does it make sense to report this upstream (again)?

There doesn't appear to be a fixed release cadence, but support for 1.25 appears to have been added: https://github.com/zalando/postgres-operator/issues/1999


--k parameter not functional?

Trying to understand what --k does.
Even with --k at 2, I still see training at further depth:
Category (japan): 	japan japanese tokyo yashiro nippon
Category (china): 	china beijing chinese li ''china
Category (britain): 	britain british wales p.l.c sainsbury
Category (russia): 	russia russian kremlin vladimir moscow
Category (germany): 	germany west_germany german east_germany west_german
Category (canada): 	canada alberta ontario ottawa saskatchewan
Category (france): 	france french paris chirac belgique

Should the category depth continue to 10 or be stopping early in training? I only need to go, say, 3 or 4 levels deep.

Hi,
The --k parameter controls the number of terms per category to be saved into the result file. In other words, it does not control the training process, but only influences how the results are stored. To early stop training, you can set a smaller value of -iter.
Thanks,
Yu

Thanks.


Hotfix for faulty and now redundant testing lib

fikk en D for mye i branch-navnet men....

@Roeland-Bouvet / @toredahl Ok, so the testing dependency should be removed everywhere from now on?

@Roeland-Bouvet / @toredahl Ok, so the testing dependency should be removed everywhere from now on?

No, just in this one. Am in the process of fixing the others.


Use sincos

In Julia master, where an efficient sincos function is defined, this change should provide a 5-10% speed-up.
Note that TimeIt.jl package doesn't work on Julia master, most probably due to this Julia issue: https://github.com/JuliaLang/julia/issues/23221.

This is the script I'm using for benchmark (the simple @benchmark macro doesn't currently work on Julia master for the same reason why @timeit is broken):
#!/usr/bin/env julia
using SkyCoords
using BenchmarkTools

io = open("julia_times.csv", "w")
println(io, "n,system,time")
for n in [1, 10, 100, 1000, 10_000, 100_000, 1_000_000]
    println("$n coordinates: ")
    if n == 1
        c = ICRSCoords(2pi*rand(), pi*(rand() - 0.5))
        t1 = minimum(run(tune!(@benchmarkable( convert(GalCoords, $c) ))).times) / 1e9
        println(io, "$n,galactic,$t1")
        t2 = minimum(run(tune!(@benchmarkable( convert(FK5Coords{2000}, $c) ))).times) / 1e9
        println(io, "$n,fk5j2000,$t2")
        t3 = minimum(run(tune!(@benchmarkable( convert(FK5Coords{1975}, $c) ))).times) / 1e9
        println(io, "$n,fk5j1975,$t3")
    else
        c = [ICRSCoords(2pi*rand(), pi*(rand() - 0.5)) for i=1:n]
        t1 = minimum(run(tune!(@benchmarkable( convert(Vector{GalCoords{Float64}}, $c) ))).times) / 1e9
        println(io, "$n,galactic,$t1")
        t2 = minimum(run(tune!(@benchmarkable( convert(Vector{FK5Coords{2000, Float64}}, $c) ))).times) / 1e9
        println(io, "$n,fk5j2000,$t2")
        t3 = minimum(run(tune!(@benchmarkable( convert(Vector{FK5Coords{1975, Float64}}, $c) ))).times) / 1e9
        println(io, "$n,fk5j1975,$t3")
    end
end
close(io)

Result on current master of SkyCoords:
n,system,time
1,galactic,1.0194373673036093e-7
1,fk5j2000,1.7845755395683453e-7
1,fk5j1975,1.8471951219512196e-7
10,galactic,9.211923076923078e-7
10,fk5j2000,9.809333333333332e-7
10,fk5j1975,9.872307692307692e-7
100,galactic,8.904e-6
100,fk5j2000,8.94e-6
100,fk5j1975,8.959666666666667e-6
1000,galactic,0.000111156
1000,fk5j2000,0.000112682
1000,fk5j1975,0.000112151
10000,galactic,0.001142591
10000,fk5j2000,0.00115665
10000,fk5j1975,0.001141865
100000,galactic,0.011440471
100000,fk5j2000,0.011608919
100000,fk5j1975,0.011480269
1000000,galactic,0.115397807
1000000,fk5j2000,0.116892337
1000000,fk5j1975,0.115666098

Result with this PR:
n,system,time
1,galactic,8.347925311203319e-8
1,fk5j2000,1.5674120603015074e-7
1,fk5j1975,1.6583355176933158e-7
10,galactic,8.52828125e-7
10,fk5j2000,9.316551724137931e-7
10,fk5j1975,9.428076923076923e-7
100,galactic,8.383e-6
100,fk5j2000,8.541333333333334e-6
100,fk5j1975,8.529e-6
1000,galactic,0.00010513
1000,fk5j2000,0.00010537
1000,fk5j1975,0.000105959
10000,galactic,0.001079322
10000,fk5j2000,0.001080466
10000,fk5j1975,0.001084719
100000,galactic,0.010842601
100000,fk5j2000,0.010863911
100000,fk5j1975,0.010902358
1000000,galactic,0.10911997
1000000,fk5j2000,0.109268721
1000000,fk5j1975,0.110885803

Performance improves from 4-18%, with the largest improvement for the single element conversion.

Looks good!


Update the modules to sync with workflow template - EPIC 196

Process for updating Gitops modules to support Epic (https://github.com/cloud-native-toolkit/software-everywhere/issues/196)

Based on the github action tab, the issue was fixed. Hence closing as the issue was not associated with PR image


OSGi native libraries take 3

Hello again!
It seems that I may have spoken too soon at the end of #216: What I had was actually working by accident due to a quirk of the OS I was using at the time.
Currently, LWJGL still can't be made to work on OSGi due to the way it looks up native libraries. I'll try to walk through what happens.
Firstly, I'm working from the assumption that I'll be repackaging the existing LWJGL jar files such that they become OSGi bundles, and therefore native libraries relative to each bundle will be embedded into each bundle. So, for example, the libjemalloc.so library will be inserted into an lwjgl-jemalloc bundle. Because OSGi (in addition to languages like Ceylon, and systems such as JBoss modules) uses a "peer to peer" classloading model, each bundle is loaded in its own classloader. This means that code can't pick an essentially arbitrary classloader and try to load a library from that classloader as if it were a resource; the file will most likely not be accessible via that specific classloader.
With that in mind, let's follow the execution path of Library.loadSystem("lwjgl")...
        @SuppressWarnings("try")
	public static void loadSystem(String name) throws UnsatisfiedLinkError {
		apiLog("Loading library (system): " + name);
		if ( Paths.get(name).isAbsolute() ) {
			System.load(name);
			apiLog("\tSuccess");
			return;
		}

The isAbsolute path won't be taken, because "lwjgl" isn't absolute. Therefore...
                String libName = Platform.get().mapLibraryName(name);

		URL libURL = Library.class.getResource("/" + libName);
		if ( libURL == null ) {
			// Try org.lwjgl.librarypath
			if ( loadSystem(libName, Configuration.LIBRARY_PATH) )
				return;
		} else {

The returned libName will be liblwjgl.so on this platform. Due to the peer-to-peer classloading described above, libURL will be null, so we jump to another overload of loadSystem...
private static boolean loadSystem(String libName, Configuration<String> property) {
	String paths = property.get();
	return paths != null && loadSystem(libName, property.getProperty(), paths);
}

Now this is conditional based on the current configuration, but in an OSGi context, I would expect the property to be unset, therefore we return false here. As a result, we end up...
		// Then java.library.path
		String paths = System.getProperty(JAVA_LIBRARY_PATH);
		if ( paths != null ) {
			try {
				System.loadLibrary(name);

				Path libFile = findLibrary(paths, libName);
				if ( libFile != null ) {
					apiLog(String.format("\tLoaded from %s: %s", JAVA_LIBRARY_PATH, libFile));
					checkHash(libFile);
				}
				return;
			} catch (Throwable t) {
				apiLog(String.format("\t%s not found in %s", libName, JAVA_LIBRARY_PATH));
			}
		}

		printError(true);
		throw new UnsatisfiedLinkError("Failed to locate library: " + libName);

Now, again, in an OSGi context, we'd expect JAVA_LIBRARY_PATH not to be set. This means that the call to System.loadLibrary won't happen and an exception is raised. We can set JAVA_LIBRARY_PATH to some nonsense value such as /nonexistent, but this is pretty damn ugly and I've no idea if it would even work properly across platforms.
The situation is slightly worse for libraries loaded via loadNative because although most of the above logic is the same, the method doesn't fall back to System.loadLibrary and therefore the library will be loaded from the host's filesystem, even though we've bundled a guaranteed-compatible native library for libraries such as jemalloc and glfw!
I think I'd like to restate my position that the cleanest way to deal with all of this is to provide a property that can be enabled that simply bypasses all of the above and uses System.loadLibrary directly. I appreciate that you've had support problems with this in the past as it's not really traceable, but it's currently required to work in systems like OSGi and languages like Ceylon and is reliable in the context of those systems (because they have standardized, well-understood ways of including native code).

Now, again, in an OSGi context, we'd expect JAVA_LIBRARY_PATH not to be set. This means that the call to System.loadLibrary won't happen and an exception is raised.

This is a bug and easily fixable. The current implementation indeed assumes that in order for System.loadLibrary to succeed, java.library.path must not be empty. Obviously this is not the case with OSGi, where ClassLoader.findLibrary is used.

The situation is slightly worse for libraries loaded via loadNative because although most of the above logic is the same, the method doesn't fall back to System.loadLibrary and therefore the library will be loaded from the host's filesystem, even though we've bundled a guaranteed-compatible native library for libraries such as jemalloc and glfw!

This cannot be fixed with System.loadLibrary, because these libraries are not JNI libraries. As I said here, loading them with System.loadNative is required so that LWJGL has a handle to the native library and can resolve symbols.
In theory they could be built as JNI libraries, but this would defeat a very important feature of LWJGL: being able to replace those libraries with custom builds. You may want to do that for a few reasons: enable debugging or stricter validation (all LWJGL libraries are built for top performance), include a fix or feature not present in the LWJGL release, or simply because you don't trust the LWJGL binaries.
There are two ways to resolve this:

Do not use Bundle-NativeCode for such libraries. Treat them as resources instead of libraries and simply add them to the classpath. LWJGL will take care of extracting and loading them, just like ClassLoader.findLibrary does. This means losing the benefits of OSGi metadata.

Keep using Bundle-NativeCode, somehow force OSGi to extract the native library, figure out where the library was extracted and use Configuration.<libname>_LIBRARY_NAME options to set the corresponding absolute path.

I'm not sure how 2) could be implemented. Maybe the OSGi's ClassLoader.findLibrary could be called using reflection?

Well, reflection works on Java 8, but not on Java 9 without --add-opens java.base/java.lang=ALL-UNNAMED.
Is there an OSGi API that can be used instead?

I'm still giving it some consideration, but I suspect that 2 is impossible. The reason being is that OSGi doesn't seem to have any APIs at all related to native code; the native code is supposed to be configured entirely via static metadata with the only interface being the standard Java loadLibrary methods. Additionally, each OSGi container implementation handles the actual loading of native libraries differently, so it's not really something we can predict.
I think handling the JNI libraries via System.loadLibrary once the java.library.path bug is fixed will be fine. As for the non-JNI libraries, I think it would be better to treat them as resources in the manner you suggested.
There is still the issue of actually accessing resources when there are multiple classloaders involved, but could this be as simple as just passing in a reference to the right classloader?  For example, the code that loads libglfw could pass a reference to its own classloader to Library.loadNative and the passed in classloader would be responsible for returning the resource that LWJGL then unpacks.

Please try the next snapshot (3.1.2 build 14). The first issue should be fixed and I have added an option to enable System.loadLibrary emulation in Library.loadNative (using reflection, Java 9 caveat still applies).

There is still the issue of actually accessing resources when there are multiple classloaders involved, but could this be as simple as just passing in a reference to the right classloader?

Could you explain this a bit more? Do you mean that different LWJGL bindings are loaded from different ClassLoaders?
Currently LWJGL uses Library.class.getResource() to resolve shared libraries. The new System.loadLibrary emulation also uses Library.class.getClassLoader(). One solution would be to add an overload that accepts a user-specified Class and uses that instead of Library.

Please try the next snapshot (3.1.2 build 14).

I'll give it a go today, thanks!

Could you explain this a bit more? Do you mean that different LWJGL bindings are loaded from different ClassLoaders?

Let's assume that I create one bundle per LWJGL module. For example, I put lwjgl-jemalloc  and all of its natives into one bundle, and then I put lwjgl and its natives into a separate bundle.
Then, JEmalloc calls Library.loadNative which then calls Library.class.getResource("/libjemalloc.so"); to locate the native library as a resource. The problem here is that Library is loaded by a different classloader than JEmalloc, because they're in different bundles. The resources of JEmalloc are not accessible via Library.
I've put together a small example of resource lookups across bundles here:
https://github.com/io7m/osgi-resource-lookups-20170205
There's a class in the loader module that presents an interface somewhat like Library.loadNative in that it tries to load a given resource. An example class in the module0 module tries various calls to the loader and prints the results. The loadFromLoaderClass("/module0.txt") → null case is the one that matches what LWJGL currently does: The com.io7m.cltest.module0.Example class asks the Loader class in a different bundle to load /module0.txt. It fails, because /module0.txt is not in the same bundle as Loader.
Going by the results obtained above, I think what needs to happen is for JEmalloc to pass its own class reference (JEmalloc.class)  to Library.loadNative and that class reference is then used look up the resource. This matches the loadFromGivenClass(class com.io7m.cltest.module0.Example, "/module0.txt") and as you can see, it appears to work correctly.
One alternative to this stuff is to pack literally every LWJGL module into one large bundle. But the reasons for not doing this are the same reasons that you don't currently pack all of the LWJGL modules into one large jar the way you used to. 😄

A bit of experimentation appears to show that the JNI libraries are loaded correctly in an OSGi context in the 3.1.2 snapshot.
I'm currently investigating a failure to load libraries via the new reflection path which I think may actually be a bug in the OSGi runtime I'm using, but from the results I posted in the example code above, I suspect that there's an issue in the LWJGL code too: Loading should fail because you're invoking the findLibrary method on Library's classloader instead of the classloader of a possibly different bundle (such as the hypothetical lwjgl-jemalloc bundle).
https://github.com/LWJGL/lwjgl3/blob/master/modules/core/src/main/java/org/lwjgl/system/Library.java#L219

Nice! Any chance you could push a snapshot so that I can verify this?

Sorry for the delay, the CI builds broke last night and it was too late to fix them. 3.1.2 build 15 is available now.

Thanks! I'll be giving this a go tomorrow. Couldn't get to it today. From reading the changes, I'm optimistic that it'll work.

Hi. Small issue discovered when testing the latest snapshot:
[INFO] --- maven-bundle-plugin:3.2.0:bundle (default-bundle) @ org.lwjgl.monolithic ---
[ERROR] Bundle com.io7m.bundles:org.lwjgl.monolithic:bundle:3.1.1 : Classes found in the wrong directory: {META-INF/versions/9/org/lwjgl/system/StackWalkUtil.class=org.lwjgl.system.StackWalkUtil}
[ERROR] Error(s) found in bundle configuration

This is a bug in the maven-bundle-plugin plugin, so my testing of the LWJGL changes may be delayed slightly until I can get that one fixed. Guess they aren't quite Java 9 ready yet...

Er, by "they",  I mean the maven-bundle-plugin maintainers.

Hi!
Well, there's still something not quite right: It seems like in the code path I end up taking via OSGi, there are no calls made to either System.load() or System.loadLibrary() for libGL.so.1. Is this expected behaviour? I don't understand enough of the LWJGL internals to determine how it handles native code in all cases. It seems like either one of those calls would need to happen though in order for native methods to be resolved. If this isn't expected behaviour, I'll try to trace the sequence of calls and get more information.
The actual symptom I'm seeing is that everything appears to work correctly right up until the first call to GL11.glClearColor(...) whereupon I straight away receive:
Exception in thread "renderer" java.lang.UnsatisfiedLinkError: org.lwjgl.opengl.GL11.glClearColor(FFFF)V
	at org.lwjgl.opengl.GL11.glClearColor(Native Method)
	at com.io7m.bundles.org.lwjgl.example.LWJGLExample.doExample(LWJGLExample.java:104)
	at java.lang.Thread.run(Thread.java:745)

It seems like in the code path I end up taking via OSGi, there are no calls made to either System.load() or System.loadLibrary() for libGL.so.1. Is this expected behaviour?

Yes, libGL.so is loaded with Library.loadNative. It just so happens in your case that java.library.path includes /usr/lib64, so it is loaded via the code path that emulates System.loadLibrary.

The actual symptom I'm seeing is that everything appears to work correctly right up until the first call to GL11.glClearColor(...)

Has liblwjgl_opengl.so loaded correctly? The OpenGL bindings require both libGL and liblwjgl_opengl.

I don't get any errors when loading liblwjgl_opengl.so:
[LWJGL] Version: 3.1.2 SNAPSHOT
[LWJGL] 	 OS: Linux v4.8.13-1-ARCH
[LWJGL] 	JRE: 1.8.0_121 amd64
[LWJGL] 	JVM: OpenJDK 64-Bit Server VM v25.121-b13 by Oracle Corporation
[LWJGL] Loading library (system): lwjgl
[LWJGL] 	Using SharedLibraryLoader...
[LWJGL] 	Found at: /tmp/lwjglsomeone/3.1.2-SNAPSHOT/liblwjgl.so
[LWJGL] 	Loaded from org.lwjgl.librarypath: /tmp/lwjglsomeone/3.1.2-SNAPSHOT/liblwjgl.so
[LWJGL] MemoryUtil accessor: MemoryAccessorUnsafe
[LWJGL] Loading library: jemalloc
[LWJGL] 	Using SharedLibraryLoader...
[LWJGL] 	Found at: /tmp/lwjglsomeone/3.1.2-SNAPSHOT/libjemalloc.so
[LWJGL] 	Loaded from org.lwjgl.librarypath: /tmp/lwjglsomeone/3.1.2-SNAPSHOT/libjemalloc.so
[LWJGL] MemoryUtil allocator: JEmallocAllocator
[LWJGL] Loading library: glfw
[LWJGL] 	Using SharedLibraryLoader...
[LWJGL] 	Found at: /tmp/lwjglsomeone/3.1.2-SNAPSHOT/libglfw.so
[LWJGL] 	Loaded from org.lwjgl.librarypath: /tmp/lwjglsomeone/3.1.2-SNAPSHOT/libglfw.so
[com.io7m.bundles.org.lwjgl.example.LWJGLExample][renderer]: creating window
[LWJGL] Loading library (system): lwjgl_opengl
[LWJGL] 	Using SharedLibraryLoader...
[LWJGL] 	Found at: /tmp/lwjglsomeone/3.1.2-SNAPSHOT/liblwjgl_opengl.so
[LWJGL] 	Loaded from org.lwjgl.librarypath: /tmp/lwjglsomeone/3.1.2-SNAPSHOT/liblwjgl_opengl.so

Are there any methods I can call that would provoke an error had it not been loaded correctly?

I cannot reproduce this. Could you please try a simple non-OSGi test on your system to see if it triggers the same error?

Oh, it works correctly outside of OSGi; that's never been an issue! Running the same code with the latest snapshot outside of OSGi works without issue.
I'm still trying to work out exactly what assumption the OSGi environment changes that LWJGL apparently depends on.
What I'm really looking for right now is a method that would raise some sort of error if and only if lwjgl_opengl hadn't been loaded correctly.

It appears to be a classloader issue. If I pack all of the LWJGL libraries and classes into a single bundle (so that they're all loaded via the same classloader), everything works correctly. This is less than ideal (and makes some of the work you did on LWJGL kind of pointless) but it's at least available as a last resort if I can't work out why LWJGL doesn't like multiple classloaders.

The problem may be related to Reflection.getCallerClass() in System.loadLibrary. Even though the loading of lwjgl_opengl is initiated from GL.java, the class that makes the System.loadLibrary call is Library.java, which is in the core bundle, not the OpenGL one. But, if my understanding of OSGi is correct, would it fail to find lwjgl_opengl is that were the case? I.e. you would see an error earlier, not when calling glClearColor.

I'm not sure. I think I may need to enlist the help of someone who knows the OSGi internals a little better.

Good point, I had this configured as a local profile. I'll add it to the pom.xml now.

Could you please test if there's a difference with 3.1.2 build 16?

I'll give it a shot today, thanks!

No change, unfortunately, but I think your suspicion about Reflection.getCallerClass() is correct. Tracing the execution of the attempt to load lwjgl_opengl in the debugger shows that Runtime.load0 is called with the first argument set to Library.class despite the explicit passing around of class references such as GL.class:

I suspect that the only way to work around this is going to be by subverting package-private accessibility and calling the Runtime.load0 method reflectively.

I suspect that the only way to work around this is going to be by subverting package-private accessibility and calling the Runtime.load0 method reflectively.

The code path that leads to Runtime.load0 is expected to fail under OSGi.
There was a bug in the previous commit, which I fixed. But then I realized that any reflection-based solution does not work under Java 9 with the default JVM settings. I ended up using method references so that System.loadLibrary is called in the context of the class that calls Library.loadSystem. Please try the next 3.1.2 build (17).

Still no luck with build 17, unfortunately.

The code path that leads to Runtime.load0 is expected to fail under OSGi.

This is actually the only code path that's taken when attempting to load lwjgl_opengl. Is this expected?
I've tried to build the bindings myself to try out a few things, but unfortunately they don't compile at the moment:
compile:
[javac: Core] Compiling 1256 source files to /home/someone/git/com.github/LWJGL/lwjgl3/bin/Core
[javac: Core] /home/someone/git/com.github/LWJGL/lwjgl3/modules/core/src/generated/java/org/lwjgl/nanovg/LibNanoVG.java:17: error: no suitable method found for loadSystem(Class<LibNanoVG>,String)
[javac: Core] 		Library.loadSystem(LibNanoVG.class, libName);
[javac: Core] 		       ^
[javac: Core]     method Library.loadSystem(String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Consumer<String>,Class<?>,String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Class<?>,String,Configuration<String>) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Class<?>,String,String,String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core] /home/someone/git/com.github/LWJGL/lwjgl3/modules/core/src/generated/java/org/lwjgl/nuklear/Nuklear.java:1182: error: no suitable method found for loadSystem(Class<Nuklear>,String)
[javac: Core] 	static { Library.loadSystem(Nuklear.class, Platform.mapLibraryNameBundled("lwjgl_nuklear")); }
[javac: Core] 	                ^
[javac: Core]     method Library.loadSystem(String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Consumer<String>,Class<?>,String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Class<?>,String,Configuration<String>) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Class<?>,String,String,String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core] /home/someone/git/com.github/LWJGL/lwjgl3/modules/core/src/generated/java/org/lwjgl/openvr/OpenVR.java:39: error: no suitable method found for loadSystem(Class<OpenVR>,String)
[javac: Core] 		Library.loadSystem(OpenVR.class, libName);
[javac: Core] 		       ^
[javac: Core]     method Library.loadSystem(String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Consumer<String>,Class<?>,String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Class<?>,String,Configuration<String>) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Class<?>,String,String,String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core] /home/someone/git/com.github/LWJGL/lwjgl3/modules/core/src/generated/java/org/lwjgl/stb/LibSTB.java:17: error: no suitable method found for loadSystem(Class<LibSTB>,String)
[javac: Core] 		Library.loadSystem(LibSTB.class, libName);
[javac: Core] 		       ^
[javac: Core]     method Library.loadSystem(String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Consumer<String>,Class<?>,String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Class<?>,String,Configuration<String>) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Class<?>,String,String,String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core] /home/someone/git/com.github/LWJGL/lwjgl3/modules/core/src/generated/java/org/lwjgl/util/lmdb/LMDB.java:327: error: no suitable method found for loadSystem(Class<LMDB>,String)
[javac: Core] 	static { Library.loadSystem(LMDB.class, Platform.mapLibraryNameBundled("lwjgl_lmdb")); }
[javac: Core] 	                ^
[javac: Core]     method Library.loadSystem(String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Consumer<String>,Class<?>,String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Class<?>,String,Configuration<String>) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Class<?>,String,String,String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core] /home/someone/git/com.github/LWJGL/lwjgl3/modules/core/src/generated/java/org/lwjgl/util/nfd/LibNFD.java:17: error: no suitable method found for loadSystem(Class<LibNFD>,String)
[javac: Core] 		Library.loadSystem(LibNFD.class, libName);
[javac: Core] 		       ^
[javac: Core]     method Library.loadSystem(String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Consumer<String>,Class<?>,String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Class<?>,String,Configuration<String>) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Class<?>,String,String,String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core] /home/someone/git/com.github/LWJGL/lwjgl3/modules/core/src/generated/java/org/lwjgl/util/par/LibPar.java:17: error: no suitable method found for loadSystem(Class<LibPar>,String)
[javac: Core] 		Library.loadSystem(LibPar.class, libName);
[javac: Core] 		       ^
[javac: Core]     method Library.loadSystem(String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Consumer<String>,Class<?>,String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Class<?>,String,Configuration<String>) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Class<?>,String,String,String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core] /home/someone/git/com.github/LWJGL/lwjgl3/modules/core/src/generated/java/org/lwjgl/util/simd/LibSSE.java:15: error: no suitable method found for loadSystem(Class<LibSSE>,String)
[javac: Core] 		Library.loadSystem(LibSSE.class, libName);
[javac: Core] 		       ^
[javac: Core]     method Library.loadSystem(String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Consumer<String>,Class<?>,String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Class<?>,String,Configuration<String>) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Class<?>,String,String,String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core] /home/someone/git/com.github/LWJGL/lwjgl3/modules/core/src/generated/java/org/lwjgl/util/tinyexr/TinyEXR.java:73: error: no suitable method found for loadSystem(Class<TinyEXR>,String)
[javac: Core] 	static { Library.loadSystem(TinyEXR.class, Platform.mapLibraryNameBundled("lwjgl_tinyexr")); }
[javac: Core] 	                ^
[javac: Core]     method Library.loadSystem(String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Consumer<String>,Class<?>,String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Class<?>,String,Configuration<String>) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Class<?>,String,String,String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core] /home/someone/git/com.github/LWJGL/lwjgl3/modules/core/src/generated/java/org/lwjgl/util/tinyfd/TinyFileDialogs.java:35: error: no suitable method found for loadSystem(Class<TinyFileDialogs>,String)
[javac: Core] 		Library.loadSystem(TinyFileDialogs.class, Platform.mapLibraryNameBundled("lwjgl_tinyfd"));
[javac: Core] 		       ^
[javac: Core]     method Library.loadSystem(String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Consumer<String>,Class<?>,String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Class<?>,String,Configuration<String>) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Class<?>,String,String,String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core] /home/someone/git/com.github/LWJGL/lwjgl3/modules/core/src/generated/java/org/lwjgl/util/xxhash/LibXXHash.java:17: error: no suitable method found for loadSystem(Class<LibXXHash>,String)
[javac: Core] 		Library.loadSystem(LibXXHash.class, libName);
[javac: Core] 		       ^
[javac: Core]     method Library.loadSystem(String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Consumer<String>,Class<?>,String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Class<?>,String,Configuration<String>) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Class<?>,String,String,String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core] /home/someone/git/com.github/LWJGL/lwjgl3/modules/core/src/generated/java/org/lwjgl/util/yoga/Yoga.java:309: error: no suitable method found for loadSystem(Class<Yoga>,String)
[javac: Core] 	static { Library.loadSystem(Yoga.class, Platform.mapLibraryNameBundled("lwjgl_yoga")); }
[javac: Core] 	                ^
[javac: Core]     method Library.loadSystem(String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Consumer<String>,Class<?>,String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Class<?>,String,Configuration<String>) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core]     method Library.loadSystem(Class<?>,String,String,String) is not applicable
[javac: Core]       (actual and formal argument lists differ in length)
[javac: Core] 12 errors

$ git log -n 1
commit 396b02333a77d54722182c99c73736b6f02bdf44
Author: Ioannis Tsakpinis<EMAIL_ADDRESS>Date:   Sat Feb 18 15:00:26 2017 +0200

    Update kotlinc to 1.1-rc

I'm curious what will happen if the following change is made to loadSystem but I can't try it locally due to the above compilation error:
    private static boolean loadSystem(Class<?> context, String libName, String property, String paths) {
        Path libFile = findLibrary(paths, libName);
        if (libFile == null) {
            apiLog(String.format("\t%s not found in %s=%s", libName, property, paths));
            return false;
        }

        try {
            final Method m = ClassLoader.class.getDeclaredMethod("loadLibrary", Class.class, String.class, boolean.class);
            m.setAccessible(true);
            m.invoke(null, context, libFile.toAbsolutePath().toString(), true);
        } catch (NoSuchMethodException | IllegalAccessException | InvocationTargetException e) {
            apiLog(String.format("\tError loading library (%s) via reflective ClassLoader call: %s %s", libName, e, e.getMessage()));
        }

        System.load(libFile.toAbsolutePath().toString());
        apiLog(String.format("\tLoaded from %s: %s", property, libFile));
        checkHash(context, libFile);
        return true;
    }

This is actually the only code path that's taken when attempting to load lwjgl_opengl. Is this expected?

Oh, I see. No, this is not expected if you use OSGi bundles for all libraries. System.load is only used with absolute paths to shared libraries. In the screenshot above I see /tmp/lwjglsomeone/3.1.2-SNAPSHOT/, which is a path generated by the SharedLibraryLoader. In a pure OSGi run, that path shouldn't be used except for non-JNI libraries.
Not sure what is happening exactly, but I would recommend cleaning up your setup so that we can test each issue separately. Delete the snapshot folder and make sure the application is launched without -Djava.library.path or -Dorg.lwjgl.librarypath.

I've tried to build the bindings myself to try out a few things, but unfortunately they don't compile at the moment:

You need to regenerate the bindings. Running ant compile-templates should do it.

I'm curious what will happen if the following change is made to loadSystem

This is effectively what was done in the previous commit. As I said, it should have worked in theory (if it didn't have the bug), but I want to avoid reflection on java.lang internals as it is not permitted (by default) on Java 9. Also, the necessary calls are different on Android (that I need to worry about from now on).

Not sure what is happening exactly, but I would recommend cleaning up your setup so that we can test each issue separately. Delete the snapshot folder and make sure the  application is launched without -Djava.library.path or -Dorg.lwjgl.librarypath.

Agreed. To be clear, I'm running the program with:
-Dorg.lwjgl.util.Debug=true -Dorg.lwjgl.system.EmulateSystemLoadLibrary=true -Dorg.lwjgl.util.DebugLoader=true -Xcheck:jni -verbose:jni

I have to run a Maven task to repackage the published LWJGL jars into OSGi bundles each time, so I'll be sure to routinely nuke ~/.m2/repository/org/lwjgl and ~/.m2/repository/com/io7m/bundles/ each time from now on.

You need to regenerate the bindings. Running ant compile-templates should do it.

Got it. Seemed like I needed an ant clean as well, for some reason.

I want to avoid reflection on java.lang internals as it is not permitted (by default) on Java 9.

Yes, definitely best to avoid this stuff if at all possible.

Hah, making the change I described instead gave me:
Exception in thread "renderer" java.lang.UnsatisfiedLinkError: Native Library /tmp/lwjglsomeone/3.1.2-SNAPSHOT/liblwjgl_opengl.so already loaded in another classloader
	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1907)
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
	at java.lang.Runtime.load0(Runtime.java:809)
	at java.lang.System.load(System.java:1086)
	at org.lwjgl.system.Library.loadSystem(Library.java:170)
	at org.lwjgl.system.Library.loadSystem(Library.java:152)
	at org.lwjgl.system.Library.loadSystem(Library.java:119)
	at org.lwjgl.opengl.GL.<clinit>(GL.java:76)
	at com.io7m.bundles.org.lwjgl.example.LWJGLExample.doExample(LWJGLExample.java:101)
	at java.lang.Thread.run(Thread.java:745)

I can't help but be amused by how awkward this problem is.

Eliminating the second call to System.load seems to work correctly:
private static boolean loadSystem(Class<?> context, String libName, String property, String paths) {
        Path libFile = findLibrary(paths, libName);
        if (libFile == null) {
            apiLog(String.format("\t%s not found in %s=%s", libName, property, paths));
            return false;
        }

        try {
            final Method m = ClassLoader.class.getDeclaredMethod("loadLibrary", Class.class, String.class, boolean.class);
            m.setAccessible(true);
            m.invoke(null, context, libFile.toAbsolutePath().toString(), true);
        } catch (NoSuchMethodException | IllegalAccessException | InvocationTargetException e) {
            apiLog(String.format("\tError loading library (%s) reflectively: %s %s", libName, e, e.getMessage()));
        }

        apiLog(String.format("\tLoaded from %s: %s", property, libFile));
        checkHash(context, libFile);
        return true;
    }

This code is wrong, obviously: Assuming that the call was going to be made reflectively, then presumably there should be an attempt made to call System.load if the reflective call failed.

Again, this right here makes no sense to me: /tmp/lwjglsomeone/3.1.2-SNAPSHOT/liblwjgl_opengl.so.
Have you cleared that folder before testing the latest build? liblwjgl_opengl.so being there is only possible if LWJGL was able to find it in the classpath and extract it. That is NOT the case in an OSGi bundle (afaict) and that's what this entire issue is about.
System.load is called when:

The library name is an absolute path.
The library is found in java.library.path.
The library is found in org.lwjgl.librarypath.
The library is found in the classpath and extracted to the temp org.lwjgl.librarypath.

The lwjgl_opengl library is loaded in the static initializer of the GL class. Using the latest build, this is what should happen when running under OSGi:

The path is not absolute, so System.load is skipped.
GL.getResource("/liblwjgl_opengl.so") should return null, so the SharedLibraryLoader is not used.
org.lwjgl.librarypath is not set, so nothing is loaded from there.
Finally, System.loadLibrary is called, from a class that exists in the OpenGL module/bundle. Reflection.getCallerClass() should return the correct class, which then returns the correct ClassLoader, which then triggers OSGi's findLibrary.

Could you please verify the above?

GL.getResource("/liblwjgl_opengl.so") should return null, so the SharedLibraryLoader is not used.

This assumption is incorrect. Thanks to the change you made a couple of commits back:
        // METHOD 2: org.lwjgl.librarypath
        URL libURL = context.getResource("/" + libName);
        if (libURL == null) {
            if (loadSystem(context, libName, Configuration.LIBRARY_PATH))
                return;
        } else {
            // Always use the SLL if the library is found in the classpath,
            // so that newer versions can be detected.
            boolean debugLoader = Configuration.DEBUG_LOADER.get(false);
            try {
                if (debugLoader)
                    apiLog("\tUsing SharedLibraryLoader...");
                // Extract from classpath and try org.lwjgl.librarypath
                try (FileChannel ignored = SharedLibraryLoader.load(name, libName, libURL)) {
                    if (loadSystem(context, libName, Configuration.LIBRARY_PATH))
                        return;
                }
            } catch (Exception e) {
                if (debugLoader)
                    e.printStackTrace(DEBUG_STREAM);
            }
        }

For lwjgl_opengl, the returned URL looks like bundle://10.0:1/liblwjgl_opengl.so which is sufficient for OSGi to locate the resource and for LWJGL to extract it. A reference to the library is also present in the OSGi metadata so we still get the benefits of that.

To clarify: Yes, I'm deleting the temporary directory between each build.
I think the main issues up to this point have been:

Locating the native libraries as resources so that LWJGL could extract them. This was originally broken because LWJGL was assuming one global classloader but was fixed when you started passing around explicit class references.

Loading the extracted native libraries in the right class loader. This one's still an issue but works given my to-be-avoided reflective call to  ClassLoader.loadLibrary. This one seems to be fixable if we can just get Reflection.getCallerClass() to return the right value via System.load. I think your previous changes would have worked, if not for the misunderstanding about which path the code would be taking on OSGi (which may've been my fault, perhaps I didn't communicate it clearly).

Build 18 is up. The call sites are a bit ridiculous now, if this works I'll try to figure out a cleaner implementation when I have more time.

SUCCESS!
🎉
Nice work, thank you! This has certainly been one of the best responses I've had to a bug report in recent memory. It's very life-affirming when developers work to assist with use-cases that they themselves don't have.
Right now, my plan is to maintain a set of OSGi bundles published to Maven Central under the com.io7m.bundles groupId. I'd be happy to move these into a repos in the LWJGL organization (perhaps with packages published under a separate org.lwjgl.osgi groupId) and maintain them in perpetuity, if you want them. They're really just a set of Maven POMs that turn the published LWJGL jars into OSGi bundles.

Awesome! Thanks for the great feedback and for having the patience to go through this process.
I have created the lwjgl3-osgi repository and made you a collaborator. It's currently empty and you're welcome to push anything to it. When I have a better understanding of the OSGi packaging process, I'll let you know how to proceed (if we'll move the scripts to a different repo, what groupId to use on Maven etc).
Lets use the new repository for further discussions.

Filed a ticket for this, just for future reference: https://github.com/LWJGL/lwjgl3-osgi/issues/4

Hey @io7m, could you please verify that the latest snapshot (LWJGL 3.1.4 build 3) doesn't break the OSGi bundles? See here for details.

Doing it now!

It works correctly.
I had to introduce a couple of small bundle metadata fixes for OSGi containers that are running on Java 9 VMs, but other than that, the bundles continue to work on both OpenJDK 8 and OpenJDK 9.


Fix Guava sha1 [phab:D2270]

Created by schroederc at 2018-04-30 19:38:13:

danielmoy-google accepted this revision at 2018-04-30 19:52:59

schroederc closed via commit d7f41d525de8bcbfa47af5454e9991356d84fa5b at 2018-04-30 19:53:23


Scrolling not working on live edit sidebar

Hello,
I have an issue on live edit. When I want to add a layout from sidebar, scroll not working (on touch device). Only working with mouse scroll on desktop. How can I fix this issue.
P.S: I'm not updating mw version. Because I changed some core code in src folder.

hi @ozanmora , thanks for report
@raikov can you check?


Region state is not saved when game is shut down

The state of the region is not saved when the game is shut down. This can cause an issue where the player saves in a region that was not dark when the game save was created but, if it's on the default list, is dark when they load.
To test:

Set a region to be dark by default through the plugin parameter menu.
Enter the game and toggle that region so it is not dark.
Save the game and exit.
Reload the game from that save file.
The region should not be dark, having had its state saved.

Fix pushed to v0.02 development branch.


Reconciliation Loop for Prometheus Converter

Current behavior of the vmprometheusconverter rely on Prometheus CRD create and update events, e.g.:
https://github.com/VictoriaMetrics/operator/blob/e261c37dc973154ff71073d7213000421416bd4e/controllers/vmprometheusconverter_controller.go#L86-L88
In some cases (like manual or circumstantial deletion of custom resources) it leads to "orphaned" Prometheus resources.
So, the problem is VictoriaMetrics Operator does not recreate converted VM-resources after ones being deleted
Steps to reproduce

Create Prometheus resource, e.g. PodMonitor

Operator creates corresponding PodScraper resource

Delete created PodScraper resource and keep original Prometheus PodMonitor

PodScraper resource will not be created until PodMonitor will be updated or recreated

Would you please consider implementing reconciliation loop for Prometheus resources in vmprometheusconverter?

Hello, it makes sense to me. And it'd be great to have an option to enable period reconciliation of prometheus watched objects.

Added new flag for period reconciliation at v0.42.2 release
It accepts duration format, -controller.prometheusCRD.resyncPeriod=10m.

@f41gh7 thx, much appreciated


Modify Side Input Data Types for cuDNN BatchNorm

Side inputs for cuDNN Fused BatchnormForward have been taking the incorrect data type till now. It should be the same data type as the input. This PR aims to correct that in the stream_executor and core/kernels.

@AyanmoI  Can you please check @sanjoy's comments and keep us posted ? Thanks!

Hi @sanjoy, I think this is an obvious wrong use of data type of side input. As shown below, the side_input should be T but we convert it later to U:
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/ops/nn_ops.cc#L228
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/kernels/fused_batch_norm_op.cc#L913.
Unfortunately or fortunately, it is hard to write a unit test case to hit it, because we've limit the U to be float, which is wider than or equal to the T (half or float). So, always using U/float for the side_input would be actually "safe" enough when we use the AsDeviceMemory to convert a half tensor to obtain its raw pointer. However, this bug can be encountered when we want to explicitly call the ThenBatchNormalizationForward as shown in tensorflow/compiler/xla/service/gpu/cudnn_batchnorm_runner.cc from this PR.


Investigate why Google cache is not rendering grommet.github.io properly

http://webcache.googleusercontent.com/search?q=cache:kCYoSj2FUYsJ:www.grommet.io/&num=1&hl=en&gl=us&strip=0&vwsrc=0
This url blinks in chrome. it works well in safari

This should be fixed by removing call to deprecated constructor (and resulting console error).  Will wait to close until verifying after Google search has re-crawled site.
https://github.com/grommet/grommet-docs/commit/26cd71d53305472671d5490cd3fee68adb1531bf

The Google cache goes blank in Chrome and Firefox (but not Safari) when the javascript for the Google Analytics loads (it appears).  At this point, the issue is not critical enough to warrant investment of more time.