Setting learning_phase to 0 leads to extremely low accuracy

Loading a persisted model and setting the learning_phase=0 reduces the accuracy from 100% to 50% in binary classification problem.
The below script contains a simple example that reproduces the problem on Keras 2.0.5 and TensorFlow 1.2 (Python 2.7, Ubuntu 14.04, Nvidia Quadro K2200 GPU). I use an extremely small dataset and I intentionally overfit the model.
Snippet:
import tensorflow as tf
import numpy as np
from keras.applications.resnet50 import ResNet50
from keras.preprocessing import image
from keras.models import Model, load_model
from keras.layers import Dense, Flatten
from keras import backend as K

epochs = 5
input_shape = (224, 224, 3)
batch_size = 32
seed = 42
dataset_path = './data/cifar2tiny' # same train/test dataset to overfit it

np.random.seed(seed)
tf.set_random_seed(seed)

K.set_learning_phase(1)
base_model = ResNet50(weights='imagenet', include_top=False, input_shape=input_shape)

x = base_model.output
x = Flatten(name='flatten')(x)
predictions = Dense(2, activation='softmax', name='predictions')(x)
model = Model(inputs=base_model.input, outputs=predictions)

for layer in model.layers[0:141]:
    layer.trainable = False

model.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])

train_gen = image.ImageDataGenerator().flow_from_directory(dataset_path, target_size=input_shape[:2], batch_size=batch_size, class_mode='categorical', shuffle=True, seed=seed)
test_gen = image.ImageDataGenerator().flow_from_directory(dataset_path, target_size=input_shape[:2], batch_size=batch_size, class_mode='categorical', shuffle=True, seed=seed)
train_steps = train_gen.samples//batch_size
test_steps = test_gen.samples//batch_size

model.fit_generator(train_gen, train_steps, epochs=epochs, validation_data=test_gen, validation_steps=test_steps)

test_gen.reset()
print('Before Save:', model.evaluate_generator(test_gen, test_steps)) # Accuracy close to 100%
model.save('/tmp/tmpModel')

K.clear_session()
K.set_learning_phase(0)
test_gen.reset()
model = load_model('/tmp/tmpModel')
print('After Load - learning_phase=0:', model.evaluate_generator(test_gen, test_steps)) # Accuracy close to 50%

K.clear_session()
K.set_learning_phase(1)
test_gen.reset()
model = load_model('/tmp/tmpModel')
print('After Load - learning_phase=1:', model.evaluate_generator(test_gen, test_steps)) # Accuracy close to 100%

Output:
Found 100 images belonging to 2 classes.
Found 100 images belonging to 2 classes.
Epoch 1/5
3/3 [==============================] - 6s - loss: 1.7213 - acc: 0.5833 - val_loss: 0.9526 - val_acc: 0.8438
Epoch 2/5
3/3 [==============================] - 3s - loss: 1.0977 - acc: 0.7642 - val_loss: 0.3841 - val_acc: 0.9559
Epoch 3/5
3/3 [==============================] - 3s - loss: 0.5245 - acc: 0.8649 - val_loss: 0.1062 - val_acc: 0.9706
Epoch 4/5
3/3 [==============================] - 3s - loss: 0.0637 - acc: 0.9885 - val_loss: 0.1152 - val_acc: 0.9853
Epoch 5/5
3/3 [==============================] - 4s - loss: 0.1557 - acc: 0.9688 - val_loss: 0.0218 - val_acc: 0.9896
('Before Save:', [0.023946404457092285, 1.0])
('After Load - learning_phase=0:', [7.8911511103312177, 0.51041666666666663])
('After Load - learning_phase=1:', [0.028959342899421852, 0.98958333333333337])

As you can see above, I explicitly save the model, clear the session and load it again. This is important for reproducing the problem. I don't believe that there is an issue on the persistence mechanism of Keras as the weights before and after the load() seem the same.
Since ResNet50 does not contain any Dropout layer, I believe the problem is caused by the BatchNormalization layers. As far as I see on Keras source, during training we use the sample mean/variance of the mini-batch while during testing we use the rolling mean/variance.
Any thoughts from Keras contributors? I'm happy to provide more info or investigate further.
@fchollet Could you provide any hint/pointers where to look next?

[x] Check that you are up-to-date with the master branch of Keras. You can update with:
pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps

[x] If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

[x] If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

[x] Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

I did a simple test to check whether the BatchNormalization was the culprit, and do not found evidence of it. Specifically,
from keras.models import load_model, Sequential
from keras.layers import Dense, Flatten, Conv2D, MaxPooling2D, BatchNormalization
from keras.datasets import mnist
from keras import backend as K
import keras

import os

# input image dimensions
img_rows, img_cols = 28, 28
num_classes = 10

# the data, shuffled and split between train and test sets
(x_train, y_train), (x_test, y_test) = mnist.load_data()

if K.image_data_format() == 'channels_first':
    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)
    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)
    input_shape = (1, img_rows, img_cols)
else:
    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
    input_shape = (img_rows, img_cols, 1)

x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255

# convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

def get_trained_model(file_path):
    if os.path.exists(file_path):
        return load_model(file_path)

    batch_size = 128
    epochs = 12

    model = Sequential()
    model.add(Conv2D(16, kernel_size=(3, 3),
                     activation='relu',
                     input_shape=input_shape))
    model.add(Conv2D(4, (3, 3), activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    # Use batch normalization instead of Dropout to test it
    model.add(BatchNormalization())
    model.add(Flatten())
    model.add(Dense(16, activation='relu'))
    model.add(BatchNormalization())
    model.add(Dense(num_classes, activation='softmax'))

    model.compile(loss=keras.losses.categorical_crossentropy,
                  optimizer=keras.optimizers.Adadelta(),
                  metrics=['accuracy'])

    model.fit(x_train, y_train,
              batch_size=batch_size,
              epochs=epochs,
              verbose=2,
              validation_data=(x_test, y_test))

    model.save(file_path)
    return model

model = get_trained_model('test.h5')

print('Before Load:', model.evaluate(x_test, y_test, verbose=0))

K.clear_session()
K.set_learning_phase(0)
model = load_model('test.h5')
print('After Load - learning_phase=0:', model.evaluate(x_test, y_test, verbose=0))

K.clear_session()
K.set_learning_phase(1)
model = load_model('test.h5')
print('After Load - learning_phase=1:', model.evaluate(x_test, y_test, verbose=0))

I get (TensorFlow, Keras)

Before Load: [0.056945249618217349, 0.98070000000000002]
After Load - learning_phase=0: [0.056945249618217349, 0.98070000000000002]
After Load - learning_phase=1: [0.056797642182186248, 0.98170000000000002]

@jorgecarleitao I think it is important to do fine-tuning on the model (freeze some of the layers) in order to reproduce the results. Note that the way I do fine-tuning here comes from the documentation. Could you add a couple more layers on your example and freeze part of the network? Alternatively you can run my snippet.
I kept digging into this and I noticed that that when you freeze part of the model, the moving mean and variance of the frozen BatchNormalization layers keep updating their values (beta and gamma are not though). Perhaps @fchollet can clarify if this is a bug or an intended behaviour. This might actually be responsible for the problem.
It is worth noting that if this is indeed a bug, it can heavily affect people who deploy their models on the live environment.

Maybe related to #4762
As explained in my comment, I think it should be possible to freeze the BatchNormalization parameters when trainable = False.

@undo76 Indeed I observe the same effect, akka the rolling mean and variance of BatchNormalization are being updated even when the layer is frozen. Still it is not clear to me if this is related to the low accuracy bug that I report here.
@fchollet Any thoughts?

@jorgecarleitao I actually added more layers to your snippet and performed fine-tuning but I can't make it break. Still uncertain on whether this is caused by the BN layer.

OK I believe I know what is the problem. It is not a bug, but a side-effect of the way we estimate the moving averages on BatchNormalization.
The mean and variance of the training data that I use are different from the ones of the dataset used to train the ResNet50 (the effect is amplified by the fact I don't subtract the average pixel & flip the channel order but you can actually get the same result even if you do). Because the momentum on the BatchNormalization has a default value of 0.99, with only 5 iterations it does not converge quickly enough to the correct values for the moving mean and variance. This is not obvious during training when the learning_phase is 1 because BN uses the mean/variance of the batch. Nevertheless when we set learning_phase to 0, the incorrect mean/variance values which are learned during training significantly affect the accuracy. The reason why @jorgecarleitao 's snippet does not reproduce the problem is because he trains the model from scratch rather than using pre-trained weights.
There are two ways to demonstrate that this is the root of the problem:
1. More iterations
On my original snippet, reduce the size of the batch from 32 to 16 (to perform more updates per epoch) and increase the number of epochs from 5 to 250. This way the moving average and variance will converge to the correct values.
Output:
('Before Save:', [1.1920930376163597e-07, 1.0])
('After Load - learning_phase=0:', [1.6205018008956054e-07, 1.0])
('After Load - learning_phase=1:', [5.8301013202329466e-07, 1.0])

2. Change the momentum of BatchNormalization
Keep the number of iterations fixed but change the momentum of the BatchNormalization layer to update more aggressively the rolling mean and variance (not recommended for production models). Note that changing the momentum field after the model has been initialised will not have any effect (the graph on Tensorflow has already been constructed using this value), so we use a hacky patch to demonstrate the case.
On my original snippet, add the following patch between reading the base_model and defining the new layers:
# ....
base_model = ResNet50(weights='imagenet', include_top=False, input_shape=input_shape)

# PATCH MOMENTUM - START
import json
conf = json.loads(base_model.to_json())
for l in conf['config']['layers']:
    if l['class_name'] == 'BatchNormalization':
        l['config']['momentum'] = 0.5

m = Model.from_config(conf['config'])
for l in base_model.layers:
    m.get_layer(l.name).set_weights(l.get_weights())

base_model = m
# PATCH MOMENTUM - END

x = base_model.output
# ....

Output:
('Before Save:', [0.025331034635504086, 1.0])
('After Load - learning_phase=0:', [0.01937261379013459, 1.0])
('After Load - learning_phase=1:', [0.14217917621135712, 0.98958333333333337])

Hope this helps others that face similar issues.

Hi @datumbox ,
For this problem what do i do ? Is necessary to set learning_phase to 1? apparently, this problem is solved in the above version of keras 2.1.0, right? if right, for fine-tuning the network is not necessary to set this learning_phase in codes?


Merge berkeley into develop

Mostly in order to get back https://github.com/MinaProtocol/mina/pull/14606 for develop

!ci-build-me

!ci-build-me

!ci-build-me


add LV-624

About the PR

did my best to manually re-create LV-624 from cm13
Why / Balance

it wasn't on the repo
the differences from the non-repo version that the main server uses is that there are floodlights, the floors aren't uniformly "CMFloorSteel", and there are podlocks for the garage/research/secure storage
Technical details

Media

[ ] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

it's too huge i can't screenshot it all
Breaking changes

Changelog

:cl:

add: LV-624 is now on the github repository.

so does sand temple finally have temple walls, and does containers finally get containers

unfortunately no

whats wrong with the sussy map that isnt on repo

the figurines are kind of randomly scattered around the map on top of tables, there's no real method to placing them so long as they're well spaced apart
Though yes, Cuir is right, this seems like a downgrade, you should ask Smug for the actual map file like she offered so you can improve what is already done?

LV is public on the repo now so I'm closing this.


Istio Virtual Service DNS not resolving

(NOTE: This is used to report product bugs:
To report a security vulnerability, please visit https://istio.io/about/security-vulnerabilities
To ask questions about how to use Istio, please visit https://discuss.istio.io)
Bug description
I am running istio v1.8.1 on an EKS v18 cluster with the demo profile, installed via the istio operator.
I created a virtual service in the hello-world namespace with the following spec:
spec:
  gateways:
  - mesh
  hosts:
  - hello-world.hello-world
  - hello-world
  - google.com
  - blah.dev
  http:
  - match:
    - port: 80
    - port: 443
    route:
    - destination:
        host: hello-world.hello-world.svc.cluster.local

But it seems like there is dns resolution problems:
Coming from a pod in the service mesh in the SAME namespace: hello-world, hello-world.hello-world and google.com all resolve to the VS' destination with no problem, but "blah.dev" could not resolve host: curl: (6) Could not resolve host: blah.dev
Coming from a pod in the service mesh in a DIFFERENT namespace: hello-world.hello-world and google.com resolve to the VS' destination with no problem, but "blah.dev" and "hello-world" do not with the same issue as before.
It seems to me that the pod (at least when using curl) can not resolve dns hosts defined by istio virtual services. Rather, the host must exist either as a public domain or one defined as a k8s service. If the dns can be resolved by the pod (e.g. google.com or hello-world in the same namespace as the service, or the expanded hello-world.hello-world which includes the namespace), then istio will route it correctly.
Affected product area (please put an X in all that apply)
[ ] Docs
[ ] Installation
[X] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
[ ] Upgrade
Affected features (please put an X in all that apply)
[ ] Multi Cluster
[ ] Virtual Machine
[ ] Multi Control Plane
Expected behavior
hello-world, and blah.dev resolve to the virtual service's destination regardless of namespace.
Steps to reproduce the bug
Please see description above
Version (include the output of istioctl version --remote and kubectl version --short and helm version --short if you used Helm)
How was Istio installed?
Via the istio operator. Here is the profile:
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: istiocontrolplane
  labels:
  {{- include "istio-vanilla-profile.labels" . | nindent 4 }}
spec:
  profile: default
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
        k8s:
          service:
            externalTrafficPolicy: Local
            ports:
            - name: status-port
              port: 15021
              protocol: TCP
              targetPort: 15021
            - name: http2
              port: 80
              protocol: TCP
              targetPort: 8080
            - name: https
              port: 443
              protocol: TCP
              targetPort: 8443
            - name: tcp
              port: 31400
              protocol: TCP
              targetPort: 31400
            - name: grpc
              port: 50051
              protocol: TCP
              targetPort: 50051
            - name: tls
              port: 15443
              protocol: TCP
              targetPort: 15443
  values:
    gateways:
      istio-ingressgateway:
        loadBalancerSourceRanges:
          - <IP_ADDRESS>/0
        serviceAnnotations:
          service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
          service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https,grpc"
          service.beta.kubernetes.io/aws-load-balancer-ssl-cert: REDACTED
          external-dns.alpha.kubernetes.io/hostname: REDACTED

Environment where the bug was observed (cloud vendor, OS, etc)
EKS v 1.18, freshly spun up
Additionally, please consider running istioctl bug-report and attach the generated cluster-state tarball to this issue.
Refer cluster state archive for more details.
bug-report.zip

Istio is not responsible for dns resolve in this case.

Istio is not responsible for dns resolve in this case.

dns-proxy could do this for you

dns-proxy could do this for you

Yes thank you, I got this clarified with the good folks in the istio slack channel. For other folks with the same question:
The VS hosts can be any valid DNS, but apps still need it to be resolvable even if you have valid routes for it. So what y'all need to do is somehow make the pods recognize a dns entry for the alias you want to use. You can actually do this in the upcoming istio release by doing something like this! https://preliminary.istio.io/latest/docs/ops/configuration/traffic-management/dns-proxy/

Yes thank you, I got this clarified with the good folks in the istio slack channel. For other folks with the same question:
The VS hosts can be any valid DNS, but apps still need it to be resolvable even if you have valid routes for it. So what y'all need to do is somehow make the pods recognize a dns entry for the alias you want to use. You can actually do this in the upcoming istio release by doing something like this! https://preliminary.istio.io/latest/docs/ops/configuration/traffic-management/dns-proxy/


Multiple escape keys

Thanks for the plugin!
I tend to map jk and it's variant kj to esc. It would be nice to be able to map multiple versions to esc

Multiple escape keys should be easy to implement. How many variants do you usually use?

I can only speak for myself but right now I only have the two: jk, kj

I use 4 keys.

@D1mon what do you mean by four keys? One mapping with 4 characters or four mapping each with 2 characters?

inoremap jj
inoremap оо
inoremap kk
inoremap лл

@D1mon @sharksforarms Multiple shortcuts feature has been added. Please update and check if it works for you.
Also see :h better-escape-shortcut-limitation for the limitation on defining multiple shortcuts.


Segfault in SparseArrayLookup; CAAction's protocol version is invalid

Hello 👋, I have encountered a segfault in SparseArrayLookup during the initialization of my port of the application GitY. Here is the backtrace from GDB:
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff79c8a15 in SparseArrayLookup () from /usr/GNUstep/Local/Library/Libraries/libobjc.so.4.6
(gdb) bt
#0  0x00007ffff79c8a15 in SparseArrayLookup () from /usr/GNUstep/Local/Library/Libraries/libobjc.so.4.6
#1  0x00007ffff79c9369 in objc_send_initialize () from /usr/GNUstep/Local/Library/Libraries/libobjc.so.4.6
#2  0x00007ffff79c9103 in objc_send_initialize () from /usr/GNUstep/Local/Library/Libraries/libobjc.so.4.6
#3  0x00007ffff79d2039 in slowMsgLookup () from /usr/GNUstep/Local/Library/Libraries/libobjc.so.4.6
#4  0x00007ffff79d77dc in objc_msgSend_fpret () from /usr/GNUstep/Local/Library/Libraries/libobjc.so.4.6
#5  0x00007ffff5f2f63a in +[CGImageDestinationTIFF load] (self=0x7ffff5f5b778 <._OBJC_CLASS_CGImageDestinationTIFF>, _cmd=0x7ffff7fab000)
    at image/OPImageCodecTIFF.m:323
#6  0x00007ffff79c552b in objc_send_load_message () from /usr/GNUstep/Local/Library/Libraries/libobjc.so.4.6
#7  0x00007ffff79c60e8 in objc_resolve_class () from /usr/GNUstep/Local/Library/Libraries/libobjc.so.4.6
#8  0x00007ffff79c628d in objc_resolve_class_links () from /usr/GNUstep/Local/Library/Libraries/libobjc.so.4.6
#9  0x00007ffff79cc28c in __objc_load () from /usr/GNUstep/Local/Library/Libraries/libobjc.so.4.6
#10 0x00007ffff5f0be8d in objcv2_load_function () from /usr/GNUstep/Local/Library/Libraries/libopal.so.0
#11 0x00007ffff7fc947e in call_init (l=<optimized out>, argc=argc@entry=1, argv=argv@entry=0x7fffffffd3d8, env=env@entry=0x7fffffffd3e8)
    at ./elf/dl-init.c:70
#12 0x00007ffff7fc9568 in call_init (env=0x7fffffffd3e8, argv=0x7fffffffd3d8, argc=1, l=<optimized out>) at ./elf/dl-init.c:33
#13 _dl_init (main_map=0x7ffff7ffe2e0, argc=1, argv=0x7fffffffd3d8, env=0x7fffffffd3e8) at ./elf/dl-init.c:117
#14 0x00007ffff7fe32ca in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#15 0x0000000000000001 in ?? ()
#16 0x00007fffffffd8a2 in ?? ()
#17 0x0000000000000000 in ?? ()

Interestingly, I have not encountered this when running any of the libs-opal tests, such as images. I have confirmed with GDB that images calls +[CGImageDestinationTIFF load] without segfaulting. Therefore, I don't know in which component this came from.
The Objective-C components that are linked to by GitY are:

GitUpKit
MiscHeaders (headers only)
libobjc2
GNUstep Makefiles Library
GNUstep-base (Foundation Kit)
My custom port of CoreFoundation to GNUstep with bridging enabled
libdispatch
GNUstep-gui (AppKit)
Opal - Core Graphics
QuartzCore - Core Animation
libs-boron (my fork)

It's possible that one of those has messed something up in their +load methods, but I'm not exactly sure. If needed, I can send you binaries of my GNUstep installation, compile libobjc2 with different flags, peek around in GDB, or whatever else would be needed.

@gcasa Is this the right way to file this, or should I move this to a different repo or to discuss-gnustep?

Do you have a reduced test case (ideally linking nothing but the runtime)?
It looks as if it may be trying to send a message in +load to a class that is not yet loaded.

I think I might have forgotten to link something. I don't have a minimal test case -- as I said, I have no idea where this came from.

I looked and it seems like I have linked everything.

Do you have a reduced test case (ideally linking nothing but the runtime)?
It looks as if it may be trying to send a message in +load to a class that is not yet loaded.

Is the order of the +load messages defined? It doesn't occur in the libs-opal testcases, so if this is the reason for the segfault, then the load message order is different.

@davidchisnall I believe this is the problem...
+[CGImageDestinationTIFF load] calls [CGImageDestination registerDestinationClass: self];. I believe this might be UB, so should this call be moved to +initialize?
+ (void)load
{
  [CGImageDestination registerDestinationClass: self];
}

I changed those to +initialize and it now segfaults in objc_msgSend_fpret in a call to +[NSString load].

Sending a message to a superclass in +load should be fine. Sending one to any other class that’s defined in the same library should also be fine.
I wonder if somehow the selector for the message being sent is not resolved. That could result in an out of bounds access in the uninitialised stable.

I have changed +load to +initialize and disabled UTILoad(), which results in:
Program received signal SIGABRT, Aborted.
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140737260606656) at ./nptl/pthread_kill.c:44
44      ./nptl/pthread_kill.c: No such file or directory.
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737260606656) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140737260606656) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140737260606656, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff6c42476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff6c287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff79ccc77 in init_protocols () from /usr/GNUstep/Local/Library/Libraries/libobjc.so.4.6
#6  0x00007ffff79ccafa in objc_init_protocols () from /usr/GNUstep/Local/Library/Libraries/libobjc.so.4.6
#7  0x00007ffff79c648d in objc_load_class () from /usr/GNUstep/Local/Library/Libraries/libobjc.so.4.6
#8  0x00007ffff79cc201 in __objc_load () from /usr/GNUstep/Local/Library/Libraries/libobjc.so.4.6
#9  0x00007ffff7fc947e in call_init (l=<optimized out>, argc=argc@entry=1, argv=argv@entry=0x7fffffffd398, env=env@entry=0x7fffffffd3a8)
    at ./elf/dl-init.c:70
#10 0x00007ffff7fc9568 in call_init (env=0x7fffffffd3a8, argv=0x7fffffffd398, argc=1, l=<optimized out>) at ./elf/dl-init.c:33
#11 _dl_init (main_map=0x7ffff7ffe2e0, argc=1, argv=0x7fffffffd398, env=0x7fffffffd3a8) at ./elf/dl-init.c:117
#12 0x00007ffff7fe32ca in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#13 0x0000000000000001 in ?? ()
#14 0x00007fffffffd88b in ?? ()
--Type <RET> for more, q to quit, c to continue without paging--
#15 0x0000000000000000 in ?? ()

@davidchisnall Might this be related? The segfault appears in pthreads in objc_init_protocols().

Does Apple objc4 work on Linux? If it does, it might be possible to test on objc4 to see if the issue is in libobjc2.

I passed -DCMAKE_BUILD_TYPE=Debug to CMake

I'm kinda busy, but here's my last GDB session if anyone wants to take a look:
Program stopped.
0x00007ffff7fe3290 in _start () from /lib64/ld-linux-x86-64.so.2
(gdb) b protocol.c:224
No source file named protocol.c.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (protocol.c:224) pending.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /home/ethan/Projects/GNUstep/Porting/GitUp/Examples/GitY/GitY.app/GitY
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, init_protocols (protocols=0x7ffff5ecb020 <objc_protocol_list>) at /home/ethan/Projects/GNUstep/plaurent2/GNUstep-build/libobjc2/protocol.c:224
224                                     fprintf(stderr, "Unknown protocol version");
(gdb) p version
$1 = 0
(gdb) p aProto
$2 = (struct objc_protocol *) 0x7ffff5ed4a50 <._OBJC_PROTOCOL_CAAction>
(gdb) p *aProto
$3 = {isa = 0x0, name = 0x7ffff5ebdfa8 "CAAction", protocol_list = 0x7ffff5ed4b50 <objc_protocol_list>,
  instance_methods = 0x7ffff5ecad40 <objc_protocol_method_list>, class_methods = 0x7ffff5ecad60 <objc_protocol_method_list>,
  optional_instance_methods = 0x7ffff5ecad58 <objc_protocol_method_list>, optional_class_methods = 0x7ffff5ecad68 <objc_protocol_method_list>,
  properties = 0x0, optional_properties = 0x0, class_properties = 0x0, optional_class_properties = 0x0}
(gdb)

@davidchisnall The issue appears to be that the isa of CAAction is 0x0, which is not a valid protocol version. I don't know why it ended up this way.

What is the -fobjc-runtime= flag that you’re passing to clang?

@davidchisnall gnustep-config says that I'm using -fobjc-runtime=2.1:
$ gnustep-config --gui-libs
-fuse-ld=/usr/bin/ld.gold -L/usr/local/lib -pthread -fexceptions -rdynamic -fobjc-runtime=gnustep-2.1 -fblocks -L/home/ethan/GNUstep/Library/Libraries -L/usr/GNUstep/Local/Library/Libraries -L/usr/GNUstep/System/Library/Libraries -lgnustep-gui -lgnustep-base -lpthread -lobjc -lm

I can try switching to the newly released v2.2.1

I'm building like:
export CC=clang-14
export CXX=clang++-14
export CXXFLAGS="-std=c++11"
export RUNTIME_VERSION=gnustep-2.1
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
export LD=/usr/bin/ld.gold
export LDFLAGS="-fuse-ld=/usr/bin/ld.gold -L/usr/local/lib"

rm -Rf build
mkdir build && cd build
cmake ../ \
  -DCMAKE_C_COMPILER=${CC} \
  -DCMAKE_CXX_COMPILER=${CXX} \
  -DCMAKE_ASM_COMPILER=${CC} \
  -DCMAKE_LINKER=${LD} \
  -DUSE_GOLD_LINKER=YES \
  -DCMAKE_MODULE_LINKER_FLAGS="${LDFLAGS}" \
  -DTESTS=OFF
cmake --build .
sudo -E make install
sudo ldconfig

The issue still appears in v2.2.1.

clang 14 is pretty old, but should be fine.  There are a few fixes for the v2 ABI after that, but I think they matter only on Windows.
I wonder if the +load is somehow being called before we've loaded the Protocol class.  The Protocol class should be linked into the runtime and the runtime's own constructors should be called before anything that links to it, but it's possible that this isn't happening on your platform.  That shouldn't be the case, because if init_protocol_classes returns NO then we don't reach the link that's failing for you.

@davidchisnall Is there anything that I should run in gdb to look into your suspicions? Also, is there anyone else we should get to look at this? Should I post this to discuss-gnustep?

I wonder if the +load is somehow being called before we've loaded the Protocol class.
I think this issue is unrelated to the +load issue. Here's the backtrace from protocol.c:224:

#0  init_protocols (protocols=0x7ffff5ecb020 <objc_protocol_list>) at /home/ethan/Projects/GNUstep/plaurent2/GNUstep-build/libobjc2/protocol.c:224
#1  0x00007ffff79ccafa in objc_init_protocols (protocols=0x7ffff726c238 <objc_protocol_list>)
    at /home/ethan/Projects/GNUstep/plaurent2/GNUstep-build/libobjc2/protocol.c:271
#2  0x00007ffff79c648d in objc_load_class (class=0x7ffff726c300 <._OBJC_CLASS_GCGitObject>)
    at /home/ethan/Projects/GNUstep/plaurent2/GNUstep-build/libobjc2/class_table.c:465
#3  0x00007ffff79cc201 in __objc_load (init=0x7ffff7266c58 <objc_init>) at /home/ethan/Projects/GNUstep/plaurent2/GNUstep-build/libobjc2/loader.c:268
#4  0x00007ffff7fc947e in call_init (l=<optimized out>, argc=argc@entry=1, argv=argv@entry=0x7fffffffd398, env=env@entry=0x7fffffffd3a8)
    at ./elf/dl-init.c:70
#5  0x00007ffff7fc9568 in call_init (env=0x7fffffffd3a8, argv=0x7fffffffd398, argc=1, l=<optimized out>) at ./elf/dl-init.c:33
#6  _dl_init (main_map=0x7ffff7ffe2e0, argc=1, argv=0x7fffffffd398, env=0x7fffffffd3a8) at ./elf/dl-init.c:117
#7  0x00007ffff7fe32ca in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#8  0x0000000000000001 in ?? ()
#9  0x00007fffffffd88b in ?? ()
#10 0x0000000000000000 in ?? ()

It runs into NSCopying (isa 0x4) and NSCoding (isa 0x4) with no issues before reaching CAAction (isa 0x0).

@davidchisnall Are you available to look at this more or not? Should I post this to discuss-gnustep?

I’m completely confused by this. It looks as if your binary contains things that shouldn’t be possible.

I have tested again with a new installation of GNUstep on the same computer with clang-18, which results in the same segfault in SparseArrayLookup. I'll go apply my patches to Opal and Boron and see what happens.

I should also try to do this on a clean Debian system, but I currently don't have the disk space to do that.

By the way, for most repositories that I haven't patched I'm on the latest commit to master as of May 31.

Is there something unusual about CAAction's protocol definition?  (Is it missing a definition?)
I can't usefully help debug this without a test case that doesn't depend on a big pile of other libraries.
I hadn't noticed previously that you were using gold.  Does the same thing happen if you use lld?

Is there something unusual about CAAction's protocol definition? (Is it missing a definition?)
No, it looks perfectly normal.

@protocol CAAction
@required
- (void)runActionForKey:(NSString *)key object:(id)anObject arguments:(NSDictionary *)dict;
@end

I can't usefully help debug this without a test case that doesn't depend on a big pile of other libraries.
I'll try to make a test case that only depends on QuartzCore (which contains CAAction) and its dependencies.
I hadn't noticed previously that you were using gold. Does the same thing happen if you use lld?
I'll try to use lld.

Using ld.lld-18, I get during make of the example binary:
/usr/bin/ld: /usr/GNUstep/Local/Library/Libraries/libGitUpKit.so: undefined reference to `CGContextStrokeEllipseInRect'
/usr/bin/ld: /usr/GNUstep/Local/Library/Libraries/libGitUpKit.so: undefined reference to `CGContextFillEllipseInRect'

This might be related to the issue, but I'm a bit concerned that these error messages come from ld instead of ld.lld-18. Is that normal?

Fixing that issue with
void CGContextFillEllipseInRect(CGContextRef ctx, CGRect rect)
{
  NSWarnLog(@"CGContextFillEllipseInRect is not implemented");
}

void CGContextStrokeEllipseInRect(CGContextRef ctx, CGRect rect)
{
  NSWarnLog(@"CGContextStrokeEllipseInRect is not implemented");
}

gives me Unknown protocol versionaborted again.

The issue may be that QuartzCore doesn't seem to link to the libraries it uses, including Opal and OpenGL. Here's a smaller example I tried to make, but it doesn't run into the CAAction 0x0 issue: https://github.com/ethanc8/CAActionPlayground

(for my reference)
export CC=clang-18
export CXX=clang++-18
export CXXFLAGS="-std=c++11"
export RUNTIME_VERSION=gnustep-2.1
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
export LD=/usr/bin/ld.lld-18
export LDFLAGS="-fuse-ld=/usr/bin/ld.lld-18 -L/usr/local/lib"

rm -Rf build
mkdir build && cd build
cmake ../ \
  -DCMAKE_C_COMPILER=${CC} \
  -DCMAKE_CXX_COMPILER=${CXX} \
  -DCMAKE_ASM_COMPILER=${CC} \
  -DCMAKE_LINKER=${LD} \
  -DUSE_GOLD_LINKER=YES \
  -DCMAKE_MODULE_LINKER_FLAGS="${LDFLAGS}" \
  -DTESTS=OFF \
  -DCMAKE_BUILD_TYPE=Debug \
  -DCMAKE_EXPORT_COMPILE_COMMANDS=1
cmake --build .
sudo -E make install
sudo ldconfig

Huh, init_protocols never finds CAAction.

Ok, by adding a category that implements CAAction, I got it to process
(gdb)
$3 = {isa = 0x4, name = 0x55555555705a "CAAction", protocol_list = 0x555555559500 <objc_protocol_list>,
  instance_methods = 0x5555555592c0 <objc_protocol_method_list>, class_methods = 0x5555555592e0 <objc_protocol_method_list>,
  optional_instance_methods = 0x5555555592d8 <objc_protocol_method_list>, optional_class_methods = 0x5555555592e8 <objc_protocol_method_list>, properties = 0x0,
  optional_properties = 0x0, class_properties = 0x0, optional_class_properties = 0x0}

Now the isa is 0x4.
With only a class implementing CAAction, however, init_protocols never processes CAAction.

Compare the GitY example:
(gdb)
$3 = {isa = 0x0, name = 0x7ffff6a9e466 "CAAction", protocol_list = 0x7ffff6ab3b60 <objc_protocol_list>,
  instance_methods = 0x7ffff6aa9d50 <objc_protocol_method_list>, class_methods = 0x7ffff6aa9d70 <objc_protocol_method_list>,
  optional_instance_methods = 0x7ffff6aa9d68 <objc_protocol_method_list>, optional_class_methods = 0x7ffff6aa9d78 <objc_protocol_method_list>, properties = 0x0,
  optional_properties = 0x0, class_properties = 0x0, optional_class_properties = 0x0}

As I recall, protocols must be adopted by at least one class or category to exist. I’m a bit surprised CAAction isn’t used by anything in Opal though.

There’s probably a compiler bug that is causing a non-adopted protocol to be emitted, but I’m not sure how it’s generated but not in the correct section. Can you send me the preprocessed source for the compilation unit that generates the .o file that contains the symbol for this protocol?

There’s probably a compiler bug that is causing a non-adopted protocol to be emitted, but I’m not sure how it’s generated but not in the correct section. Can you send me the preprocessed source for the compilation unit that generates the .o file that contains the symbol for this protocol?

Do you know how to find the associated compilation unit?

nm will dump the symbols for each .o file, you should be able to find the CAAction protocol in that list for one file.  Once you know that, find the .m file that generated it and add -E  and -o {some temp file} to the compile command.

Ok, thanks.

I found<PHONE_NUMBER>000000 D ._OBJC_PROTOCOL_CAAction in CAAnimation.m.o.
I'll attach the output of -E at https://gist.github.com/ethanc8/46be627ad36326f9b32843fe6409d77d. I wanted to also try -rewrite-objc -fno-ms-extensions -fpermissive, but it looks like libs-quartzcore has a lot of obviously incorrect code (like assigning BOOLs to variables of type id) that only works in C, and not in C++. -rewrite-legacy-objc -fno-ms-extensions -fpermissive caused a compiler crash (which I reported at https://github.com/llvm/llvm-project/issues/94380).

(for myself)
for file in *.o; do echo $file; nm $file; done | grep CAAction

This is especially perplexing because nothing in libs-quartzcore implements the protocol CAAction, there are just methods that take and return CAAction values (returned values are ones that the user previously provided to libs-quartzcore, so libs-quartzcore is not producing any id itself). Despite that, CAAnimation ends up with ._OBJC_PROTOCOL_CAAction, so maybe it gets it because it's the first file to be compiled that imports CAAction.h?

Does something in that file do @protocol(CAAction)?  Can you attack the preprocessed source for CAAnimation.m and I'll take a look.

Here is the preprocessed source (I posted it above, but might not have been clear): https://gist.github.com/ethanc8/46be627ad36326f9b32843fe6409d77d
The word CAAction appears nowhere in the un-preprocessed source.

Wait, I didn't see this:
@interface CAAnimation : NSObject <NSCoding, NSCopying, CAAction, CAMediaTiming>

Yup, looks like the protocol is adopted, so it is correct for it to be emitted in this file.  But for some reason it's not initialised correctly.  Looks like a compiler bug.  If you can create a reduced test case that would be helpful (delete bits of the file, compile with -S -emit-llvm and see if the resulting file has 0 for the first field of _OBJC_PROTOCOL_CAAction), otherwise I'll try to look at it later in the week.

Ok, thanks. I'm not sure whether it's a compiler bug, since it seems to be initialised correctly for my test case which creates classes and categories implementing CAAction and instantiates CAAnimation.

The original CAAnimation.m gets CAAction compiled to:
$._OBJC_PROTOCOL_CAAction = comdat any
// a few hundred lines later
@52 = private unnamed_addr constant [9 x i8] c"CAAction\00", align 1
@._OBJC_PROTOCOL_CAAction = global { ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr } { ptr inttoptr (i32 4 to ptr), ptr @52, ptr @.objc_protocol_list.10, ptr @.objc_protocol_method_list.11, ptr @.objc_protocol_method_list.13, ptr @.objc_protocol_method_list.12, ptr @.objc_protocol_method_list.14, ptr null, ptr null, ptr null, ptr null }, section "__objc_protocols", comdat, align 8
// a few lines later
@52 = private unnamed_addr constant [9 x i8] c"CAAction\00", align 1
@._OBJC_PROTOCOL_CAAction = global { ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr, ptr } { ptr inttoptr (i32 4 to ptr), ptr @52, ptr @.objc_protocol_list.10, ptr @.objc_protocol_method_list.11, ptr @.objc_protocol_method_list.13, ptr @.objc_protocol_method_list.12, ptr @.objc_protocol_method_list.14, ptr null, ptr null, ptr null, ptr null }, section "__objc_protocols", comdat, align 8

I've attached the full LLVM IR to https://gist.github.com/ethanc8/d02ef13441e1eb58d372cf650fa3f63b

That looks correct.  The isa pointer is that ptr inttoptr (i32 4 to ptr) one: we're initialising it to 4 in the compiler, so it should be a GNUstepV2 protocol.  So now the question is: where does that become 0?  Can you put a watchpoint on the load address and see if there are any writes to it?

Can you put a watchpoint on the load address and see if there are any writes to it?
Which load address are you talking about?

Ok, I set a watchpoint at 0x7ffff6ab3a60 (the address of CAAction).
Hardware watchpoint 1: *(int*)0x7ffff6ab3a60

Old value = 4
New value = 0
registerProtocol (proto=0x7ffff6ab3a60 <._OBJC_PROTOCOL_CAAction>) at /home/ethan/Projects/GNUstep/plaurent2/gnustep/libobjc2/protocol.c:608
608             if (protocol_for_name(proto->name) == NULL)
(gdb)

If I try to forcefully reset it back to 4, it does:
set var proto->isa=0x4
(gdb) continue
Continuing.

Hardware watchpoint 1: *(int*)0x7ffff6ab3a60

Old value = 4
New value = -143207376
init_protocols (protocols=0x7ffff6aaa030 <objc_protocol_list>) at /home/ethan/Projects/GNUstep/plaurent2/gnustep/libobjc2/protocol.c:245
245                     if (NULL != aProto->protocol_list)
(gdb)

and then fails with Unknown protocol version.

Okay, this looks like a load-order problem. The line above the one in your gdb session is proto->isa = protocol_class_gsv2;. If we haven’t yet loaded the protocol classes, this will be null, and that seems to be what you’re seeing.
Most of the paths that should be here are guarded on having loaded the runtime. Can you print a backtrace there? I suspect the simplest thing to do is treat the built-in classes as special and define them in C rather than relying on the normal load sequence.
Does the library defining this protocol explicitly link libobjc2? If it does, we should see the libobjc2 constructors run first (I think).

All of the libraries and apps should be linking libobjc2:

$ gnustep-config --objc-libs
-ldispatch -pthread -fexceptions -rdynamic -fobjc-runtime=gnustep-2.1 -fblocks -L/home/ethan/GNUstep/Library/Libraries -L/usr/GNUstep/Local/Library/Libraries -L/usr/GNUstep/System/Library/Libraries -lpthread -lobjc -lm

Please can you test #294 and see if this fixes it for you?

Please can you test #294 and see if this fixes it for you?

I will try to do so tonight, but I'm a bit busy today.

Please can you test #294 and see if this fixes it for you?

Yes, it seems to be working now. Thanks! Do you want me to confirm anything in the debugger for you? I have just noticed that it no longer aborts with "Unknown protocol version".

No, that’s great. Thanks.

We still need to figure out what caused the segfault in SparseArrayLookup.


Graph-based matching for optimal (and correct) reference to estimate event matching

Just found out about this library, great work! (was about to start writing basically the same but decided to have one final look to see if there's anything out there already - glad I found it).
For the event-based metrics the evaluate function matches events by iterating over all reference and estimate events in a nested loop. I have two questions about this:

Doesn't this mean that the same estimated event can be matched against more than one reference event? I'm not sure that's a desired behavior - presumably every reference event should be matched again at most one estimated event and vice versa?
Assuming the goal of (1) is indeed to match every reference event against at most one estimated event (and vice versa), then using a nested loop means the matching is greedy and not necessarily optimal (i.e. it might not find the optimal pairing of matching events) leading to an underestimation of performance. This is exactly the case for note transcription (which is basically the same problem: matching pairs of events based on onset, offset and label (pitch) criteria). To get around this in mir_eval we match notes using bi-partite graph matching, which guarantees to find the optimal pairing of reference and estimated events. The same approach is also used everywhere else in mir_eval where to sets of events needs to be matched into pairs.

Let me know what you think, cheers!

Thanks for bringing these issues up.
Yes, you are correct with first issue. The intention in this nested loop is to find correctly detected events, however, only one correctly detected event per reference event is accepted. The rest will be counted as substitutions.
On the other hand, the code did not implement this idea fully. I fixed the code now (commits 857a0adcfcaec5174e84cc662abfb6f82a81a398 and 686d1603d5fe0ce022572d3a02ce15203ae0526a) so that  estimated event marked already matched cannot be matched again. On rather small test dataset (two files from DCASE2013), this fix did not change much ER metric numbers.
As for the second issue, with current metrics I do not see the need to find best matched event pairs. Since we are only counting whether an event is found or not, if multiple events match within the required conditions, one will be correct and the others counted as insertions (or substitutions), no matter which is best match. In music the pitch tolerance is something you also care for, but in our case that is fixed, and only temporal position matters. We may consider optimal matching later, if we move to some temporal overlap related metrics.

On the other hand, the code did not implement this idea fully. I fixed the code now (commits 857a0ad and 686d160) so that estimated event marked already matched cannot be matched again.

👍

On rather small test dataset (two files from DCASE2013), this fix did not change much ER metric numbers.

But on a larger set it might, right?

As for the second issue, with current metrics I do not see the need to find best matched event pairs. Since we are only counting whether an event is found or not, if multiple events match within the required conditions, one will be correct and the others counted as insertions (or substitutions), no matter which is best match.

Consider the following case (somewhat contrived, but possible):
Reference =
1, 3, carhorn
1.5, 3, carhorn
Estimate =
1.25, 3, carhorn
1.75, 3, carhorn
The first estimated event matches both reference events (assuming a tolerance of .25), however the second estimated event only matches the second reference event. With greedy matching you might (if you're unlucky) match the first estimated event to the second reference event, and then the other pair (the second estimated event vs the first reference event) won't match. With graph matching you're guaranteed the first estimated event will be paired with the first reference event and the second with the second.
I doubt the metrics would change dramatically, but it's worth considering.

As for the second issue, with current metrics I do not see the need to find best matched event pairs. Since we are only counting whether an event is found or not, if multiple events match within the required conditions, one will be correct and the others counted as insertions (or substitutions), no matter which is best match.

Sorry to jump in here a year and change late, but maybe I can provide some context.  We discuss this broadly in section 4.1 of the mir_eval paper.
The reason to care about finding an optimal matching -- note "an" not "the", as they need not be unique -- is for reproducibility.  Greedy matching will generally be sensitive to the order in which observations are presented, and the same set of observations presented in different orders could produce different scores.
In @justinsalamon's example above, if the estimates were presented in reverse order, it would match (1.75, 1.5) first, leaving (1.25, 1) as a second match (2 hits).  In the order given, it could match (1.25, 1) leaving (1.75, 1) which would not be a match.  Note that this is phrased here in terms of time "tolerance", not pitch tolerance.  When we implemented this for onset detection and beat tracking, we observed a small, but consistent increase in the resulting scores, indicating that this phenomenon does occur in real data, and previous implementations were under-reporting accuracy metrics.

After a long time, I managed to find time to address this issue. Graph based event matching is added with commits 13cef668f55f0ea7a47cb6b8c719ac86db75b708 and f4726a603601758413f3b0317c94711036656742, and is available in the release version 0.2.1.
Previously used greedy matching approach is still available though optional parameter event_matching_type='greedy' for backward compatibility. The greedy matching approach was implemented such a way that it was sensitive to the order of events in the reference event list. This could produce slightly different metric scores, if event order is changed in cases with events having onsets very close each other (distance smaller than t_collar) and assigned to same event label.


[HomeKit] RGB Light color wheel not working properly only on iPhone

Home Assistant release with the issue:
0.94.2
Last working Home Assistant release (if known):
N/A
Operating environment (Hass.io/Docker/Windows/etc.):
Hass.io 2.12 on a VM running on Windows 10
Component/platform:
https://www.home-assistant.io/components/homekit/
Connection path:
Philips Hue RGB Light >Abode > Abode component > HA > Homekit
Working points:
Abode OK.
Home Assistant lovelace OK.
Behavior with Homekit:
Watch OS 5.1.2:
Home app OK
Siri OK
iOS 12.3.2:
Home app FAIL
Siri OK
iOS Home App:
When selecting any new color via color wheel or a precise one the color changes for a moment but rolls back immediately to a white ambiance one, different that the one previous the request. This only happens on iOS through Home App and the issue persist on 2 different iPhones (other one on iOS 12.2).
When making this changes via iPhone's Siri the light changes and stays on requested color without problems.
Also, on the Apple Watch changing color with the wheel (Home App) or with Siri works without any issue.
The problem seems to be only on Home App on iOS, it seems odd.
Please let me know if I can help in any other way 😀
Traceback (if applicable):
No errors showing
Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):
N/A
Additional information:
I found out issue #24094 but I'm not sure if it is the same root cause so proceeded to create a new one.

I can confirm almost the same issue but with an different connection path:
Mine is:
Philips Hue RGB Light > Hue Bridge > Hue Component > HA > Homekit (AppleTV as hub)
The color wheel has been the broken for a while on my iPhone but I THINK that my wifes iPhone got this issue more recently. It's sometime possible to trick the wheel by dragging the selector outside the spectrum and holding there for a while or flipping mode between "Color" and "Temperature" a few times ( or is it just timing?).
The same behaviour seems to happen when using the Home.app on macOS but Siri on all devices is working.

This seems to be the same issue as https://github.com/home-assistant/home-assistant/issues/23394

This issue is due to the fact that in the backend of the Homekit integration : Home Assistant setup RGB light with both RGB AND Temperatures color properties and this end up with glitchy behavior on the Home.app on iOS and macOS. Those 2 characteristics must be mutually exclusive according to my testing.
I've already posted this here a while ago but nothing has been made since then.
The only workaround to this issue that I've found is to setup my own bridge and setup a colored lightbulb with the properties that doesn't make HomeKit to freakout.

The only workaround to this issue that I've found is to setup my own bridge and setup a colored lightbulb with the properties that doesn't make HomeKit to freakout.

How did you do this?
I recently switched to DeConz for all my lights and then hit this issue after previously going through the Philips Hue Bridge. I posted on the DeConz github and ebauuw kindly gave this advice, if its of any use...

I hate to be fingerpointing, but it looks like an issue with HA. Hue lights report back the closest ct value when setting a light colour through xy. It seems like HA uses that as a hint to update the colour temperature, setting ct. Then, the Hue light reports back the closest xy value on setting ct, which HA takes as a hint to update xy, etc. Basically HA should check colormode and not take action on changed ct values unless colormode is ct (and similar for xy).

To confirm, you could try and set a yellow-white-ish colour and see high ct values (up to 500) and a blue-white-ish colour to see low ct values (down to 153).

As far as I know, only Hue lights report back their actual state (rather than the last state set), and only Hue lights update the attributes for another colour mode, so you won't see this behaviour for other lights.

How did you do this?

I'm pretty sure my workaround isn't the best way to achieve the intended result here but here it is anyway
I'm running this docker Homebridge container with the homebridge-mqttthing plugin.
I'm setting up my lights like so in homebridge :
{
            "accessory": "mqttthing",
            "type": "lightbulb",
            "name": "DMX 1",
            "url": "http://<IP_ADDRESS>:1883",
            "username": "mqttuser",
            "password": "strongpassword",
            "topics": {
                "getOn": "homebridge/DMX/1/getOn",
                "setOn": "homebridge/DMX/1/setOn",
                "setRGB": "homebridge/DMX/1/setRGB",
                "getRGB": "homebridge/DMX/1/getRGB",
                "getOnline": "homebridge/DMX/1/getOnline"
            },
            "logMqtt": true,
            "integerValue": false,
            "onlineValue": "Online",
            "hex": false,
            "manufacturer": "LEDS-c4",
            "model": "GEA RGB DMX",
            "serialNumber": "0001"
}

And now I can control my RGB lights with iOS and macOS using the color wheel without any trouble.

Thanks Seb.
Do you know if you can run this on a RPi using Hassio?

Do you know if you can run this on a RPi using Hassio?

Unfortunately I can't help you with Hass.io since I'm not using it.
I'm running Homeassistant in a docker container. Sorry if I can't help you that much

I have exactly the same issue with homekit and Lifx on Hassio 0.98.1

On iOS13 the issue seems to be fixed. Closing for now. Thanks!

Not fixed for me on iOS 13.2


Memory leaks with resumes?

The following code works fine. But by changing small things I seem to get memory leaks and I dont know why. I would like to know if this is a me issue or Effekt issue does anyone know about this? I provide example memory usage after about 20s of running each code.
Working:
effect Dual(idx: Int): Array[Bool]

def main() = {
  with on[OutOfBounds].panic
  val listsize = 50
  var arr = array(listsize, false)
  try {
    each(0,listsize) { v =>
      arr = do Dual(v)
      println(v)
    }
  } with Dual { idx =>
    arr.set(idx, true)
    resume(arr.copy)
    arr.set(idx, false)
    resume(arr)
  }
}

Problem1:
If I either change it to this (add a counter to the effect):
effect Dual(idx: Int): Array[Bool]

def main() = {
  with on[OutOfBounds].panic
  var countFalse = 0
  val listsize = 50
  var arr = array(listsize, false)
  try {
    each(0,listsize) { v =>
      arr = do Dual(v)
      println(v)
    }
  } with Dual { idx =>
    arr.set(idx, true)
    resume(arr.copy)
    countFalse = countFalse + 1
    arr.set(idx, false)
    resume(arr)
    countFalse = countFalse - 1
  }
}

Problem2:
Or to this (exchange array with list):
effect Dual(idx: Int): List[Bool]

def main() = {
  with on[OutOfBounds].panic
  var countFalse = 0
  val listsize = 50
  var arr = fill(listsize, false)
  try {
    each(0,listsize) { v =>
      arr = do Dual(v)
      println(v)
    }
  } with Dual { idx =>
    resume(arr.updateAt(idx){_ => true})
    resume(arr.updateAt(idx){_ => false})
  }
}

Thanks for the issue! We often have issues with memory leaks, so I wouldn't be surprised.
The images aren't loading, so I'm not entirely sure what you're referring to (I can't reproduce abnormal memory usage for Problem1 in LLVM, for example). What backend are you using and how are you measuring the memory leaks?

Exchanged the images with new versions hope they work now. I use the JavaScript backend on Windows 10 and use the Task Manager to track the memory (as the images would show)

In general, we are not the most memory efficient language. However, in your example, you are creating $2^{n + 1} - 2$ many arrays of size $n$ (where n=50).
JS is garbage collected, so in general we cannot guarantee that the memory will be freed right-away.
It is also not very surprising that the list version allocates a lot more, since there we allocate roughly 50 linked objects.
The following will not make a lot of sense to you, but what we could investigate is whether the "shadow stack" of regions potentially holds onto the arrays for too long preventing GC to do its work.

I have to say, I like this as a benchmark though ;)
Here (for N=24) is JS (/usr/bin/time -l):
       11.20 real        10.98 user         0.12 sys
            77856768  maximum resident set size
                   0  average shared memory size
                   0  average unshared data size
                   0  average unshared stack size
                5279  page reclaims
                   1  page faults
                   0  swaps
                   0  block input operations
                   0  block output operations
                   0  messages sent
                   0  messages received
                   0  signals received
                   0  voluntary context switches
               17846  involuntary context switches
<PHONE_NUMBER>22  instructions retired
<PHONE_NUMBER>7  cycles elapsed
            60757632  peak memory footprint

and llvm:
        4.95 real         4.43 user         0.51 sys
<PHONE_NUMBER>  maximum resident set size
                   0  average shared memory size
                   0  average unshared data size
                   0  average unshared stack size
              433213  page reclaims
                   4  page faults
                   0  swaps
                   0  block input operations
                   0  block output operations
                   0  messages sent
                   0  messages received
                   0  signals received
                   0  voluntary context switches
                  37  involuntary context switches
<PHONE_NUMBER>6  instructions retired
<PHONE_NUMBER>8  cycles elapsed
<PHONE_NUMBER>  peak memory footprint

Here LLVM uses 5,6 GB peak memory, while JS is around 58 MB! (cc @phischu). Nevertheless, it is still twice as fast.

In general, we are not the most memory efficient language. However, in your example, you are creating 2 n + 1 − 2 many arrays of size n (where n=50).
So, assuming something like 500 bytes per array, this is roughly 1000 Petabytes.
JS is garbage collected, so in general we cannot guarantee that the memory will be freed right-away. However, given that the program terminates on your computer says a lot about it not leaking too much memory ;)
It is also not very surprising that the list version allocates a lot more, since there we allocate roughly 50 linked objects.

Sorry I was not clear enough. The programm did not terminate for me, but that is not the important part. What I wanted to demonstrate is that adding a counter or changing an array to a list will cause a memory leak. While the example without them rarely uses more than 50MB of RAM the other ones quickly run out of memory when the println is removed.
I tested a bit with the counting. When running the following code we get an every increasing memory usage (task manager screenshots after running for 20s or Out of memory, what happens first):

effect Dual(idx: Int): Array[Bool]

def main() = {
  with on[OutOfBounds].panic
  var count = 0
  val listsize = 50
  var arr = array(listsize, false)
  try {
    each(0,listsize) { v =>
      arr = do Dual(v)
    }
  } with Dual { idx =>
    arr.set(idx, true)
    resume(arr.copy)
    arr.set(idx, false)
    resume(arr)
    count = count + 1
  }
}

If the count is moved outside (directly before the effect), which does exactly the same thing as far as I understand, the memory usage increases slowly or not at all (Only show try with as the rest stays the same):

try {
each(0,listsize) { v =>
  count = count + 1
  arr = do Dual(v)
 }
} with Dual { idx =>
  arr.set(idx, true)
  resume(arr.copy)
  arr.set(idx, false)
  resume(arr)
}

It seems the counting at the end of the effect has some weird behaviour. When the count is increased before the end of the effect this issue also does not occur:

try {
  each(0,listsize) { v =>
    arr = do Dual(v)
  }
} with Dual { idx =>
  arr.set(idx, true)
  resume(arr.copy)
  arr.set(idx, false)
  count = count + 1
  resume(arr)
}

I haven't found anything new about the lists but I don't think the behaviour is intended as at a moment no more than 50 lists should be required to track a branch of the resumes which are resolved one after another. Maybe the other lists are not freed for garbage collection even though they are not used anymore? It shouldn't use much more memory than the arrays solution as it also creates a similar amount of arrays (also tried copying the array in both resumes with similar results).

Thank you for minifying these examples. The expected result if you would print count at the end is 2^50, right?
  } with Dual { idx =>
    arr.set(idx, true)
    resume(arr.copy)
    arr.set(idx, false)
    resume(arr)
    count = count + 1
  }

In a handler like this, the increments will only be done after the very last resume has finished. Operationally, we remember that we will have to increment the count as a stack frame. This requires a stack of size 2^50.

Wouldn't the following code then have the same issue? I reduced the iterations to 26 then code without memory issues finishes on my PC and the other fails due to OOM.
effect Dual(): Unit

def main() = {
  var count = 0
  try {
    each(0,26) { _ =>
      do Dual()
    }
  } with Dual {
    resume(())
    resume(())
    count = count + 1
  }
}

I removed the first set as it does not have an effect on the issue. I think I pinned the issue down to these two lines, when both lines are there the memory issues exist otherwise its fine.
effect Dual(idx: Int): Array[Bool]

def main() = {
  with on[OutOfBounds].panic
  var count = 0
  val listsize = 26
  var arr = array(listsize, false)
  try {
    each(0,listsize) { v =>
      arr = do Dual(v)
    }
  } with Dual { idx =>
    resume(arr.copy)
    arr.set(idx, false) // if this is removed there is no issue
    resume(arr)
    count = count + 1 // same for this (only one has to be removed)
  }
}


feat(webhook): Cron trigger for updating Zapier [DEV-1481]

https://app.clickup.com/t/6600954/DEV-1481

Fixed all (removed ky) @ankurdotb.


DropdownMenu does not correctly handle incoming maxWidth and minHeight constraints

Consider the following demo:
import 'package:flutter/material.dart';

void main() => runApp(const MyApp());

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      home: Container(
        alignment: Alignment.center,
        color: Colors.amber,
        child: SizedBox(
          height: 100.0,
          width: 100.0,
          child: Card(
            child: DropdownMenu(
              dropdownMenuEntries: [
                DropdownMenuEntry(
                  value: '',
                  label: 'This is a very long label',
                ),
              ],
            ),
          ),
        ),
      ),
    );
  }
}

It should display a 100x100 widget, or at least a widget not wider than 100 pixels, whose menu appears below it.
Instead it displays a widget that is wider than the maxWidth (painting outside of its allowed bounds), and the menu renders below the minHeight, not under the actual widget:

The width aspect of this in particular is quite serious as it makes it impossible to use this widget with user-controlled text without risking overflow. You can't even set a labelWidget that makes the text wrap or truncate, because the labels don't receive the constraints.

Reproducible using the code sample provided above.

flutter doctor -v
[!] Flutter (Channel stable, 3.19.6, on macOS 14.4.1 23E224 darwin-arm64, locale en-GB)
    • Flutter version 3.19.6 on channel stable at /Users/nexus/dev/sdks/flutter
    ! Warning: `flutter` on your path resolves to /Users/nexus/dev/sdks/flutters/bin/flutter, which is not inside your current Flutter SDK checkout at /Users/nexus/dev/sdks/flutter. Consider adding /Users/nexus/dev/sdks/flutter/bin to the front of your path.
    ! Warning: `dart` on your path resolves to /Users/nexus/dev/sdks/flutters/bin/dart, which is not inside your current Flutter SDK checkout at /Users/nexus/dev/sdks/flutter. Consider adding /Users/nexus/dev/sdks/flutter/bin to the front of your path.
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 54e66469a9 (4 days ago), 2024-04-17 13:08:03 -0700
    • Engine revision c4cd48e186
    • Dart version 3.3.4
    • DevTools version 2.31.1
    • If those were intentional, you can disregard the above warnings; however it is recommended to use "git" directly to perform update checks and upgrades.

[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
    • Android SDK at /Users/nexus/Library/Android/sdk
    • Platform android-34, build-tools 34.0.0
    • Java binary at: /Users/nexus/Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.7+0-17.0.7b1000.6-10550314)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 15.3)
    • Xcode at /Applications/Xcode-15.3.0.app/Contents/Developer
    • Build 15E204a
    • CocoaPods version 1.15.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2023.1)
    • Android Studio at /Users/nexus/Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.7+0-17.0.7b1000.6-10550314)

[✓] IntelliJ IDEA Ultimate Edition (version 2023.2.5)
    • IntelliJ at /Users/nexus/Applications/IntelliJ IDEA Ultimate.app
    • Flutter plugin version 77.2.2
    • Dart plugin version 232.10286

[✓] VS Code (version 1.88.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.86.0

[✓] Connected device (4 available)
    • Nexus (mobile)       • 00008020-001875E83A38002E • ios            • iOS 17.4.1 21E236
    • Dean’s iPad (mobile) • 00008103-000825C811E3401E • ios            • iOS 17.4.1 21E236
    • macOS (desktop)      • macos                     • darwin-arm64   • macOS 14.4.1 23E224 darwin-arm64
    • Chrome (web)         • chrome                    • web-javascript • Google Chrome 124.0.6367.60

[✓] Network resources
    • All expected network resources are available.

! Doctor found issues in 1 category.

[✓] Flutter (Channel master, 3.22.0-14.0.pre.67, on macOS 14.4.1 23E224 darwin-arm64, locale en-GB)
    • Flutter version 3.22.0-14.0.pre.67 on channel master at /Users/nexus/dev/sdks/flutters
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 1a905d508d (20 hours ago), 2024-04-21 06:44:23 -0400
    • Engine revision 75ca2195c9
    • Dart version 3.5.0 (build 3.5.0-83.0.dev)
    • DevTools version 2.35.0-dev.8

[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
    • Android SDK at /Users/nexus/Library/Android/sdk
    • Platform android-34, build-tools 34.0.0
    • Java binary at: /Users/nexus/Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.7+0-17.0.7b1000.6-10550314)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 15.3)
    • Xcode at /Applications/Xcode-15.3.0.app/Contents/Developer
    • Build 15E204a
    • CocoaPods version 1.15.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2023.1)
    • Android Studio at /Users/nexus/Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.7+0-17.0.7b1000.6-10550314)

[✓] IntelliJ IDEA Ultimate Edition (version 2023.2.5)
    • IntelliJ at /Users/nexus/Applications/IntelliJ IDEA Ultimate.app
    • Flutter plugin version 77.2.2
    • Dart plugin version 232.10286

[✓] VS Code (version 1.88.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.86.0

[✓] Connected device (5 available)
    • Nexus (mobile)                  • 00008020-001875E83A38002E • ios            • iOS 17.4.1 21E236
    • Dean’s iPad (mobile)            • 00008103-000825C811E3401E • ios            • iOS 17.4.1 21E236
    • macOS (desktop)                 • macos                     • darwin-arm64   • macOS 14.4.1 23E224 darwin-arm64
    • Mac Designed for iPad (desktop) • mac-designed-for-ipad     • darwin         • macOS 14.4.1 23E224 darwin-arm64
    • Chrome (web)                    • chrome                    • web-javascript • Google Chrome 124.0.6367.60

[✓] Network resources
    • All expected network resources are available.

• No issues found!

For the width, it seems to be cause by the textfield not constraint by the widget constraint here link
child.layout(constraints.enforce(innerConstraints),parentUsesSize: true); would work
But I don't think I should threat children differently base on order.

@PurplePolyhedron Great! Looks like you found one root cause of this issue which seems to be getMaxIntrinsicWidth being called with width as a parameter instead of height and similar problem with getMaxIntrinsicHeight. Are you interested in filing a PR? I assigned you for the moment and I can help to review/write test.

Thanks, I will try to file a PR soon.

@bleroux I've filed an PR and written some tests. Let me know if it need more tests or any changes.


df/nep_examples

This contains the examples for NEP C-diamond and graphite as well as the TDEP methods I developed, do not merge the TDEP changes, just example changes.

@dafolkner do we want to close this PR or merge?


sbt tasks do not use scope specific settings

Hello,
Firstly, thanks for this excellent library. I am really enjoying the compactness and direct SQL-access.
I am trying to use the sbt plugin with a play application. I would like to use different databases for dev and test. I have overridden the flywayUrl in the Test scope. But the test:flywayMigrate task still uses the flywayUrl setting value and not the test:flywayUrl setting value.
I have a sample app - https://github.com/diwa/flyway_play_test to reproduce the problem
The following is the configuration extract from build.sbt
flywaySettings

flywayUrl := "jdbc:h2:file:target/devbar"

flywayUser := "SA"

flywayLocations := Seq("filesystem:conf/db/migrations/main")

flywayUrl in Test := "jdbc:h2:file:target/testbar"

flywayLocations in Test := Seq("filesystem:conf/db/migrations/main", "filesystem:conf/db/migrations/test")

SBT session
Global settings are recognized
➜  flyway_play_test git:(master) sbt
[info] Loading project definition from /Users/diwa/code/sandbox/flyway_play_test/project
[info] Set current project to flyway_play_test (in build file:/Users/diwa/code/sandbox/flyway_play_test/)

[flyway_play_test] $ flywayUrl
[info] jdbc:h2:file:target/devbar
[flyway_play_test] $ flywayLocations
[info] List(filesystem:conf/db/migrations/main)
[flyway_play_test] $ flywayInfo
[info] Database: jdbc:h2:file:target/devbar (H2 1.3)
[info] +----------------+----------------------------+---------------------+---------+
[info] | Version        | Description                | Installed on        | State   |
[info] +----------------+----------------------------+---------------------+---------+
[info] | 1              | Create person table        |                     | Pending |
[info] +----------------+----------------------------+---------------------+---------+
[success] Total time: 1 s, completed May 25, 2014 10:24:01 PM

Test Scope settings are recognized
[flyway_play_test] $ test:flywayUrl
[info] jdbc:h2:file:target/testbar
[flyway_play_test] $ test:flywayLocations
[info] List(filesystem:conf/db/migrations/main, filesystem:conf/db/migrations/test)

flywayInfo Task does not use test scope settings (see database)
[flyway_play_test] $ test:flywayInfo
[info] Database: jdbc:h2:file:target/devbar (H2 1.3)
[info] +----------------+----------------------------+---------------------+---------+
[info] | Version        | Description                | Installed on        | State   |
[info] +----------------+----------------------------+---------------------+---------+
[info] | 1              | Create person table        |                     | Pending |
[info] +----------------+----------------------------+---------------------+---------+
[success] Total time: 0 s, completed May 25, 2014 10:24:25 PM

Normal migration works
[flyway_play_test] $ flywayMigrate
[info] Database: jdbc:h2:file:target/devbar (H2 1.3)
[info] Validated 1 migration (execution time 00:00.003s)
[info] Creating Metadata table: "PUBLIC"."schema_version"
[info] Current version of schema "PUBLIC": << Empty Schema >>
[info] Migrating schema "PUBLIC" to version 1
[info] Successfully applied 1 migration to schema "PUBLIC" (execution time 00:00.027s).
[success] Total time: 0 s, completed May 25, 2014 10:25:09 PM

Test Scope migration does not work (nothing found to migrate)
[flyway_play_test] $ test:flywayMigrate
[info] Database: jdbc:h2:file:target/devbar (H2 1.3)
[info] Validated 1 migration (execution time 00:00.003s)
[info] Current version of schema "PUBLIC": 1
[info] Schema "PUBLIC" is up to date. No migration necessary.
[success] Total time: 0 s, completed May 25, 2014 10:25:15 PM

Can you please help with this as it is essential to have different dbs for dev and test ?
@jsuereth - Am I doing something wrong in the sbt settings for scopes here ?
Thanks in advance for all your support.

+1

Fixed. Thanks @cdedie !

I'm not sure if I am doing this correctly, but can't figure out how to use a different scope.  With the IntegrationTest scope defined in Play I tried:
// build.sbt
lazy val api = Project.project.in(file("api")).
  // ...
  settings(flywaySettings: _*).
  settings(
    // ...
    flywayUrl in IntegrationTest := "jdbc:postgresql://localhost:5432/mydb_test"
    flywayUser in IntegrationTest := "user"
    flywayPassword in IntegrationTest := "pw"
  )

Then in sbt:
> api/it:flywayUrl
[info] jdbc:postgresql://localhost:5432/mydb_test
> api/it:flywayUser
[info] user
> api/it:flywayPassword
[info] pw
> api/it:flywayMigrate
// ...
[info] Flyway 3.2.1 by Boxfuse
[trace] Stack trace suppressed: run last api/*:flywayMigrate for the full output.
[error] (api/*:flywayMigrate) org.flywaydb.core.api.FlywayException: Unable to connect to the database. Configure the url, user and password!
[error] Total time: 3 s, completed 27-Jan-2016 1:33:08 PM

Sorry, I see https://github.com/flyway/flyway/pull/891 isn't in the current release.

Is there a snapshot version of flyway available with this fix in it somewhere?

+1 for an intermediate release to consume this.
I am the original reporter of the issue 19 months ago.

@diwalak I tried to compile the version in master, but it doesn't appear to fix this issue.
$ git clone<EMAIL_ADDRESS>$ cd flyway
$ git checkout -b scopefix 3ba94fb683745b0f1ada835c60d111fd579d57cd
$ mvn install -P-CommercialDBTest -P-CommandlinePlatformAssemblies -DskipTests=true
$ cd flyway-sbt
$ sbt publish-local

// plugins.sbt
resolvers += "Local Maven" at Path.userHome.asFile.toURI.toURL + ".m2/repository"
addSbtPlugin("org.flywaydb" % "flyway-sbt" % "0-SNAPSHOT", "0.13", "2.10")

// build.sbt
lazy val api = Project.project.in(file("api")).
  // ...
  enablePlugins(FlywayPlugin).
  configs(IntegrationTest).
  settings(Defaults.itSettings: _*).
  settings(
    // ...
    flywayUrl in IntegrationTest := "jdbc:postgresql://localhost:5432/mydb_test"
    flywayUser in IntegrationTest := "user"
    flywayPassword in IntegrationTest := "pw"
  ),
 (test in IntegrationTest) <<= (test in IntegrationTest) dependsOn flywayMigrate,

Unfortunately, I still get exactly the same error:
> api/it:flywayUrl
[info] jdbc:postgresql://localhost:5432/mydb_test
> api/it:flywayUser
[info] user
> api/it:flywayPassword
[info] pw
> api/it:flywayMigrate
//...
[info] Flyway 0-SNAPSHOT by Boxfuse
[error] (api/*:flywayMigrate) org.flywaydb.core.api.FlywayException: Unable to connect to the database. Configure the url, user and password!
[error] Total time: 26 s, completed 31-Jan-2016 3:10:55 PM

@axelfontaine & @cdedie Unless I'm doing something wrong, should this issue be re-opened?  I think the problem is not fixed.

I assume this is connected to this but both those are completely unintelligible to me.  Whatever's happening, flywayMigrate isn't reading from the right configuration...?

@cdedie Thoughts?

@axelfontaine  I can invest some time in trying to figure this out, but it will take me some time to get up-to-speed on how to write sbt plugins---unless @cdedie has a quick fix. The problem seems to be that Test is hardcoded.

@axelfontaine I believe that the current implementation in master is correct.  This appears to be the magic sbt incantation to execute tasks in a different Configuration scope:
// Copy the flywaySettings into the IntegrationTest configuration.
 settings(inConfig(IntegrationTest)(flywaySettings): _*).

With that set, I can run either it:flywayMigrate or flywayMigrate and they each find the variables from the correct scope.

What is the way in Flyway 4.0 to do this? By default, only default and Test configuration seem to be respected, and @mikebridge 's solution does not work since flywaySettings is no longer defined.
lazy val Production: Configuration = config("prod") extends Compile

flywayUrl in Test := "jdbc:postgresql://localhost:5432/test"
flywayUser in Test := "test"
flywayPassword in Test := ""

flywayUrl in Production := "jdbc:postgresql://localhost:5432/prod"
flywayUser in Production := "production"
flywayPassword in Production := ""

flywayUrl in IntegrationTest := "jdbc:postgresql://localhost:5432/it"
flywayUser in IntegrationTest := "it"
flywayPassword in IntegrationTest := ""

Results in:
$ sbt test:flywayInfo
[info] Flyway 4.0 by Boxfuse
[info] Database: jdbc:postgresql://localhost:5432/test (PostgreSQL 9.5)
[info] +---------+---------------------------------+---------------------+---------+
[info] | Version | Description                     | Installed on        | State   |
[info] +---------+---------------------------------+---------------------+---------+
/* etc */

$sbt it:flywayInfo
[info] Flyway 4.0 by Boxfuse
org.flywaydb.core.api.FlywayException: Unable to connect to the database. Configure the url, user and password!

$sbt prod:flywayInfo
[info] Flyway 4.0 by Boxfuse
org.flywaydb.core.api.FlywayException: Unable to connect to the database. Configure the url, user and password!


Swap memory not being used

Hey, I installed the swap memory and then increased the size of the swap memory. But in spite of that the swap memory is not being used.
While running R Viz and Open Pose together the nano freezes.
The physical ram is at full while the swap is just at 2 Gb.
Can you kindly let me know how to make the nano use the max swap memory please?
thanks

Following this method worked in my case:
sudo fallocate -l 8G /mnt/8GB.swap
sudo chmod 600 /mnt/8GB.swap
sudo mkswap /mnt/8GB.swap
sudo swapon /mnt/8GB.swap
sudo vim /etc/fstab

// add the following and save it
/mnt/8GB.swap swap swap defaults 0 0


add new helper functions

This diiff adds new helper funcs validateTicketHash() and getCommitmentAddress(). Also it modifies helper func validateSignature(). Originally part of #322

Going to merge this with a couple of minor outstanding issues. I will fix those myself in a follow-up PR shortly.


Upgrade SVG-Edit 2.7

I have created branch that includes @stevobengtson upgrade, rename of folder from svg-edit-2.7 to svg-edit and note in README on Customizing SVG-Edit.

I didn't review this request but would appreciate I verion that ships with svg-edit 2.7.
Is 2.7.1 in this merge included or is it 2.7.0?

Its 2.7.0


fix to allow write detailed list of products pp

I am also trying  $response = $this->merchant->purchase($params);  but I can't get any response or cursor on  $response.
What can i get in $response and how. ?

and another question is .. I test with paypal and I did complete whole process pay from payal (demo) and redirect are working.. but I couldn't get any entry in sandbox.paypal account..
any idea .?
what should i do .?


fix: pfe tabs padding

Updating padding for tabs.

https://rally1.rallydev.com/#/270861059696d/teamboard?detail=%2Fuserstory%2F492652479940%2Ftasks&view=113b0e72-0781-41c7-b6d6-e161d3349c72
https://xd.adobe.com/view/cc7627a7-ead9-4c7a-8664-ac526c680bd9-9925/specs/.
https://trello.com/c/2zvfhO7K/847-fix-pfe-tabs-padding
Update variables to new standard with fallbacks in pfe-tabs as well as update some other code while I am in there.

Preview

Link(s) to demo page(s) where this element can be viewed:

https://deploy-preview-1457--happy-galileo-ea79c4.netlify.app/elements/pfe-tabs/demo/

What has changed and why

pfe-tabs.scss has been updated with new padding vars and values as well as some other updates.

Testing instructions

Open the tabs demo page and you should see the following changes in the XD file (purple spacer is 48px by 48px)
Demo page

XD file:

https://xd.adobe.com/view/cc7627a7-ead9-4c7a-8664-ac526c680bd9-9925/specs/

Test style overrides via the inspector-stylesheet
paste in
:root {
    --pfe-tabs__panel--PaddingTop: 100px;
    --pfe-tabs__panel--PaddingRight: 100px;
    --pfe-tabs__panel--PaddingBottom: 100px;
    --pfe-tabs__panel--PaddingLeft: 100px;

    --pfe-tabs__panel--BorderTop: 4px dashed purple;
    --pfe-tabs__panel--BorderRight: 4px dashed purple;
    --pfe-tabs__panel--BorderLeft: 4px dashed purple;
    --pfe-tabs__panel--BorderBottom: 4px dashed purple;

    --pfe-tabs__panel--BackgroundColor: pink;
}

Make sure overrides work and default styles are as expected from mockups. Also make sure there is not duplicate padding (panel and container)
Browser requirements
Your component should work in all of the following environments:

[ ] Latest 2 versions of Edge
[x] Internet Explorer 11 (should be useable, not pixel perfect)
[ ] Latest 2 versions of Firefox (one on Mac OS, one of Windows OS)
[ ] Firefox 68 (or latest version for Red Hat Enterprise Linux distribution)
[ ] Latest 2 versions of Chrome (one on Mac OS, one of Windows OS)
[x] Latest 2 versions of Safari
[ ] Android mobile device (such as the Galaxy S9)
[ ] Apple mobile device (such as the iPhone X)
[ ] Apple tablet device (such as the iPhone Pro)

Ready-for-merge Checklist

[x] Expected files: all files in this pull request are related to one request or issue (no stragglers or scope-creep).
[x] Tests have been updated to cover these changes.
[ ] Browser testing passed.
[ ] Repository compiles and tests pass.
[ ] Changelog updated.
[x] Documentation (README.md, WHY.md, etc.) updated or added.
[ ] Link to the demo recording:
[ ] Approved by designer.

Merging
Please squash when merging and ensure your commit message uses conventional commit formatting.
Be sure to share your updates with the<EMAIL_ADDRESS>mailing list!

With the updated padding, the demo page is looking a bit cramped. Should we put them inside bands or maybe add a layout from the pfe-layouts file to add a little spacing between them? cc @starryeyez024

Love it! A couple minor nits in the Sass but this is fabulous. Thanks for taking care of this!

Thanks! Glad to help! It was fun


cmd/cue: get go panics because of a nil pointer dereference (net/http)

What version of CUE are you using (cue version)?

$ cue version
cue version 0.5.0

go version go1.20.3
      -buildmode exe
       -compiler gc
       -trimpath true
     CGO_ENABLED 1
          GOARCH amd64
            GOOS linux
         GOAMD64 v1

Does this issue reproduce with the latest stable release?
Yes
What did you do?
At first I ran into this with
$ go get "github.com/grafana/agent/pkg/operator/apis/monitoring/v1alpha1"
$ cue get go "github.com/grafana/agent/pkg/operator/apis/monitoring/v1alpha1"

After running again with --verbose it seemed to stop after net/http, so I also tried that directly and got a slightly shorter stack trace, with fewer recursive calls to cuelang.org/go/cmd/cue/cmd.(*extractor).extractPkg
What did you expect to see?
No output, as as usual from cue get go
What did you see instead?
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xaff439]

goroutine 1 [running]:
cuelang.org/go/cmd/cue/cmd.recoverError(0xc00caf9e90)
        cuelang.org/go/cmd/cue/cmd/root.go:261 +0x7b
panic({0xbaeea0, 0x1205b30})
        runtime/panic.go:884 +0x213
cuelang.org/go/cmd/cue/cmd.(*extractor).extractPkg(0xc012de85a0, {0xc000040064, 0x18}, 0x0)
        cuelang.org/go/cmd/cue/cmd/get_go.go:438 +0x79
cuelang.org/go/cmd/cue/cmd.(*extractor).extractPkg(0xc012de85a0, {0xc000040064, 0x18}, 0xc00001bc80)
        cuelang.org/go/cmd/cue/cmd/get_go.go:554 +0xb1a
cuelang.org/go/cmd/cue/cmd.extract(0xc0002dcea0, {0xc00024ffa0, 0x1, 0x1})
        cuelang.org/go/cmd/cue/cmd/get_go.go:417 +0x3e5
cuelang.org/go/cmd/cue/cmd.mkRunE.func1(0xc000302000?, {0xc00024ffa0, 0x1, 0x1})
        cuelang.org/go/cmd/cue/cmd/root.go:72 +0xb3
github.com/spf13/cobra.(*Command).execute(0xc000302000, {0xc00024ff70, 0x1, 0x1})
        github.com/spf13/cobra@v1.4.0/command.go:856 +0x67c
github.com/spf13/cobra.(*Command).ExecuteC(0xc0002ea780)
        github.com/spf13/cobra@v1.4.0/command.go:974 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.4.0/command.go:902
cuelang.org/go/cmd/cue/cmd.(*Command).Run(0xc0002dcea0, {0x4?, 0x4?})
        cuelang.org/go/cmd/cue/cmd/root.go:246 +0x6b
cuelang.org/go/cmd/cue/cmd.mainErr({0xd959a8, 0xc00003a0d8}, {0xc0000360b0?, 0xc0000061a0?, 0x200000003?})
        cuelang.org/go/cmd/cue/cmd/root.go:183 +0x55
cuelang.org/go/cmd/cue/cmd.Main()
        cuelang.org/go/cmd/cue/cmd/root.go:165 +0x7c
main.main()
        cuelang.org/go/cmd/cue/main.go:24 +0x19

Is this a duplicate of #2046?

Definitely looks like it, not sure how I managed to miss that in my search for existing issues. Apologies.


IsEmpty() should check for .empty(), not .size() == 0

Any update on this?

Thanks for reporting!
Is there a practical consequence to this?

Yes, inability to check for "emptiness" of things that are not containers (custom smart pointers or wrappers, for example).

Yes, that makes sense. Thanks.
Right now, IsEmpty() is implemented as a special case of HasLength(). We need to promote IsEmpty() to using its own IsEmptyConstraint.

Any update on this?

Any update on this?

I just looked into it in the hope that it would be simple.
For example, C++11's std::forward_list has no size() method. I included it in sequence_container_tests.cpp and noticed that a lot more than isEmpty() uses the size() method, for example, also the All operator does. Interestingly, also the so-far compiling tests fail on std::forward_list... Very weird. I pushed the work in progress to the more-container-tests branch, if interested, but I am not going to work on it now.

Update: The before-mentioned branch is already merged and deleted; it contained only preliminary work. I am now working on the issue in the issue-14 branch.


Cleaned up comparison levels documentation to be a multi-line code bl…

Cleaned up comparison levels documentation to be a multi-line code block as well as changed label to label_for_charts as label is not a value property. Added a note that dictionaries must be defined by double quotes as if they're valid json, not single quotes. I haven't encountered this before but it certainly tripped me up.

Linked Issue: https://github.com/moj-analytical-services/splink/issues/1237


Implement Chronoamperometry

Port from E4.

Arbitrary time potential steps
Reasonable number of steps
Integration script

Did none of that. We will quite likely only be interested in stepping to a single potential and back to the initial potential, so Sunny's implementation is fine.
Fitting script not written yet.

Bare-bones fitting script CA-fitting was uploaded and tested with E4 data.

Need to compute integrals. Probably on a case-by-case basis (given arbitrary time of steps, offset) using np.trapz(). I am worried that the contribution from Rm * Cm will be significant.
Determined empirical prefactor fit to CA data needs to be converted into diffusion coefficients.


Detect missing Rubies and offer to install them

If we detect a configured Ruby version that isn't installed, we might be able to offer the option for installing it from the VS Code extension. That would be a pretty nice experience.
This won't work for version managers that don't handle Ruby installation as well (like Shadowenv or chruby). But it would work for rbenv, rvm, mise and asdf - as long as detecting that the configured Ruby is missing is straightforward.

rbenv doesn't actually come with the build command by default: https://github.com/rbenv/ruby-build
https://github.com/rbenv/rbenv?tab=readme-ov-file#installing-ruby-versions

I think that such a tool could be useful even as something independent from Ruby LSP (but the VS Code extension could make use it, for a better experience).

Potentially related: https://github.com/Shopify/ruby-lsp/issues/2583


script: uniform interval factor and steppedLine in tidb_summary (#882)

cherry-pick #882 to release 3.0

LGTM


⚠ ClamAV has degraded performance

In 8fa8b52, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 224 ms

Resolved: ClamAV performance has improved in 2ceb1aa.


Frequency configuration

After only installing the integration about 12 hours ago, it has already performed the most hits of any device on my entire network;
https://i.imgur.com/vFBJlfd.png
It would be good to be able to customise the frequencies that are noted in README.md to tweak based on user preferences

Thanks for reporting this - the reason it makes so many hits is that although the refresh times are in line with typical integrations, annoyingly the data needed comes from 4 different API endpoints.
I've released a beta (v1.0.1-beta.2) which allows the refresh intervals to be configured. Would you mind testing this out please?
Dan

Thanks, that was fast! Installed and configured, will keep an eye on the calls

If you enable debug logging you'll also be able to see when the coordinators run and data is fetched. You'll probably have to filter the logs though as you'll get a lot!

@gstevenson could you please let me know which tool you used to visualise the network calls? Thanks a bunch!

@xenjke  I have Adguard Home configured as my DNS server at router level, so any device doing a DNS lookup shows in AGH's logs


iOS installation

I have not had any luck implementing either this lib, or your https://github.com/prscX/react-native-fluidic-slider lib, with ios. They use kind of the same installation pattern.
Is there a reason for the setup using this specific pattern, or would it be possible to install them the way most libs do. I.e adding needed pods to to the project own podfile and os on.
Also BRYXbanner throws this error for me:
BRYXBanner does not specify a Swift version and none of the targets (...) integrating it have the SWIFT_VERSION` attribute set.
How did you get around it?

As explained in this Issue https://github.com/prscX/react-native-notification-banner/issues/5, there's currently an issue with the repo that means that the Pods.xcodeproj cannot be installed in a version of cocoapods greater than 1.5.3
How to downgrade to 1.5.3:
sudo gem uninstall cocoapods
sudo gem install cocoapods -v 1.5.3
If you now run pod install whilst inside directory  ~/Documents/Coding/drinks-app/mobile/node_modules/react-native-notification-banner/ios you should be able to see that Pods.xcodeproj is downloaded and you can continue with the installation 😊

Wow, i tried just installing cocoapods 1.5.3 and use that setup.
uninstall and then install solved it! Thanks!

Hi: I have found the fix. I will push the release.
Basically, based on latest pods, the native library should define swift version which was earlier not needed.
Thanks
</ Pranav >

Thank you for all of the wonderful work on this library, it’s been a fantastic addition to my current React Native project!
That’s excellent news, many thanks Pranav!

Hi All: I have pushed the release with the cocoapods installation issue.
Could you please install latest release and verify the same. Please let me know if you still face the same issue.
Thanks
</ Pranav >


TestApp: add Highlights UI

This PR aims to add a Highlights support to the TestApp using an existing R2Navigator API.
A video of how the first version of this PR works:  video-link-1
What was changed:

Navigator/EditingAction was refactored to manage the context menu also for Highlight, not only for Selection.
As a context menu is the system singleton (UIMenu.shared) and is updated when selection property updated from the navigator, the same flow was added for the highlight.
If not add this, and store/manage a highights menu e.g. in the TestApp, then EditingAction may interfere and discard the menu which was set for Highlight when it gets a "ghost" SelectionDidChange callback from JS.

HighlightableNavigator was added by analogy to SelectableNavigator (it seems, HighlightableNavigator should be a separate concept from DecorableNavigator).

A persistence was added for highlights by analogy to e.g. Bookmarks.

A list view was added for highlights in OutlineViewController.

A following reasoning was used: Decoration is a "view", and Highlight is a model. So their ids (String and UUID) should match for a simple 1:1 mapping (otherwise it's hard to find a corresponding highlight for a user-tapped decoration, requires some persisted dictionary etc). But the dictionary for the Decoration color mapping was still added, because when we are showing a menu for changing the highlight color, we must know a current Decoration model color instantly, not in a callback from GRDB "repository".

I assume I did a lot of things not in a preferred way, so will be glad to fix them after a feeback.
Happy New Year! 🎄

@mickael-menu  Thanks for the feedback!

You should be able to implement highlighting directly in the Test App without modifying the R2Navigator library. This will make sure that the highlight implementation is very flexible for reading apps.
...
Regarding the highlight color menu, it might be easier to use a dedicated SwiftUI view in a popover instead of the UIMenuController, to avoid messing with the editing actions.

Just realised that I went down the rabbit hole with changing Navigator just because I had issues with synchronising UIMenuControllers. 🙂
After I fix this, will force-push a new branch and ping you.

Hi @mickael-menu! Can you please review this again?
I've tried to address your feedback, there's at least one thing I'm sure of: only the TestApp folder has changes. 🙂

@gatamar Great, thank you! I'll review this Friday or earlier if I can.


Not yet loaded cubes launch NPE in attempt to set block in them.

Long story short, player stuck in unloaded cube, tried to place a block in it and catches exception.
A detailed walkthrough of the error, its code path and all known details is as follows:
---------------------------------------------------------------------------------------

-- Head --
Thread: Client thread
Stacktrace:
	at net.minecraft.world.chunk.Chunk.setEBS_CubicChunks(Chunk.java:1757)
	at net.minecraft.world.chunk.Chunk.redirect$setBlockState_CubicChunks_EBSSetRedirect$zzj000(Chunk.java:2010)
	at net.minecraft.world.chunk.Chunk.func_177436_a(Chunk.java:556)
	at net.minecraft.world.World.func_180501_a(World.java:342)
	at net.minecraft.item.ItemBlock.placeBlockAt(ItemBlock.java:180)
	at net.minecraft.item.ItemBlock.func_180614_a(ItemBlock.java:54)
	at net.minecraft.item.ItemStack.func_179546_a(ItemStack.java:187)
	at net.minecraft.client.multiplayer.PlayerControllerMP.func_187099_a(PlayerControllerMP.java:466)
	at net.minecraft.client.Minecraft.func_147121_ag(Minecraft.java:1577)
	at net.minecraft.client.Minecraft.func_184117_aA(Minecraft.java:2252)
	at net.minecraft.client.Minecraft.func_184118_az(Minecraft.java:2018)

Would be appropriate fix to load missing cube synchronously whenever player reach it or tried to place block in it?

You can't load a cube clientside. What should be done is that setBlockState should do nothing in that case (but only clientside).

Yes, you are right. Also a shivering near a border reminds me a way when player collided with blocks server side only.
Could it be that network thread overflowed with tasks and cube packet does not reach client in time due to that?

I think this is some failure in PlayerCubeMap. I even thing I know what causes it. Unload packets also should be queued the same way as chunk data packets.

cubesToSend field from PlayerCubeMap should be a List of CubeWatchers?

Is this still an issue?


Crash on Login Page

When ever clicked on Login button, the website crashes after loading Social Login buttons
Browser :- Firefox 104.0.2 64 Bit
OS :- Pop! OS (Linux)

I guess it a issue with MetaMask