Change BN layer to use moving mean/var if frozen

During fine-tuning, if a Batch Normalization layer is frozen it uses the mini-batch statistics. I believe this is incorrect and it can lead to reduced accuracy especially when we use Transfer learning. A better approach in this case would be to use the values of the moving mean and variance.
Changes on this PR:
In this PR I update the Batch Normalization layer to use the learned statistics if frozen during training. This is achieved by making the trainable flag part of the computational graph and by depending the behavior of the BN  not only on the learning_phase but also on the value of the trainable property.
Brief explanation:
Assume we use one of the pre-trained CNNs of Keras and we want to fine-tune it. Unfortunately, we get no guarantees that the mean and variance of our new dataset inside the BN layers will be similar to the ones of the original dataset. As a result, if we fine-tune the top layers, their weights will be adjusted to the mean/variance of the new dataset. Nevertheless, during inference the top layers will receive data which are scaled using the mean/variance of the original dataset. This discrepancy can lead to reduced accuracy.
I understand that this is a significant change that requires thorough review. To faciliate the situation I've documented why making such a change is important and provided detailed comparisons before and after applying the patch on my blog.

Thanks for the effort.
You are misunderstanding the meaning of the "trainable" property of layers. Historically, it has initially "this layer should not be trainable, i.e. the weights of this layer should not be updated during backprop (specifically layer.trainable_weights should be empty)". Then it has been extend to mean "the state of the layer should be frozen during training" (which means that, in addition to the previous definition, layer updates are not run).
What you want is a BN layer in inference mode. There is an argument to control training/inference mode in BN (and other layers): it's the training argument in call (boolean).
What you want is:
x = BatchNormalization()(y, training=False)

For fine-tuning, you could do something like:
# Set up inference-mode base
K.set_learning_phase(0)
inputs = Input(...)
x = layer1(...)(inputs)
x = layer2(...)(x)
...
x = layerN(...)(x)

# Add training-mode layers
K.set_learning_phase(1)
x = layerNp1(...)(x)
x = layerNp2(...)(x)

Hi @fchollet,
First of all thanks for taking the time to review and respond. I was aware that this is a significant change in the default behaviour and that there would be debate. :)
I understand that you main concern is around the semantic meaning of the trainable property and how it is being used in this PR. I agree that semantically the training parameter that you proposed is closer to what I do, nevertheless this parameter can't change after the network definition. For instance when you use one of the pre-trained models of keras or when you load a persisted model you have no control over this variable. Would you be open to discuss a solution that would make the training variable changeable after the network definition (or perhaps another property)? If you are open in this, I could update my PR to reflect the agreed behaviour.
Concerning your second recommendation of updating the learning_phase as the network is defined, I see two limitations:

Again this will work only if the network is defined based on code. It will not work for the pretrained models of Keras or when a model is loaded from disk. The latter is quite important; models are trained in multiple rounds usually after restoring it from a checkpoint.
After setting the learning_phase(1) in your example, the learning_phase will be static for the remaining of the session. This will overwrite all the nice mechanisms that keras has for switching between phases depending on whether it trains or predicts. Thus if we call fit() with validation data, the model will predict while being in inference mode.

I'm not sure if you had a look on the blog post (it is understandably a bit long), but you can see how significant perfomance boost you get by making it possible to set the BN in inference mode. I hope, we can re-open this PR; I'm happy to update it until it satisfies the semantic definitions.
Cheers!

Again, there is an existing API that does exactly what you want: the training argument in call. There is no point in having two differently name APIs that do the exact same thing. layer.trainable = False is not what you need, therefore don't use it.
Additionally, your proposed PR adds a computational overhead (which might amount to a ~5% slowdown for a BN-heavy model like InceptionV3) to every single convnet that uses BN, fine-tuning or not. This is a heavy price to pay for supporting an incrementally simpler UX (disputable) for a very specific use case.

For instance when you use one of the pre-trained models of keras or when you load a persisted model you have no control over this variable.

Typically if you want to heavily modify an existing model, rather than merely use it in inference mode, you should have access to the code for the model.
But even if you don't, you can still do your style of fine-tuning in this case:

set learning phase to 0
load model
retrieve features you want to train on
set learning phase to 1
add new layers on top
optionally load weights from initial model layers to corresponding new layers
train

@fchollet My main point is that the training argument can't be changed after model definition, so the existing API does not cover this valid case. I don't argue that there are workarounds, but they are hacky/non-elegant and the default behaviour leads to much confusion to users. Interesting what you mention about the 5% slow down, I would love to see the benchmarks; perhaps it can be resolved. Finally something you don't address here is whether this discrepancy in the scaling makes sense (theoretically or otherwise) and whether the accuracy decrease is worth it.
At any case, let's agree we disagree. I do hope though that you will revise your decision on the future, as it happened with the update of the mini-batch statistics on the BN.

I would love to see the benchmarks

This is based on something I've observed in the past for InceptionV3 with static learning phase vs. with dynamic learning phase. Only difference between the two settings is cond ops. Control seems pretty expensive, especially on GPU. Your PR adds the exact same number cond ops, so I would expect the same overhead.

Thanks for the clarifying that you are referring to a different benchmark and not to something you ran on this PR. I can't comment on the results without seeing them but when I ran comparisons on CIFAR10 the time difference was negligible (current branch: 4216 secs vs patched: 4251 secs); both ran on GPUs on the same server. Note that the snippet that I used (and listed on my article) comes from Keras' documentation on how to fine-tune a network.
Admittedly the above measurements are single point estimates but especially the 5 point accuracy increase I report is consistent with what I've been observing for almost a year while applying workarounds (first time I reported this is on #7177). I don't know if the speed is currently your main concern for reopening this but I would say that this is unlikely to affect the majority of the users of Keras. This is because by default the Learning Phase is dynamic and the training argument of call is None. This will force the in_train_phase method on backend to use a switch statement that depends on learning phase, so in a sense the "if" statement is already there.
At any case I don't insist that it should me who changes this or that my current solution is the one we should use. I'm just raising a valid use case that is taken directly from Keras' documentation on how fine-tuning is performed. Currently there is no straightforward way to do what I describe (the current API doesn't cover it), nevertheless if you provide specific guidelines on what tickboxes the update should check it would be useful. Or perhaps some other longtime contributor of the BatchNormalization layer has an opinion or can offer a more elegant solution on this? @ozabluda @taehoonlee @farizrahman4u

Sorry for late reply, still trying to understand the issues. For example, I am trying to understand if this is related at all to https://github.com/keras-team/keras/issues/9214

What sort of batch sizes were you using in your linked experiments?
Some datasets are only viable with very small batch sizes of 1-4, like with image segmentation on a GPU with 8GB of memory. After briefly skimming this diff, I think the documentation would need to be updated to clearly delineate the different modes and when/why each should typically be chosen. In my case the current frozen behavior improved performance quite a lot over the previous behavior in which mean/var could shift when trainable=False, so I'm a bit hesitant about this though I'll reiterate I haven't reviewed what's happening in full detail.
Here is a PR with some past discussion on BN https://github.com/keras-team/keras/pull/8616

@ozabluda First of all thank you for spending time on this. I wish I had provided on my PR the example that you posted on the issue #9214; perhaps this would have built a stronger case for this patch. What you showed on your post is exactly what I've been observing on real-word non-opensource datasets for the last year (close to 100% accuracy on training mode and 50% during inference on the same dataset). As @fchollet  said the are lots of hacks that can help you avoid it but none of them should have been necessary.
Based on the code you provided, I'm 100% certain you are being bitten by the behaviour of the BN layer that I'm trying to fix in this PR. In a nutshell, during training mode the frozen BN layers are scaled with different statistics that in inference mode. There is absolutely no theoretical foundation to support this behaviour. As a result this can have devastating results when you try to deploy the model or when you try to validate it's results. I am certain that the majority of people who face this believe they have overfitted the model while in reality this is just a side-effect of how Keras implements the Batch Normalization layer.
So let's test your example on my branch of Keras where the BN layer is patched:
pip install -U --force-reinstall --no-dependencies git+https://github.com/datumbox/keras@bugfix/trainable_bn

Below I run your code for ResNet50. As you can see the problem that you report is fixed once the BN behaviour is changed:
Epoch 1/100
50/50 [==============================] - 19s 387ms/step - loss: 0.8738 - acc: 0.5000 - val_loss: 1.3021 - val_acc: 0.5000
Epoch 2/100
50/50 [==============================] - 18s 367ms/step - loss: 1.3021 - acc: 0.5000 - val_loss: 0.9412 - val_acc: 0.5000
Epoch 3/100
50/50 [==============================] - 18s 363ms/step - loss: 0.9412 - acc: 0.5000 - val_loss: 0.6904 - val_acc: 0.5000
Epoch 4/100
50/50 [==============================] - 18s 364ms/step - loss: 0.6904 - acc: 0.5000 - val_loss: 0.9428 - val_acc: 0.5000
Epoch 5/100
50/50 [==============================] - 18s 361ms/step - loss: 0.9428 - acc: 0.5000 - val_loss: 0.9180 - val_acc: 0.5000
Epoch 6/100
50/50 [==============================] - 20s 401ms/step - loss: 0.9180 - acc: 0.5000 - val_loss: 0.7111 - val_acc: 0.5000
Epoch 7/100
50/50 [==============================] - 21s 415ms/step - loss: 0.7111 - acc: 0.5000 - val_loss: 0.6802 - val_acc: 0.5200
Epoch 8/100
50/50 [==============================] - 20s 406ms/step - loss: 0.6802 - acc: 0.5200 - val_loss: 0.8039 - val_acc: 0.5000
Epoch 9/100
50/50 [==============================] - 20s 391ms/step - loss: 0.8039 - acc: 0.5000 - val_loss: 0.8075 - val_acc: 0.5000
Epoch 10/100
50/50 [==============================] - 21s 425ms/step - loss: 0.8075 - acc: 0.5000 - val_loss: 0.6963 - val_acc: 0.5000
Epoch 11/100
50/50 [==============================] - 21s 417ms/step - loss: 0.6963 - acc: 0.5000 - val_loss: 0.6406 - val_acc: 0.7000
Epoch 12/100
50/50 [==============================] - 21s 419ms/step - loss: 0.6406 - acc: 0.7000 - val_loss: 0.7017 - val_acc: 0.5000
Epoch 13/100
50/50 [==============================] - 21s 425ms/step - loss: 0.7017 - acc: 0.5000 - val_loss: 0.7408 - val_acc: 0.5000
Epoch 14/100
50/50 [==============================] - 22s 441ms/step - loss: 0.7408 - acc: 0.5000 - val_loss: 0.6895 - val_acc: 0.5000
Epoch 15/100
50/50 [==============================] - 22s 432ms/step - loss: 0.6895 - acc: 0.5000 - val_loss: 0.6267 - val_acc: 0.7200
Epoch 16/100
50/50 [==============================] - 23s 460ms/step - loss: 0.6267 - acc: 0.7200 - val_loss: 0.6376 - val_acc: 0.5600
Epoch 17/100
50/50 [==============================] - 22s 439ms/step - loss: 0.6376 - acc: 0.5600 - val_loss: 0.6775 - val_acc: 0.5400
Epoch 18/100
50/50 [==============================] - 23s 456ms/step - loss: 0.6775 - acc: 0.5400 - val_loss: 0.6675 - val_acc: 0.5400
Epoch 19/100
50/50 [==============================] - 21s 414ms/step - loss: 0.6675 - acc: 0.5400 - val_loss: 0.6209 - val_acc: 0.6000
Epoch 20/100
50/50 [==============================] - 19s 375ms/step - loss: 0.6209 - acc: 0.6000 - val_loss: 0.6055 - val_acc: 0.7400
Epoch 21/100
50/50 [==============================] - 18s 367ms/step - loss: 0.6055 - acc: 0.7400 - val_loss: 0.6309 - val_acc: 0.5800
Epoch 22/100
50/50 [==============================] - 18s 370ms/step - loss: 0.6309 - acc: 0.5800 - val_loss: 0.6392 - val_acc: 0.5600
Epoch 23/100
50/50 [==============================] - 18s 369ms/step - loss: 0.6392 - acc: 0.5600 - val_loss: 0.6111 - val_acc: 0.6400
Epoch 24/100
50/50 [==============================] - 19s 390ms/step - loss: 0.6111 - acc: 0.6400 - val_loss: 0.5890 - val_acc: 0.7800
Epoch 25/100
50/50 [==============================] - 20s 394ms/step - loss: 0.5890 - acc: 0.7800 - val_loss: 0.5990 - val_acc: 0.6200
Epoch 26/100
50/50 [==============================] - 22s 445ms/step - loss: 0.5990 - acc: 0.6200 - val_loss: 0.6105 - val_acc: 0.5800
Epoch 27/100
50/50 [==============================] - 21s 413ms/step - loss: 0.6105 - acc: 0.5800 - val_loss: 0.5961 - val_acc: 0.6000
Epoch 28/100
50/50 [==============================] - 19s 388ms/step - loss: 0.5961 - acc: 0.6000 - val_loss: 0.5759 - val_acc: 0.8000
Epoch 29/100
50/50 [==============================] - 20s 391ms/step - loss: 0.5759 - acc: 0.8000 - val_loss: 0.5767 - val_acc: 0.7400
Epoch 30/100
50/50 [==============================] - 19s 372ms/step - loss: 0.5767 - acc: 0.7400 - val_loss: 0.5857 - val_acc: 0.7400
Epoch 31/100
50/50 [==============================] - 22s 433ms/step - loss: 0.5857 - acc: 0.7400 - val_loss: 0.5785 - val_acc: 0.7600
Epoch 32/100
50/50 [==============================] - 19s 373ms/step - loss: 0.5785 - acc: 0.7600 - val_loss: 0.5627 - val_acc: 0.7800
Epoch 33/100
50/50 [==============================] - 21s 417ms/step - loss: 0.5627 - acc: 0.7800 - val_loss: 0.5597 - val_acc: 0.7800
Epoch 34/100
50/50 [==============================] - 21s 422ms/step - loss: 0.5597 - acc: 0.7800 - val_loss: 0.5651 - val_acc: 0.7000
Epoch 35/100
50/50 [==============================] - 18s 365ms/step - loss: 0.5651 - acc: 0.7000 - val_loss: 0.5606 - val_acc: 0.7200
Epoch 36/100
50/50 [==============================] - 18s 362ms/step - loss: 0.5606 - acc: 0.7200 - val_loss: 0.5488 - val_acc: 0.8000
Epoch 37/100
50/50 [==============================] - 18s 361ms/step - loss: 0.5488 - acc: 0.8000 - val_loss: 0.5449 - val_acc: 0.7800
Epoch 38/100
50/50 [==============================] - 18s 361ms/step - loss: 0.5449 - acc: 0.7800 - val_loss: 0.5473 - val_acc: 0.8000
Epoch 39/100
50/50 [==============================] - 18s 361ms/step - loss: 0.5473 - acc: 0.8000 - val_loss: 0.5433 - val_acc: 0.8000
Epoch 40/100
50/50 [==============================] - 18s 368ms/step - loss: 0.5433 - acc: 0.8000 - val_loss: 0.5344 - val_acc: 0.8000
Epoch 41/100
50/50 [==============================] - 19s 372ms/step - loss: 0.5344 - acc: 0.8000 - val_loss: 0.5311 - val_acc: 0.8600
Epoch 42/100
50/50 [==============================] - 18s 367ms/step - loss: 0.5311 - acc: 0.8600 - val_loss: 0.5318 - val_acc: 0.7800
Epoch 43/100
50/50 [==============================] - 18s 366ms/step - loss: 0.5318 - acc: 0.7800 - val_loss: 0.5278 - val_acc: 0.7800
Epoch 44/100
50/50 [==============================] - 18s 367ms/step - loss: 0.5278 - acc: 0.7800 - val_loss: 0.5208 - val_acc: 0.8800
Epoch 45/100
50/50 [==============================] - 18s 363ms/step - loss: 0.5208 - acc: 0.8800 - val_loss: 0.5181 - val_acc: 0.8200
Epoch 46/100
50/50 [==============================] - 18s 367ms/step - loss: 0.5181 - acc: 0.8200 - val_loss: 0.5175 - val_acc: 0.8200
Epoch 47/100
50/50 [==============================] - 19s 372ms/step - loss: 0.5175 - acc: 0.8200 - val_loss: 0.5131 - val_acc: 0.8400
Epoch 48/100
50/50 [==============================] - 19s 372ms/step - loss: 0.5131 - acc: 0.8400 - val_loss: 0.5075 - val_acc: 0.8600
Epoch 49/100
50/50 [==============================] - 19s 384ms/step - loss: 0.5075 - acc: 0.8600 - val_loss: 0.5053 - val_acc: 0.9000
Epoch 50/100
50/50 [==============================] - 19s 382ms/step - loss: 0.5053 - acc: 0.9000 - val_loss: 0.5035 - val_acc: 0.8400
Epoch 51/100
50/50 [==============================] - 18s 369ms/step - loss: 0.5035 - acc: 0.8400 - val_loss: 0.4989 - val_acc: 0.9000
Epoch 52/100
50/50 [==============================] - 20s 394ms/step - loss: 0.4989 - acc: 0.9000 - val_loss: 0.4944 - val_acc: 0.8800
Epoch 53/100
50/50 [==============================] - 19s 372ms/step - loss: 0.4944 - acc: 0.8800 - val_loss: 0.4920 - val_acc: 0.8800
Epoch 54/100
50/50 [==============================] - 18s 367ms/step - loss: 0.4920 - acc: 0.8800 - val_loss: 0.4890 - val_acc: 0.8800
Epoch 55/100
50/50 [==============================] - 19s 371ms/step - loss: 0.4890 - acc: 0.8800 - val_loss: 0.4845 - val_acc: 0.9000
Epoch 56/100
50/50 [==============================] - 18s 369ms/step - loss: 0.4845 - acc: 0.9000 - val_loss: 0.4811 - val_acc: 0.8800
Epoch 57/100
50/50 [==============================] - 18s 362ms/step - loss: 0.4811 - acc: 0.8800 - val_loss: 0.4792 - val_acc: 0.9000
Epoch 58/100
50/50 [==============================] - 18s 367ms/step - loss: 0.4792 - acc: 0.9000 - val_loss: 0.4759 - val_acc: 0.9000
Epoch 59/100
50/50 [==============================] - 18s 368ms/step - loss: 0.4759 - acc: 0.9000 - val_loss: 0.4721 - val_acc: 0.8800
Epoch 60/100
50/50 [==============================] - 18s 366ms/step - loss: 0.4721 - acc: 0.8800 - val_loss: 0.4695 - val_acc: 0.9200
Epoch 61/100
50/50 [==============================] - 18s 370ms/step - loss: 0.4695 - acc: 0.9200 - val_loss: 0.4670 - val_acc: 0.9000
Epoch 62/100
50/50 [==============================] - 18s 368ms/step - loss: 0.4670 - acc: 0.9000 - val_loss: 0.4634 - val_acc: 0.9200
Epoch 63/100
50/50 [==============================] - 22s 433ms/step - loss: 0.4634 - acc: 0.9200 - val_loss: 0.4602 - val_acc: 0.9200
Epoch 64/100
50/50 [==============================] - 19s 370ms/step - loss: 0.4602 - acc: 0.9200 - val_loss: 0.4578 - val_acc: 0.9200
Epoch 65/100
50/50 [==============================] - 19s 374ms/step - loss: 0.4578 - acc: 0.9200 - val_loss: 0.4548 - val_acc: 0.9200
Epoch 66/100
50/50 [==============================] - 19s 383ms/step - loss: 0.4548 - acc: 0.9200 - val_loss: 0.4515 - val_acc: 0.9400
Epoch 67/100
50/50 [==============================] - 20s 393ms/step - loss: 0.4515 - acc: 0.9400 - val_loss: 0.4488 - val_acc: 0.9200
Epoch 68/100
50/50 [==============================] - 19s 373ms/step - loss: 0.4488 - acc: 0.9200 - val_loss: 0.4462 - val_acc: 0.9200
Epoch 69/100
50/50 [==============================] - 19s 373ms/step - loss: 0.4462 - acc: 0.9200 - val_loss: 0.4431 - val_acc: 0.9400
Epoch 70/100
50/50 [==============================] - 18s 364ms/step - loss: 0.4431 - acc: 0.9400 - val_loss: 0.4402 - val_acc: 0.9400
Epoch 71/100
50/50 [==============================] - 18s 366ms/step - loss: 0.4402 - acc: 0.9400 - val_loss: 0.4376 - val_acc: 0.9800
Epoch 72/100
50/50 [==============================] - 19s 370ms/step - loss: 0.4376 - acc: 0.9800 - val_loss: 0.4347 - val_acc: 0.9800
Epoch 73/100
50/50 [==============================] - 18s 369ms/step - loss: 0.4347 - acc: 0.9800 - val_loss: 0.4317 - val_acc: 0.9400
Epoch 74/100
50/50 [==============================] - 18s 369ms/step - loss: 0.4317 - acc: 0.9400 - val_loss: 0.4291 - val_acc: 0.9400
Epoch 75/100
50/50 [==============================] - 19s 372ms/step - loss: 0.4291 - acc: 0.9400 - val_loss: 0.4264 - val_acc: 0.9400
Epoch 76/100
50/50 [==============================] - 18s 369ms/step - loss: 0.4264 - acc: 0.9400 - val_loss: 0.4235 - val_acc: 0.9400
Epoch 77/100
50/50 [==============================] - 19s 376ms/step - loss: 0.4235 - acc: 0.9400 - val_loss: 0.4208 - val_acc: 0.9600
Epoch 78/100
50/50 [==============================] - 19s 377ms/step - loss: 0.4208 - acc: 0.9600 - val_loss: 0.4182 - val_acc: 0.9800
Epoch 79/100
50/50 [==============================] - 19s 381ms/step - loss: 0.4182 - acc: 0.9800 - val_loss: 0.4154 - val_acc: 0.9600
Epoch 80/100
50/50 [==============================] - 19s 370ms/step - loss: 0.4154 - acc: 0.9600 - val_loss: 0.4127 - val_acc: 0.9400
Epoch 81/100
50/50 [==============================] - 18s 369ms/step - loss: 0.4127 - acc: 0.9400 - val_loss: 0.4101 - val_acc: 0.9400
Epoch 82/100
50/50 [==============================] - 19s 371ms/step - loss: 0.4101 - acc: 0.9400 - val_loss: 0.4075 - val_acc: 0.9400
Epoch 83/100
50/50 [==============================] - 18s 364ms/step - loss: 0.4075 - acc: 0.9400 - val_loss: 0.4048 - val_acc: 0.9600
Epoch 84/100
50/50 [==============================] - 18s 365ms/step - loss: 0.4048 - acc: 0.9600 - val_loss: 0.4022 - val_acc: 0.9800
Epoch 85/100
50/50 [==============================] - 18s 367ms/step - loss: 0.4022 - acc: 0.9800 - val_loss: 0.3996 - val_acc: 0.9800
Epoch 86/100
50/50 [==============================] - 18s 368ms/step - loss: 0.3996 - acc: 0.9800 - val_loss: 0.3970 - val_acc: 0.9600
Epoch 87/100
50/50 [==============================] - 18s 370ms/step - loss: 0.3970 - acc: 0.9600 - val_loss: 0.3945 - val_acc: 0.9600
Epoch 88/100
50/50 [==============================] - 18s 367ms/step - loss: 0.3945 - acc: 0.9600 - val_loss: 0.3919 - val_acc: 0.9600
Epoch 89/100
50/50 [==============================] - 18s 366ms/step - loss: 0.3919 - acc: 0.9600 - val_loss: 0.3894 - val_acc: 0.9600
Epoch 90/100
50/50 [==============================] - 18s 366ms/step - loss: 0.3894 - acc: 0.9600 - val_loss: 0.3869 - val_acc: 0.9800
Epoch 91/100
50/50 [==============================] - 19s 371ms/step - loss: 0.3869 - acc: 0.9800 - val_loss: 0.3844 - val_acc: 0.9800
Epoch 92/100
50/50 [==============================] - 18s 368ms/step - loss: 0.3844 - acc: 0.9800 - val_loss: 0.3819 - val_acc: 0.9800
Epoch 93/100
50/50 [==============================] - 18s 368ms/step - loss: 0.3819 - acc: 0.9800 - val_loss: 0.3795 - val_acc: 0.9800
Epoch 94/100
50/50 [==============================] - 18s 366ms/step - loss: 0.3795 - acc: 1.0000 - val_loss: 0.3770 - val_acc: 1.0000
Epoch 95/100
50/50 [==============================] - 18s 369ms/step - loss: 0.3770 - acc: 1.0000 - val_loss: 0.3746 - val_acc: 1.0000
Epoch 96/100
50/50 [==============================] - 18s 366ms/step - loss: 0.3746 - acc: 1.0000 - val_loss: 0.3722 - val_acc: 1.0000
Epoch 97/100
50/50 [==============================] - 18s 368ms/step - loss: 0.3722 - acc: 1.0000 - val_loss: 0.3698 - val_acc: 1.0000
Epoch 98/100
50/50 [==============================] - 18s 366ms/step - loss: 0.3698 - acc: 1.0000 - val_loss: 0.3674 - val_acc: 1.0000
Epoch 99/100
50/50 [==============================] - 18s 367ms/step - loss: 0.3674 - acc: 1.0000 - val_loss: 0.3651 - val_acc: 1.0000
Epoch 100/100
50/50 [==============================] - 18s 368ms/step - loss: 0.3651 - acc: 1.0000 - val_loss: 0.3627 - val_acc: 1.0000

I would love to know if you can reproduce my results and whether you can observe any speed degradation that @fchollet suspects.

@ahundt Thanks for your comment!
In this very specific experiment I used a fixed batch size of 32. Nevertheless in this dummy example I try to reproduce a behaviour we've been facing for over a year now on real-world datasets and problems. In those cases a large number of different batch sizes were tested and the results were comparable.
Please note that his PR DOES NOT undo the recent change where the mean/var no longer shifts when trainable=False. I 100% agree with you that this change is very beneficial. This PR actually takes it a step further and makes sure that the moving mean/var are used instead of the mini-batch statistics when trainable=False. This ensures that the non-frozen layers will be trained on data scaled the same way as in inference mode.
BTW thanks for sending me the discussion on #8616. Give me sometime to read all the details to see how this is related.

@ahundt I've read the discussion on #8616. I understand it focuses on the previous change on BN that correctly stopped the update of the moving mean/var when trainable=False. I totally agree with this change. As I said on my previous comment, this PR takes this a step further to ensure that the data after a frozen BN are scaled in the same way during training as during inference.
What I find interesting is that during the original discussion on #8616, @fchollet raises similar concerns about the semantic meaning of trainable as in this PR. Nevertheless in that discussion, he proposed the introduction of another property to extend the API. I also see he tried to implement another property called "updatable" which was reverted due to the increased complexity (and at the end we settled with extending the semantics of trainable). I wonder if in this case it makes sense to extend the API to cover this valid case OR update the semantics of trainable (preferred solution) OR update the documentation/examples.
I would love to have an opinion from @lukedeo on this since he reviewed the code on the other PR.

@datumbox Ok think I see what you are saying, I might try this out on my dataset. Do I need to change any settings like trainable in my training code or can I just pull this in? In my example I use frozen vgg16 imagenet pretrained weights as a feature extractor with additional trainable layers afterwards.
I think your PR, variable names, and description might be worth rewording to better separate the concepts and clarify the conditions under which different data is fixed vs changing.

Ok so based on the test_batchnorm_trainable() changes this should be active by default in all cases except when both learning phase=1 and trainable=True.
# In all other cases we should use the moving mean and variance from BN.

Correct?

@ahundt Thanks for looking into this. My PR affects only networks that use Batch Normalization layers, so VGG will not be affected. No additional configuration is required other than setting trainable=False on the BN layers. Pulling this in should work fine just note that my fork is not synced with the latest stable version of Keras; I plan to do this soon as other people are interested.

One thing that might help with getting this through is a few improvements to the PR, variable names, and description.

Sure thing, send me your comments and I'll make the changes. :-)

I tried it out and I didn't see any substantial difference in my case in either speed or accuracy on my own problem. Perhaps I didn't apply or configure it correctly.

I think on the step 3, you meant to say "all layers from (1) to trainable=false and from (2) to trainable=True".
That can work but for a more profound effect you can also try fine-tuning a few layers from (1). In other words setting trainable=True to a few convolutions and Bns of (1) will do the trick. There is a great example on Keras documentation for fine-tuning here. This example fails immediately for any layer that uses BN.
Looking forward to hearing your results.

I've tested this update (Keras 2.1.6 fork + TF 1.7 + CUDA 9 on a p3.2xlarge AWS instance) using the following architectures: ResNet50, InceptionV3, InceptionResNetV2, Xception, DenseNet and NASNetLarge.
My setup is similar to the "Fine-tune InceptionV3 on a new set of classes" example of the documentation. In all tests, I've used the pre-trained ImageNet weights of the framework and fine-tuned the networks on CIFAR10. The point to which I've frozen the pre-trained layers varies from network to network but in all cases, I've set trainable=True to at least 1 BatchNormalization and Convolution layer from the top of the base model. All other configurations (optimizer, batch-size, epochs etc) were kept the same across all executions. Using the fork, my accuracy improved by 3 to 6 points (depending on the architecture) comparing to the standard Keras. I've also noticed that on the fork the models converged faster and that between epochs the loss/accuracy deltas were "smoother". This is especially true for ResNet50 and InceptionV3 where the metrics of standard Keras jumped all over the place from one epoch to the next.
@ahundt Did you have the chance to run the test on your data? Please let me know if you observed a similar effect.
@fchollet I think it would be beneficial to extend Keras and allow BatchNormalization layers to run on Inference Mode after being initialized. One approach could be to have an inference property that the users can change after initialization. If you like that, let me know and I'd be happy to write it and send a pull request.

@datumbox  After reading the related issues I am on your side.
@fchollet

Again, there is an existing API that does exactly what you want: the training argument in call. There is no point in having two differently name APIs that do the exact same thing. layer.trainable = False is not what you need, therefore don't use i
I partly agree with author but in the fine-tune document you should point this issue out. For any beginner of Keras, it will lead to some misunderstanding.

I haven't abandoned this.

I think on the step 3, you meant to say "all layers from (1) to trainable=false and from (2) to trainable=True".

@datumbox thanks for your correction, yes that was a typo.
I've had hyperopt (~2k model variations for 1 epoch) running on a dataset to see if there is any movement. I have the numbers from the fork, but I'm waiting on the keras master run to finish for comparison. I might also do a third pair of runs where a random fraction of each network is trainable.
Ah so the two features we are discussing here are:

Layer.__call__(training=True)
Layer.trainable = True aka Model.layers[i].trainable = True

Ok I ran a hyperopt search on both master and this PR to compare. It is really set up for my research and not analyzing this PR so please forgive the glaring omissions of detail and the numerous ways this evaluation isn't perfect. :-)
There were 1143 models in two runs of a hyperparameter search with a fixed initial numpy seed, where all but ~30 configurations were random in two cases: master and pr 9965. The ~30 non random ones were chosen with Bayesian optimization on the hyperparameters via gpyopt. Examples of Hyperparams that changed included the model with pretrained imagenet weights: densenet, inception resnet v2, vgg, resnet, nasnet mobile, additional layers appended to the end as a resnet, densenet or vgg blocks, and a few other things. Training was for 1 epoch on each model with SGD and a learning rate of 0.02. It is a binary classification problem: success or failure of a robot to grasp an object.
Be aware:

1 epoch per model (sorry, I had 1 GPU to spare for this)
some of these models don't have batch norm at all
50% of them will be completely trainable
50% of them the imagenet weights will be completely locked

Here are the averages and medians of the hyperparameter configurations with each keras version:
master:

binary_accuracy
loss
lr
mean_pred
mean_true
test_binary_accuracy
test_loss
test_mean_pred
test_mean_true
val_binary_accuracy
val_loss

0.7047966469
1.2768393246
0.02
0.6532160745
0.6456679511
0.6923316583
1.5249089349
0.6347057594
0.6149205068
0.6680574575
1.6015010751

0.7218137255
0.6378628281
0.02
0.6444845796
0.6458333333
0.6767411301
0.642174027
0.6356934232
0.6149802891
0.660326087
0.6659815013

PR 9965:

binary_accuracy
loss
lr
mean_pred
mean_true
test_binary_accuracy
test_loss
test_mean_pred
test_mean_true
val_binary_accuracy
val_loss
val_mean_pred

0.7032608824
1.3332693654
0.02
0.6535641914
0.6457873238
0.6922808729
1.5722599631
0.6542453542
0.6153116809
0.6663950069
1.6660901248
0.6577351063

0.7172947304
0.6426681823
0.02
0.6440287049
0.6459865196
0.673455979
0.6477609455
0.648474915
0.6149802891
0.6576086957
0.671154305
0.6532216195

In this case, it looks like the typical model performed the same or a fraction of a percentage worse. I looked at a few specific examples such as inception resnet v2 weights with training locked and either a densenet or resnet block on the end, and they might have done ~1-2% better, but they didn't suddenly become the best models found in the search because of this change.
So in other words what I ran was mostly inconclusive (and possibly misconfigured), because I forgot to try locking random fractions of the imagenet weights and so trainable=True/False was either on all imagenet weights or none. If you think it'd be worthwhile to re-run with random proportions I might be able to give it a try. Sorry for the delay and the imperfections.

At the very least perhaps someone could make a PR with an example which demonstrates fine tuning correctly with the current API?
That might help other users deal with the current API, then if a PR like this eventually gets merged the example can be updated accordingly.

First of all, thanks for all the effort trying to solve this problem. Also, I should say that I am new to Keras, but I have some experience with Tensorflow.
Learning to work with the Applications module of Keras I bumped into this same issue. In the beginning, I thought I was overfitting, but when I tried to finetune VGG16 it did not overfit, so I started guessing that the problem could be the same as you are trying to fix.
I created a simple example that finetunes InceptionV3 to classify MNIST.
I am running it in inside of a Conda environment with:
Linux Mint 18.3
Python 2.7.14
tensorflow-gpu 1.7.0
cv2 2.4.13
numpy 1.14.1
Here is the code:
'''
This code has been adapted from the mnist_cnn.py example of Keras
and from the Applications examples on the Keras webpage
'''
from __future__ import print_function, division
import numpy as np
import keras
from keras.applications.inception_v3 import InceptionV3
from keras.models import Model, load_model
from keras.layers import Input
from keras.layers import Dropout
from keras.layers import Dense, GlobalAveragePooling2D
from keras import backend as K
from keras.datasets import mnist
import cv2

def load_mnist_dataset(num_train=None, num_test=None):
    '''from keras mnist example
    '''
    # input image dimensions
    img_rows, img_cols = 28, 28
    # the data, split between train and test sets
    (x_train, y_train), (x_test, y_test) = mnist.load_data()
    if K.image_data_format() == 'channels_first':
        x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)
        x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)
    else:
        x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
        x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
    x_train = x_train.astype('float32')
    x_test = x_test.astype('float32')
    # Center data
    x_train -= np.mean(x_train)
    x_test -= np.mean(x_test)
    # convert class vectors to binary class matrices
    y_train = keras.utils.to_categorical(y_train, NUM_CLASSES)
    y_test = keras.utils.to_categorical(y_test, NUM_CLASSES)
    # Select less data
    x_train, y_train = x_train[:num_train], y_train[:num_train]
    x_test, y_test = x_test[:num_test], y_test[:num_test]
    print('x_train shape:', x_train.shape)
    print(x_train.shape[0], 'train samples')
    print(x_test.shape[0], 'test samples')
    return x_train, y_train[:num_train], x_test, y_test

def resize_images(images, target_shape):
    print("resizing images...")
    images_resized = np.zeros((images.shape[0],) +
                              target_shape + (3,)).astype('float32')
    for i, im in enumerate(images):
        im = cv2.resize(im, target_shape, interpolation=cv2.INTER_LINEAR)
        im = np.tile(np.expand_dims(im, axis=2), (1, 1, 3))
        images_resized[i, ...] = im
    return images_resized

def get_model():
    input_tensor = Input(shape=(IMAGE_SIZE, IMAGE_SIZE, 3))
    # create the base pre-trained model
    base_model = InceptionV3(input_tensor=input_tensor,
                             weights='imagenet',
                             include_top=False)
    # add a global spatial average pooling layer
    x = base_model.output
    x = GlobalAveragePooling2D()(x)
    # let's add a fully-connected layer
    x = Dense(1024, activation='relu')(x)
    x = Dropout(.2)(x)
    # and a logistic layer -- let's say we have 200 classes
    predictions = Dense(10, activation='relu')(x)
    # this is the model we will train
    model = Model(inputs=base_model.input, outputs=predictions)
    return base_model, model

def set_conv_layers_to_not_trainable(base_model):
    # first: train only the top layers (which were randomly initialized)
    # i.e. freeze all convolutional layers
    for layer in base_model.layers:
        layer.trainable = False

def compile_model(model):
    # compile the model (should be done *after* setting layers to
    # non-trainable)
    optimizer = keras.optimizers.Adam(lr=0.00001,
                                      beta_1=0.9,
                                      beta_2=0.999,
                                      epsilon=None,
                                      decay=0.0)
    model.compile(loss=keras.losses.categorical_crossentropy,
                  optimizer=optimizer,
                  metrics=['accuracy'])

def fit_model(model, x_train, y_train, x_test, y_test):
    # train the model on the new data for a few epochs
    callbacks = None
    model.fit(x_train, y_train,
              batch_size=BATCH_SIZE,
              epochs=NUM_EPOCHS,
              verbose=1,
              validation_data=(x_test, y_test),
              callbacks=callbacks)
    return model

def test_model(model, x_test, y_test):
    score = model.evaluate(x_test, y_test, verbose=0)
    print('Test loss:', score[0])
    print('Test accuracy:', score[1])

def detect_bug(x_test, y_test):
    print('DYNAMIC LEARNING_PHASE')
    K.clear_session()
    model = load_model('tmp.h5')
    test_model(model, x_test, y_test)
    print('STATIC LEARNING_PHASE = 0')
    K.clear_session()
    K.set_learning_phase(0)
    model = load_model('tmp.h5')
    test_model(model, x_test, y_test)
    print('STATIC LEARNING_PHASE = 1')
    K.clear_session()
    K.set_learning_phase(1)
    model = load_model('tmp.h5')
    test_model(model, x_test, y_test)

def main():
    x_train, y_train, \
        x_test, y_test = load_mnist_dataset()
    x_train = resize_images(x_train, target_shape=(IMAGE_SIZE, ) * 2)
    x_test = resize_images(x_test, target_shape=(IMAGE_SIZE, ) * 2)
    base_model, model = get_model()
    set_conv_layers_to_not_trainable(base_model)
    compile_model(model)
    model = fit_model(model, x_train, y_train, x_test, y_test)
    # Store the model on disk
    model.save('tmp.h5')
    detect_bug(x_test, y_test)
    # In every test we will clear the session and reload the model to force
    # Learning_Phase values to change.

if __name__ == '__main__':
    # constants
    BATCH_SIZE = 32
    NUM_CLASSES = 10
    NUM_EPOCHS = 10
    IMAGE_SIZE = 150
    main()

I am running it in inside of a Conda environment with:
Linux Mint 18.3
Python 2.7.14
tensorflow-gpu 1.7.0
cv2 2.4.13
numpy 1.14.1
When I run it on Master (installed using pip install keras):
Using TensorFlow backend.
x_train shape: (60000, 28, 28, 1)
60000 train samples
10000 test samples
resizing images...
resizing images...
Train on 60000 samples, validate on 10000 samples
Epoch 1/10
60000/60000 [==============================] - 95s 2ms/step - loss: 2.5095 - acc: 0.2271 - val_loss: 2.5627 - val_acc: 0.1209
Epoch 2/10
60000/60000 [==============================] - 94s 2ms/step - loss: 1.8493 - acc: 0.4713 - val_loss: 4.6491 - val_acc: 0.0986
Epoch 3/10
60000/60000 [==============================] - 94s 2ms/step - loss: 1.6288 - acc: 0.5601 - val_loss: 5.3107 - val_acc: 0.0968
Epoch 4/10
60000/60000 [==============================] - 93s 2ms/step - loss: 1.5313 - acc: 0.6252 - val_loss: 7.1140 - val_acc: 0.1626
Epoch 5/10
60000/60000 [==============================] - 94s 2ms/step - loss: 1.4519 - acc: 0.6245 - val_loss: 5.0897 - val_acc: 0.1188
Epoch 6/10
60000/60000 [==============================] - 94s 2ms/step - loss: 1.4263 - acc: 0.6426 - val_loss: 7.3525 - val_acc: 0.1446
Epoch 7/10
60000/60000 [==============================] - 93s 2ms/step - loss: 1.3967 - acc: 0.6783 - val_loss: 8.0834 - val_acc: 0.1282
Epoch 8/10
60000/60000 [==============================] - 94s 2ms/step - loss: 1.3662 - acc: 0.6935 - val_loss: 8.2590 - val_acc: 0.1004
Epoch 9/10
60000/60000 [==============================] - 94s 2ms/step - loss: 1.3252 - acc: 0.6916 - val_loss: 9.6779 - val_acc: 0.1361
Epoch 10/10
60000/60000 [==============================] - 94s 2ms/step - loss: 1.2750 - acc: 0.7176 - val_loss: 8.2793 - val_acc: 0.1176
DYNAMIC LEARNING_PHASE
Test loss: 8.27926933746
Test accuracy: 0.1176
STATIC LEARNING_PHASE = 0
Test loss: 8.27926933746
Test accuracy: 0.1176
STATIC LEARNING_PHASE = 1
Test loss: 1.25008283653
Test accuracy: 0.7246

Note that here we have 10 classes so the validation accuracy stays around 1/10. Which it is similar to the 50% accuray you were mentioning before for binary class classification problems.
When I run in on the version from datumbox/keras/bug_fix/trainable_bn:
Using TensorFlow backend.
x_train shape: (60000, 28, 28, 1)
60000 train samples
10000 test samples
resizing images...
resizing images...
Train on 60000 samples, validate on 10000 samples
Epoch 1/10
60000/60000 [==============================] - 143s 2ms/step - loss: 1.9587 - acc: 0.3896 - val_loss: 1.3875 - val_acc: 0.6431
Epoch 2/10
60000/60000 [==============================] - 141s 2ms/step - loss: 1.6391 - acc: 0.4478 - val_loss: 1.3699 - val_acc: 0.5701
Epoch 3/10
60000/60000 [==============================] - 141s 2ms/step - loss: 1.5103 - acc: 0.5982 - val_loss: 1.1702 - val_acc: 0.7386
Epoch 4/10
60000/60000 [==============================] - 140s 2ms/step - loss: 1.4111 - acc: 0.6180 - val_loss: 1.0836 - val_acc: 0.7739
Epoch 5/10
60000/60000 [==============================] - 136s 2ms/step - loss: 1.3594 - acc: 0.6306 - val_loss: 1.1120 - val_acc: 0.7061
Epoch 6/10
60000/60000 [==============================] - 140s 2ms/step - loss: 1.3693 - acc: 0.5404 - val_loss: 1.0732 - val_acc: 0.7784
Epoch 7/10
60000/60000 [==============================] - 140s 2ms/step - loss: 1.2663 - acc: 0.7187 - val_loss: 1.0213 - val_acc: 0.7933
Epoch 8/10
60000/60000 [==============================] - 140s 2ms/step - loss: 1.2458 - acc: 0.6815 - val_loss: 0.9229 - val_acc: 0.8242
Epoch 9/10
60000/60000 [==============================] - 140s 2ms/step - loss: 1.1706 - acc: 0.7479 - val_loss: 0.8795 - val_acc: 0.8247
Epoch 10/10
60000/60000 [==============================] - 140s 2ms/step - loss: 1.1648 - acc: 0.7358 - val_loss: 0.9132 - val_acc: 0.8206
DYNAMIC LEARNING_PHASE
Test loss: 0.913178708649
Test accuracy: 0.8206
STATIC LEARNING_PHASE = 0
Test loss: 0.913178708649
Test accuracy: 0.8206
STATIC LEARNING_PHASE = 1
Test loss: 1.11110808325
Test accuracy: 0.7584

It seems to me that the patch must be working, but I do not understand very well why when I set STATI LEARNING_RATE =1 the test accuracy drops to 0.7584.
I still need to do some work to find a good set of hyper-parameters that train better (some ideas?), but I thought the example could be useful.
Hope it helps

@ahundt:

So in other words what I ran was mostly inconclusive (and possibly misconfigured), because I forgot to try locking random fractions of the imagenet weights and so trainable=True/False was either on all imagenet weights or none.

Thanks for running and posting the experiments. I can't say I follow fully your setup but if the run was misconfigured then this is probably why the results were inconclusive.

At the very least perhaps someone could make a PR with an example which demonstrates fine tuning correctly with the current API?

Completely agree. Currently the documentation provides an incorrect example and that's why so many people are affected. If @fchollet believes that this can done with the current API, he should update the documentation.
@pacorofe:

Note that here we have 10 classes so the validation accuracy stays around 1/10. Which it is similar to the 50% accuray you were mentioning before for binary class classification problems.

Yeap, spot on. This is somehow an extreme example because you take weights from ImageNet to initiate a classifer for MNIST. The distributions of the two datasets are so far away that you get quite extreme results.

It seems to me that the patch must be working, but I do not understand very well why when I set STATI LEARNING_RATE =1 the test accuracy drops to 0.7584.

Getting lower accuracy when you set STATIC LEARNING_PHASE=1 is expected because the entire network is run on training mode. Thus the BNs use mini-batch statistics instead of the learned parameters.

https://medium.com/@joeyearsley/the-batch-normalisation-in-keras-is-not-broken-358b5e46af54

@joeyearsley thanks for your post. I checked the suggested solution but I still feel this does not cover all cases. Here are the two main points that in my opinion remain not addressed:

Setting the BNs to trainable and resetting their statistics is not a good solution because the next convolution weights (which remain with trainable=False) were estimated using the initial BN statistics. Resetting their numbers will adapt them to the distribution of the new data but as I describe on the blog post the next convolutions and non-linearities will never get to adapt in this change because they are frozen. This can have very bad effects especially if the new BN stats cause the output to become positive (from negative) as it will affect the next RELUs. Also since the default momentum on BNs can't be changed after their definition, you will require a large number of iterations to adapt the moving mean/var of the BNs.
The problem does not only appear on the pretrained models included in the applications, but also when you load any other pretrained model stored on disk. Imagine that you might have trained a classifier from scratch on one domain and now you want to use it as initial weights on a different problem. Just updating the applications API will not fix the problem as most likely you need to apply the same patch on load_model().

Recently there was an interesting discussion on twitter and other people have pointed out this problem in the past and not all Deep Learning Frameworks are affected. At any case I think it is clear that the documentation is very misleading and as you said needs to be updated. Personally I think that there is no reason to scale differently the data when a BN layer is frozen. There is no theoretical or mathematical justification on why such a different rescaling policy takes place when the layer is frozen. At any case I hope someone will patch this on the future as the effects of the problem are likely to affect lots of people and they might not be aware as it has similar symptoms as overfitting.
@joeyearsley thanks for your post. I checked the suggested solution but I still feel this does not cover all cases. Here are the two main points that in my opinion remain not addressed:

Setting the BNs to trainable and resetting their statistics is not a good solution because the next convolution weights (which remain with trainable=False) were estimated using the initial BN statistics. Resetting their numbers will adapt them to the distribution of the new data but as I describe on the blog post the next convolutions and non-linearities will never get to adapt in this change because they are frozen. This can have very bad effects especially if the new BN stats cause the output to become positive (from negative) as it will affect the next RELUs. Also since the default momentum on BNs can't be changed after their definition, you will require a large number of iterations to adapt the moving mean/var of the BNs.
The problem does not only appear on the pretrained models included in the applications, but also when you load any other pretrained model stored on disk. Imagine that you might have trained a classifier from scratch on one domain and now you want to use it as initial weights on a different problem. Just updating the applications API will not fix the problem as most likely you need to apply the same patch on load_model().

Recently there was an interesting discussion on twitter and other people have pointed out this problem in the past and not all Deep Learning Frameworks are affected. At any case I think it is clear that the documentation is very misleading and as you said needs to be updated. Personally I think that there is no reason to scale differently the data when a BN layer is frozen. There is no theoretical or mathematical justification on why such a different rescaling policy takes place when the layer is frozen. At any case I hope someone will patch this on the future as the effects of the problem are likely to affect lots of people and they might not be aware as it has similar symptoms as overfitting.

I think this all still indicates there is an issue: the keras API could be more clear and easy to use when fine tuning, with an explicit and easy to use choice between the options mentioned by @ppwwyyxx (plus perhaps Group Normalization).  Anyone have time to design one?
@ppwwyyxx are there any publicly available pretrained weights with group normalization?
I believe typical users won't have the resources to train ImageNet weights from scratch. GroupNorm is currently only a pull request in keras-contrib, and not in keras.
It sounds like you're fairly certain that GroupNorm will work better for most use cases, you made a pretty strong statement. :-)

This discussion is at a good decision point, @titu1994 that sounds reasonable. I only wish there were a better word than mode, which is used in so many programming situations that it can lose all meaning.
François may not notice this again unless again unless he is pointed to the discussion through some other channel, he probably gets 100s or 1000s of @ mentions each day at this point.

@ahundt I agree with the semantics of mode. If the general idea about that functionality is pushed forward, then some better keyword can be discussed.
Whatever it is called, I feel it should not be an integer number like before. It caused confusion about whether mode=1 was equal to layer norm or not. This time, in my opinion, semantically meaningful word (s) which attempt to describe what that mode will do should be used.

Good ideas. If there is concensus about the API I am happy to update the PR. It would be good if someone can ping @fchollet  to give his input; I want to avoid wasting time again if he has no plans to merge this.

@titu1994 A few well named string modes would be more than good enough for most users, I think that's the best way to go.
@datumbox The easiest option may be creating a new branch and a new PR, with a link pointing back to this.
Considering the variability in approaches, an easy way to "code your mode" may have value but I think that can be left as future work.

I'd agree with datumbox's original patch of changing the current behavior of Keras BN layer during training when the BN layer is frozen, e.g., using the pretrained moving average instead of mini-batch statistics.  By doing this as a default when freezing this layer, at least in transfer learning you are not seeing that huge difference between training and predicting using the same training dataset. I've seen a lot of confusions from different discussions forums when people are trying to using pre-trained ImageNet models and I have encountered it myself. The current behavior is really not intuitive at all, at least to me. If people want different mode switch, that's fine to add from this patch. Just my two cents.

We are still having new and old issues related to BN/finetuning:
https://github.com/keras-team/keras/issues/6977
https://github.com/keras-team/keras/issues/9214
https://github.com/keras-team/keras/issues/10554
https://github.com/keras-team/keras/issues/10214
etc.. etc..
Probably there is something not so user friendly or under documented for Keras expected average usability:
@ppwwyyxx made a good analysis in https://github.com/keras-team/keras/pull/9965#issuecomment-398183440
Also there is some "downstream" effect in tf.keras etc...

For those of you who use the fork, there is a new sync with Keras 2.2.2 here.

The Batch Normalization layer of Keras is NOT broken !
please just change this lines: (in your source code written in blog.datumbox)
base_model = ResNet50(weights='imagenet', include_top=False, input_shape=(224, 224, 3))
l = Flatten()(base_model.output)
predictions = Dense(2, activation='softmax')(l)
model = Model(inputs=base_model.input, outputs=predictions)

to this:
base_model = ResNet50(weights='imagenet', include_top=False, input_shape=(224, 224, 3))
input = layers.Input(shape=(224, 224, 3))
x = base_model(input)
l = Flatten()(x)
predictions = Dense(2, activation='softmax')(l)
model = Model(inputs=input, outputs=predictions)

you should define input as a separate layer

Just wanted to chime in and share my experience. I ran some code last year training a Resnet50 model (Keras 2.0.2, Tensorflow 1.2) and got pretty good results with a validation set accuracy of 98%.  Then this year I wanted to make sure I could still reproduce the results (same data). I reran the code on my old python virtual environment (Keras 2.02, TF 1.2) and I got 97% accuracy. Good enough for me. I then wanted to check if I could reproduce the results with and updated Keras and TF.
I created a new python virtual environment with Keras and Tensorflow updated (Keras 2.2.2, TF 1.9) and updated some of the dependencies and resolved any breaking API changes (Resnet50 model last layer changed). I trained but noticed that my model was over fitting very quickly (after about 2 epochs). I tried multiple times and tweaking the model but it never reached a validation accuracy greater than 92%.
I searched around and found 8430 which lead me to look into batch normalization which lead me to this thread.  I applied the patch in this pull request, and reran.  I am getting validation set accuracy of about 96.6% with the exact same model.
There is still a discrepancy that I can try to hunt out, but for now I am happy. Thanks for this work!

@30yavash I think you misunderstood on what I write on the blog. How you pass the input has nothing to do with this problem.

@JeremyBYU,

The difference between 97% and 98% may not be statistically significant. What is the size of your validation set?

Try reducing your learning rate. There is a general trend in TF/CuDNN to make kernels faster at a cost of reduced numerical accuracy.

Thanks @ozabluda. Yeah the 97 vs 98 difference doesn't bother me (nor even the 96.6).   IIt was the drop to 92% that bothered me on the updated Keras 2.2.2.  Validation set is kind of small (about 400 images, 6 classes).   I will try your suggestions, thanks!

even with with only 400 images, the difference between 92% and 97% is probably statistically significant (depends on the confusion matrix). 97-vs-98 is not.

@30yavash , even if what you suggest had been a valid solution (which is not related to the issue at all, according to @datumbox ), your suggestion yields the following error:
AttributeError: Layer resnet50 has multiple inbound nodes, hence the notion of "layer input" is ill-defined. Use get_input_at(node_index)instead.

@datumbox thank you for your contribution i will try it out. i have a question though: why not simply reinitialize the layer completely when importing a model and making it trainable?.
you say that one should use the statistics the BN layer learned whilst training the original model but that resetting the statistics and having BN learn the new statistics is not good "because the next convolution weights (which remain with trainable=False) were estimated using the initial BN statistics.".
but if i understand correctly - the next layers are "Expecting" "whitened" data, it's the data distribution the matters, not the rescaling operation itself.
so why wouldn't it be more proper to make the BN layer trainable and reinitialize statistics and momentums to make the BN layer adjust to the new data distribution quickly?.
however, no one can argue with the results of you approach and i would like to know if there's a specific small change to the BN layer definition to make to apply your patch to avoid the being dependent on you publishing anew for each keras version?.
Thank you very much!

@30yavash - update - your code works as-is when I copy-pasted,
However, validation accuracy is extremely low. Much lower than VGG16
129/129 [==============================] - 203s 2s/step - loss: 0.1954 - acc: 0.9651 - val_loss: 9.8208 - val_acc: 0.0334
129/129 [==============================] - 192s 1s/step - loss: 0.0266 - acc: 0.9929 - val_loss: 10.1189 - val_acc: 0.0276

Am I understanding/summarizing the issue correctly?

Assume one has loaded in a pretrained model with batchnorm, e.g., ResNet50.

If a BatchNormalization's .trainable property is set to False, none of its weights will be updated, i.e., the rescaling variables and the mean/variance variables will be static. At training time, a second set of mean/variance variables will be tracked, updated with each new batch, and used at training time.

At test time, the old mean/variance variance from the original pretrained model will be used instead of the newly computed mean/variance from training time. This results in the accuracy at test time being lower than at training because all of the scales of the variables are messed up.

The solution to this is to use K.set_learning_phase(0) and/or set training=False when loading/constructing a BatchNormalization layer. If training=False, then the moving average/std computed at training time is used instead of the moving averages loaded from the pretrained model.

If this is correct, then that works, but does invite the question -- is there a use case for the current setting of setting BatchNormalization's .trainable property to False, aside from preserving the semantics of .trainable? Semantics aside, I can't think of any reason why someone would want to use the old mean/variance instead of the one for the training set they are trying to use.

@jmhessel
On point 1: During training time, the mean/variance is estimated based on mini-batch statistics. The rest of your comments are correct.
On point 2: Spot on.
On point 3: Setting the learning phase to 0 is not exactly solution as you won't be able to train the network at all. Nevertheless you are right to say that the BN will use the mean/variance of the preloaded model.
The reason what you want to use the old mean/variance of the network for those BNs that are frozen is because the layers after them were trained with specific mean/variance in mind. Changing them means that the weights of the next convolutions won't adjust to the changes of the statistics. Even worse, the RELU non linearities will produce completly different ouputs. This is the reason why many people report so low accuracies.

@datumbox Thanks for the info!!! I think that the main thing to avoid, then, is using training=True and trainable=False, which, unfortunately, seems to be the default API usage if you don't think about it. I guess for now I will freeze everything at training and test time in BN layers from pretrained conv nexts and hope that the optimization process can fix this weirdness.

I tried fchollet's solution that he mentioned in this issue, both for training and testing. The model contains BN layers.
Now I am loading this model and want to test the model performance on the test set using evaluate_generator.
When I set K.set_learning_phase(0) before calling the evaluate_generator, it is performing weirdly. I get very low accuracy (around 30%). However, I would expect to get at around 80% accuracy.
Even though it is a very weird approach, I tried setting K.set_learning_phase(1). Seems like I get between 70% - 85% accuracy but it is changing at each trial. This seems also weird because I would expect the model to output the same accuracy on the test set.
I have another script (customized version of the tensorflow tutorial) for testing and it is giving me a fixed accuracy which is 73%.
So I think either evaluate generator is performing weirdly or  K.set_learning_phase(0) before testing does not solve the problem.

I have been following this discussion for quite a while now and it still gives me headaches.
My current situation is the following: We would like to use DenseNet as a classifier and freeze the first blocks and leave the remaining blocks trainable. When we first tried this approach we were completely surprised by the poor results (actually it did not detected anything) and it took us several working days to find out, that the BatchNormalization layer is causing this issue. In conclusion, this behavior is not user friendly and especially since it is not even documented somewhere. So thank you @datumbox for clarifying things!
Now I wonder how can freeze the blocks in the DenseNet. I do not really want to copy&paste the code from Keras github but it seems like this is the only solution. Should I rather user the set_learning_phase method or set the training parameter to False for those BN layers which I want to freeze?

This is my first post ever. I just had to share my experience, although I'm a little late to chime in.
First of all, @datumbox - thank you so much for all your efforts on this issue. Especially for newcomers with a background in a field other than computer science, your explanations made a lot of sense. It's fair to say that you might as well have saved the entire project that I'm currently working on. I'm trying to implement "BN heavy" pretrained networks and all of the performance measures suggested that the models were overfitting. But something seemed suspicious, since I have a lot of data (a couple of million images) to answer a rather simple question.
I first decided to freeze the first 133 layers of the Inception V3 network almost exactly as its described here with for layer in model.layers[:133]: layer.trainable = False etc. Accuracy was around 87%, taking quite some time to converge. I then set all layers layer.trainable = True and very quickly achieved accuracy rates of over 98%. Would this be a behavior consistent with your findings @datumbox ?
If so, I don't understand, why this would not be implemented? Although I read @fchollet 's explanation multiply times, this exceeds my comprehension (although it is very likely that I'm just too stupid). I will try to setup keras with the patch - unfortunately have not managed to get it to work just yet.
I would really appreciate if this would be looked into again. Cheers. Sebastian

@datumbox Could you please take a look at my problem whether it can be the BN issue?
Thank you for your time!
George

@AGFoersch Hi Sebastian. I can't be sure if you are being affected by this policy. If your training accuracy is high while the validation one is low and you use a pretrained network and freeze large number of layers then you are likely to be affected. The problem masquerades itself as overfitting, but it might be caused by the BNs. I think the best way to confirm is either to set a static learning_phase to 1 one the very top of your script (that's not a solution just a way to confirm the problem) or install the following patch and retrain:
pip install -U --force-reinstall --no-dependencies<EMAIL_ADDRESS>
Though I've been maintaining this fork for some time now (and I plan to continue doing so), please note that you will be messing with custom forks which is always suboptimal.
@george-gssy Based on your description, you might be affected. Try the above patched version to confirm.

@datumbox Thank you, I will then try and will report the results.

https://medium.com/@joeyearsley/the-batch-normalisation-in-keras-is-not-broken-358b5e46af54

The author deleted this Medium story
Browse for great reads on Medium.

Yes I did, as I wrote it extremely one sided, were the side was the batch norm stats should be zero mean and unit variance, which if you use a different input distribution to original network you should learn the batch norm stats along with the final few layers.
@ppwwyyxx highlights all the approaches in a short simple manor.

Thanks @datumbox for both the blog and the PR/fork.  After spending more time than I want to admit trying to figure out why keras wasn't giving me transfer learning results comparable to TensorFlow's retrain.py script, you gave both the answer and the fix.
After installing your fork, my best validation accuracy went from 77% to 93%.
A note about this issue in the documentation's fine-tuning example might save others confusion like this in the future.

Is it safe to assume this also affects VGG16 and that I should install this Keras fork? I'm trying to fine tune and while training does very well (including validation), and even testing gives good results, when I try to use the model to predict, it's horribly wrong. Any hint please ?

@Osumann No --- VGG16 doesn't have batchnorm layers.

When I set static learning_phase to 0, I get accuracies much close to the training (basically the same) but when I install the patch, it goes back to having a very low validation accuracy in comparison. Is there a way to check that the patch is installed and working?

I also have this problem. My problem is that I cant install the patch on the server I'm working on. Is there a hack to achieve the same results as from using this patch?

I have one question to ask:
My idea is coming from during inference phase we have to freeze the mean and variance and apply it with
the statistic from training (moving mean and moving variance )
My proposal  is following steps but not sure if it's necessary or going to work:
(1) In the function call set the training =False when testing and training=True when training.
(2)Use hsitory_weight = ModelCheckpoint(filepath="weights.{epoch:02d}.hdf5",save_weights_only=True,save_best_only=False) to store the normalization weight(including moving average and variance of course)
(3) initialize it with what we get from step (2)
I am not sure if there's any method could be quicker?

@fchollet
I found that if I set the trainable= False (batch_normalization)
All parameters in batch normalization layer would stop updating
However,if I use what you said up there set the training =False
The moving mean and moving variance would be fixed and the others would still change while training
I would like to know what's the purpose to make it different?
Thanks!

Is there a reason why this fix was not implemented?

The tensorflow repo would've been a better place to post this but there is no existing threads there
Here is a very clever trick that works for tensorflow 2.0 in eager mode (otherwise probably wont work)
class BatchNormalization(tf.keras.layers.BatchNormalization):
    """
    Make trainable=False freeze BN for real
    """
    def call(self, x, training=False):
        training = tf.logical_and(training, self.trainable)
        return super().call(x, training)

The generally acceptable behavior for BN is (in pytorch/fast.ai ?? correct me if i'm wrong)

training normal: use batch-wide stats, then update weights
training frozen: use pre-trained stats, no update
eval: use pre-trained stats

but keras does this

training normal: use batch-wide stats, then update weights
training frozen: use batch-wide stats, no update
eval: use pre-trained stats

The first version makes more sense because we usually freezes batch_norm and conv layers together
The keras version works better when you have frozen batchnorm sandwiched between unfrozen conv (does anybody really do this ???)
lucky the desirable outcome can be easily replicated in tf 2.0 eager mode with
tf.logical_and(training, self.trainable)
This should work in keras too but I did not test it. I also did not test this in non-eager mode. if this doesn't work, you can achieve the same with more complicated code like this in keras_resnet
This solution is probably better suited for people that don't want to run a forked keras repo just for batchnorm.

Just spent two days on this before finding this thread.
One solution is the pre-computer the output of the ResNet part and then use that as the input to a new graph with just the last few layers.

Here is a simple workaround for those who want a quick fix for the BatchNorm layer (and don't mind some additional overhead):
from tensorflow.python.keras import backend as K
from tensorflow.python.keras import layers as kl

class BatchNormalization(kl.BatchNormalization):
  def call(self, inputs, training=None):
    true_phase = int(K.get_session().run(K.learning_phase()))
    trainable = int(self.trainable)
    with K.learning_phase_scope(trainable * true_phase):
      ret = super(BatchNormalization, self).call(inputs, training)
  return ret

Note that you'll have to use/ship your own BatchNorm i.e. when loading a model from disk, this class has to be passed as an external object.

Note also that this works only for Tensorflow's Keras (as learning_phase_scope is not available in the framework-independent version)

Thanks to @datumbox for keeping careful track, and raising awareness for this issue.

@datumbox Again, many thanks.
I don't understand quite the code. But I guess from your blog that your fix actually "freezes" all the BN layers during the training phase in transfer learning. Is that correct ?
But if so, it is still not using the values of the moving mean and variance of the new dataset, as you suggested, right ?
Cheers!

The generally acceptable behavior for BN is (in pytorch/fast.ai ?? correct me if i'm wrong)

training normal: use batch-wide stats, then update weights
training frozen: use pre-trained stats, no update
eval: use pre-trained stats

but keras does this

training normal: use batch-wide stats, then update weights
training frozen: use batch-wide stats, no update
eval: use pre-trained stats

This is what I want to know as well.

@spalaciob can you provide an example of how to use the custom BatchNormalization class on a loaded model?

@ellolo The key thing for me is to note that models in keras.applications support passing a layers keyword argument which overrides the layers module used when building the model. This means we can patch layers with our custom frozen batch normalization layer before passing.
from keras import applications, layers

class FrozenBatchNormalization(layers.BatchNormalization):
    def call(self, inputs, training=None):
        return super().call(inputs=inputs, training=False)

# Keep a copy of the original class
BatchNormalization = layers.BatchNormalization

# Patch the class temporarily
layers.BatchNormalization = FrozenBatchNormalization

# Build the model with our patched version of layers
pretrained = applications.MobileNetV2(layers=layers, weights='imagenet')

# Undo the patch
layers.BatchNormalization = BatchNormalization

Hope this helps!

@faustomorales thanks! If I understand correctly how keras.applications works: the layers argument is initially assigned keras.layers via the keras_modules_injection decorator. We can then override keras.layers temporarely to define a new layer.

@ozabluda @taehoonlee @farizrahman4u @Dref360 @fchollet
when the patch will be merged?

Are there any plans to merge this path? It would make my work on semantic segmentation so much easier!

Hi, may I know what is the correct way of finetuning pre-trained models with BN layers on a new dataset?
From what I read so far, it seems like the Inception V3 finetuning example in Keras documentation is not the correct way, due to the way BN layers behave when frozen.

@Toukenize I saw there is a note in https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/keras/layers/normalization_v2.py that suggest in tensorflow 2.0, setting trainable=False should cause the bn layer to use the saved mean and variance rather than calculating a new one for the batch. So maybe give that a try.
In my case, I just want to do some simple transfer learning (output of ResNet50 plus a few trainable dense layers). My workaround was this:

Create the base model (e.g. ResNet50 + ImageNet) and use it to generate the vectors
Create a separate model for the Dense layers with vectors as input and train it. (Let's call it model_dense)
Join them together using functional interface.

E.g.
model_base = ResNet50(.....)
train_vectors = model_base.predict(...)
test_vectors = model_base.predict(...)
model_dense = ...
model_dense.fit(train_vectors,...)
joined = model_base(model_dense.outputs[0])
model_joined = Model(model_base.inputs[0], joined)

Note that if you want to freeze the resulting graph for production etc there is a bug in tf.keras (1.13.1, 1.14.0 and 1.15.0 nightly) where the conversion to frozen graph either doesn't work (1.14.0) or works but creates a frozen graph that gives errors on import (1.13.1 and 1.15.0.dev20190807 nightly). See my workaround here: https://github.com/tensorflow/tensorflow/issues/31331#issuecomment-518655879

@Toukenize I saw there is a note in https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/keras/layers/normalization_v2.py that suggest in tensorflow 2.0, setting trainable=False should cause the bn layer to use the saved mean and variance rather than calculating a new one for the batch. So maybe give that a try.
In my case, I just want to do some simple transfer learning (output of ResNet50 plus a few trainable dense layers). My workaround was this:

Create the base model (e.g. ResNet50 + ImageNet) and use it to generate the vectors
Create a separate model for the Dense layers with vectors as input and train it. (Let's call it model_dense)
Join them together using functional interface.

E.g.
model_base = ResNet50(.....)
train_vectors = model_base.predict(...)
test_vectors = model_base.predict(...)
model_dense = ...
model_dense.fit(train_vectors,...)
joined = model_base(model_dense.outputs[0])
model_joined = Model(model_base.inputs[0], joined)

Note that if you want to freeze the resulting graph for production etc there is a bug in tf.keras (1.13.1, 1.14.0 and 1.15.0 nightly) where the conversion to frozen graph either doesn't work (1.14.0) or works but creates a frozen graph that gives errors on import (1.13.1 and 1.15.0.dev20190807 nightly). See my workaround here: tensorflow/tensorflow#31331 (comment)

@geometrikal Thanks for your prompt response. Does that mean I got to extract the train and test vectors every epoch, if I'm doing data augmentation?
My use case is slightly more complicated, meant for a sequence of images (video classification), the actual model is as such:
pretrained = ResNet50(include_top=False, weights='imagenet', pooling='max')

for layer in pretrained.layers:
    layer.trainable = False

nn_input = Input(shape=(SEQ, 224, 224, 3))
x = TimeDistributed(pretrained)(nn_input)
x = Reshape(target_shape=(SEQ, pretrained.output_shape[-1]))(x)
x = CuDNNLSTM(128, return_sequences=True)(x)
x = GlobalAveragePooling1D()(x)
x = Dense(128, activation='relu')(x)
nn_output = Dense(1, activation='sigmoid')(x)   # To extend to multi-label, change the number of hidden units

model = Model(inputs=nn_input, outputs=nn_output)

Do you have any suggestion on other work arounds for this?

@Toukenize I was just talking about this with a colleague, who said that they use the normal keras (not tensorflow.keras) ResNet50 for transfer learning and have no problem. I have not tried that myself however, but this issue suggests setting layer.trainable = False should work for plain keras:  https://github.com/keras-team/keras/issues/7085

Does the issue affect the models that are saved and loaded using model.save() and load_model?

Cannot believe that this patch hasn't been merged after 1.5 years! Keras people seem to be very arrogant to accept others' delicate solutions; ignoring 50% performance drop, but saying 5% extra cost without benchmark evidence. Really angry. This stupid Keras issue costs me two weeks.

@ybsave, datumbox has previously said to use the following command to install his fork:
pip install -U --force-reinstall --no-dependencies<EMAIL_ADDRESS>

@TheGuywithTheHat Thank you for your help. After install his fork, I test the codes in @datumbox 's  blog. But the results are still not fixed...
I tested the Tensorflow's batch normalization. As long as set is_training to False during testing, the performance is the same as during training.
 I would switch to pure Tensorflow instead. Thanks @datumbox 's excellent solution and @TheGuywithTheHat 's help.
  You should not define it else the behaviour of the patch won't be what you expect.
Also I understand the frustration and your point of view but if you think about it, Keras is a good solution and at the end of the day it is an open source project that anyone can fork and adjust to their needs. That's what we do here. It's up to the owner to merge or not PRs if they meet their criteria. Maybe one day someone will provide a more elegant solution and it will be part of Keras. :)

@datumbox Thank you for your kind comments. My previous codes used tensorflow.keras, but not the patched keras. After fixing it, I got the same results as shown in your blog.
Thank you for your kind and warm words. I just lose confidence on the Keras people. "After training, testing on the same training data should produce the same results in the last training round". This is a common sense to me. But I think Keras people lack common sense. You give very detailed explanations, experiments, fixes, and discussions. But they do not listen. I cannot understand their ignoring the huge performance drop (>30% in my own work) but attacking your solution for tiny (<5% extra cost, and no experimental evidence).
If you will produce new Keras version, I will definitely use it. But without your patch, I will never use Keras anymore. Thank you so much for saving my world!

@datumbox Do you have a plan to also produce a fix to the tensorflow.keras? Thank you.

@ybsave Try tensorflow 2.0, I think the behavious has been changed there

@geometrikal Thanks for the suggestion. Have you tried @datumbox 's example codes on his blog by changing keras to tf.keras? In my test, the problem still exists on Tensorflow 2.0. Does this problem disappear in your test?

@ybsave I haven't done any more tests so I'm not sure of the status of this. I just pre-calculate the vectors as per my workaround.

@ybsave can you share keras codes please? I haven't seen complete code, still dont know where I will put this pack in code? "install pip install -U --force-reinstall --no-dependencies<EMAIL_ADDRESS>thank you

@ybsave Do you know what the earilier version of tf has the fix for this?

It seems we could safely use tf.keras from the newest Tensorflow, instead of waiting for arrogant keras people fixing this.

I used Tensorflow 2.0.0-rc0, when I fine-tuning resnet have same problem.

In tensorflow2 GPU there is still this problem occurring, I have to use @faustomorales approach in order to fix the issue. But another issue is that it increases prediction time. The prediction time gone from 17ms to 26ms easily.
I don't know why transfer learning is still a hack?
Is there some article teaching fine-tuning with CNN and dealing with BatchNorm stuff? I don't want to be a master in BatchNorm!

After some try out, I think that a fesible yet possible solution is this (pseudo code, taking inceptionV3 as example):
1. K.set_learning_phase(0) # test mode, freezing everything
2. myModel = InceptionV3(weights='imagenet', include_top=False, input_shape=(299,299,3)) # load pre-trained model and weight
3. nn_inputs = myModel.input # save input for layer use
4. for layer in myModel.layers:
        layer.trainable = False # freeze the weights in each layer. Notice that BNs are also freezed
5. K.set_learning_phase(1) # switch to training mode
6. # build the top layers
   myModelOut = myModel.output
    myModel = GlobalAveragePooling2D()(myModelOut)
    myModel = Dense(D, activation="relu")(myModel)
    myModel = Dense(10, activation="softmax")(myModel)
7. # build the whole model
    finalModel = Model(inputs=nn_inputs, outputs=myModel)
8. # verify the model structure and parameters
    print(model.summary())

@captainst Hello, this way can work. But after I save the finalModel, and want to continue a training through load the saved_model, the problem still there. (╥╯^╰╥)

@captainst your approach doesn't work, when you want to fine-tune top-k layers part of your base-model that may have BN layers.

@lovejing0306

I used Tensorflow 2.0.0-rc0, when I fine-tuning resnet have same problem.

@off99555

In tensorflow2 GPU there is still this problem occurring, I have to use @faustomorales code in order to fix the issue.

For those using TensorFlow 2.0 and trying to fine-tune resnet, inceptionv3, etc., the problem seems to persist due to the injection of tensorflow.python.keras.layers. This references the TF 1.0 behaviour batch normalisation in keras_applications when calling layers.BatchNormalization (for example, in inceptionv3).
Similar to what @faustomorales suggested, I found that simply injecting tf.keras.layers references the TF 2.0 behaviour batch normalisation and fixes this issue (see here for the change in behaviour). When loading models from tf.keras.applications simply add the argument layers=tf.keras.layers. For example:
import tensorflow as tf

pretrained = tf.keras.applications.inception_v3.InceptionV3(
    layers=tf.keras.layers, weights='imagenet')

@datumbox,
If one day I meet you, I promise I'll pay you a beer.
You have not idea how this thread saved me. I've spent two weeks struggling with transfer learning without having any clue of why it was going completely wrong. A simple transfer learning, it was non-sense.
Unfortunately, none of the suggested workaround worked for me. I'm currently using TF 2.0.0.
The only network that you won't have any headaches is VGG once it has not batch norm layers,
For all others, what works in my case is to do transfer-learning on a 2-step process: Extract embeddings first (into tfrecords shards or not, it is up to you) for further classification.
@rpeloff , your workaround worked for me on TF 1.15.0 but not at TF 2.0.0, but thanks anyway.

I can't believe this is not fixed yet
I have spent 1 week to finally find what's going wrong with my code
Thanks @datumbox , I'll buy you a beer too

To add to the suggestion made by @faustomorales, I found it useful to actually update the call function of the BatchNormalisation layer base class, after creating the layers, rather than creating a subclass. This means that you don't have to modify your original model code for loading. You could use something similar to the following:
import copy tf.keras.layers.BatchNormalization._original_call_function = copy.deepcopy(tf.keras.layers.BatchNormalization.call) # We potentially need this to be a different object to avoid recursion                         def non_trainable_call_function(_self, inputs, training=None):                             return tf.keras.layers.BatchNormalization._original_call_function(_self, inputs=inputs, training=False)                         tf.keras.layers.BatchNormalization.call = non_trainable_call_function
This means that when any BatchNormalization layer is called, the moving mean and variance are calculated, rather than the minibatch mean and variance.
Alternately, you could recurse through only your frozen model and individually set the call functions of the layer if it is an instance of BatchNormalization, rather than changing the global behaviour.
This is a problem that needs to be addressed IMO since training with parts of the model frozen that include BatchNorm is a very difficult thing to spot as a problem and can lead to unnecessary wastage of time trying to identify the problem.

Can anyone simply answer me about If I am using pre-trained model Densenet121 that has BN layers but I am not freezing any layer and I am using tf.keras instead of standalone keras, should I be affected by the batch normalization behavior?

Dear all,
Can anyone write a summary on last situation?
if you are using tensorflow 2.x,  Does below code will be enough  without @datumbox 's patch?
(with partly frozen structures, fine tuning bottom layers)
import tensorflow as tf
pretrained = tf.keras.applications.inception_v3.InceptionV3(
layers=tf.keras.layers, weights='imagenet')
and also what about if you are using tensorflow 1.x?

I suggest one solution by solving the fundamental problem.
The problem
: BN layer uses average and variance of a mini-batch while training, but we want it to use trained moving_average and moving_variance while training.
Summary of the solution

Copy a trained model
While copying, BN layers are replaced by Lambda layer that normalizes the input from the previous layer in the same way as the BN layer does. (Lambda layer uses moving_average, moving_variance, gamma, beta from the trained model)
set trainable=False

Validation code
`import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
import tensorflow.keras.backend as K
from tensorflow.keras.layers import Dense, BatchNormalization, Lambda
get base_model
base_model = tf.keras.Sequential()
base_model.add(Dense(8, activation='relu', input_shape=(1,)))
base_model.add(BatchNormalization())
base_model.add(Dense(1))
base_model.compile(loss="mse")
get simple dataset
X = np.arange(0, 1, 0.1).reshape(-1, 1)
Y = np.copy(X)
fit base_model
base_model.fit(X, Y, epochs=500)
make frozen_model for validation
for BatchNormalization layer, we reconstruct the layer with moving_average, moving_variance, gamma, beta from base_model
the normalization equation of the BN layer is:: gamma*[(z-moving_mean)/sqrt(moving_variance + epsilon)] + beta
frozen_model = tf.keras.Sequential()
frozen_model.add(base_model.layers[0])
frozen_model.add(Lambda(lambda z: base_model.layers[1].gamma*((z-base_model.layers[1].moving_mean)/K.sqrt(base_model.layers[1].moving_variance + base_model.layers[1].epsilon)) + base_model.layers[1].beta))
frozen_model.add(base_model.layers[2])
frozen_model.trainable = False
verify if the output from the two models are the same
yhat = base_model.predict(X).flatten()
yhat2 = frozen_model.predict(X).flatten()
print(f"""
yhat: {yhat}\n
yhat2: {yhat2}
""")
make tf_model for transfer learning
tf_model = tf.keras.Sequential()
tf_model.add(base_model.layers[0])
tf_model.add(Lambda(lambda z: base_model.layers[1].gamma*((z-base_model.layers[1].moving_mean)/K.sqrt(base_model.layers[1].moving_variance + base_model.layers[1].epsilon)) + base_model.layers[1].beta))
tf_model.trainable = False  # freeze the existing layers
tf_model.add(Dense(4, activation="relu"))  # add new layer
tf_model.add(Dense(1))  # add new layer
tf_model.compile(loss="mse")
tf_model.fit(X, Y, epochs=10)
`

Contrary to most, I agree with @fchollet; the existing API can fulfill this PR's intent. The PR does ease the process, at expense of increased computing time per adding an iteration-level conditional to the graph - but it's a valid patch that could've been merged with a printed warning.
The solution is rather simple; use a model-building function with an argument, e.g. make_model(bn_trainable) that's used as BatchNormalization()(x, trainable=bn_trainable). That this "can't be changed after model compilation" isn't a serious limitation, as models can be easily saved/restored uncompiled, and you're likely loading models for transfer learning anyway.

@datumbox  Solved part of my puzzle， But there is another way：
The purpose is to adjust and calculate BN parameters with a set of data set. In the first method, the following layers can be fine-tuned and trained first, and all layers can be trained after the parameters of the pre-trained model are available，it works for me.
Anyway  tks. @datumbox @fchollet ,

For some reason, @datumbox 's patch did not solve my problem with BN layer. I am using tf 2.0.0 and Keras 2.3.1 (in an anaconda environment). I made sure to correctly alter the normalization.py and tensorflow_backend.py. I have run the testing code from @datumbox blog, but the different behavior between training and testing still remains.
I also tried @rpeloff and @faustomorales solution without success on @datumbox  testing script.

Hi @RomainSabathe
did you solve problem just like you mentioned (pretrained = tf.keras.applications.inception_v3.InceptionV3( layers=tf.keras.layers, weights='imagenet'))
) without any patch?
or after set up a patch (if so which patch is suitable? I am using TF2 and keras 2.3.1)

Hi @rpeloff
did you solve problem just like you mentioned (pretrained = tf.keras.applications.inception_v3.InceptionV3( layers=tf.keras.layers, weights='imagenet'))
)  without any patch?
or after set up  @datumbox's patch?
if so which patch is suitable? I am using TF2 and keras 2.3.1

Regarding tf.keras, there's now a clear explanation of the relevant behavior in the docs:
https://www.tensorflow.org/api_docs/python/tf/keras/layers/BatchNormalization#output_shape_2

... This behavior has been introduced in TensorFlow 2.0, in order to enable layer.trainable = False to produce the most commonly expected behavior in the convnet fine-tuning use case.

@mathandy Thanks for the reference. I think it makes sense that TF 2 changed the behavior of the layer to facilitate transfer learning.
Perhaps that's an opportunity to resubmit this fix for consideration. I filed a new PR at #13892, let's see if it will be accepted. :)

@datumbox Thank you so much!  As you mentioned, this has a severe impact when implementing transfer learning and fine tuning part of the model.  I was getting high accuracy during training while inference accuracy was dipping by around 30%.  I spent a significant amount of time ensuring that math and code was correct.   I implemented your fix and inference started giving F1 and accuracy on par with training.  This saved me a significant amount of time and head ache.   Thank you again!

@lovejing0306

I used Tensorflow 2.0.0-rc0, when I fine-tuning resnet have same problem.

@off99555

In tensorflow2 GPU there is still this problem occurring, I have to use @faustomorales code in order to fix the issue.

For those using TensorFlow 2.0 and trying to fine-tune resnet, inceptionv3, etc., the problem seems to persist due to the injection of tensorflow.python.keras.layers. This references the TF 1.0 behaviour batch normalisation in keras_applications when calling layers.BatchNormalization (for example, in inceptionv3).
Similar to what @faustomorales suggested, I found that simply injecting tf.keras.layers references the TF 2.0 behaviour batch normalisation and fixes this issue (see here for the change in behaviour). When loading models from tf.keras.applications simply add the argument layers=tf.keras.layers. For example:
import tensorflow as tf

pretrained = tf.keras.applications.inception_v3.InceptionV3(
    layers=tf.keras.layers, weights='imagenet')

This worked perfectly for me. Thanks everyone for their support

If I'm still on TF 1.15 (the last 1.x) release. What is the most succinct solution for this problem? Is it?
import tensorflow as tf

pretrained = tf.keras.applications.inception_v3.InceptionV3(
    layers=tf.keras.layers, weights='imagenet')

Or is it... https://github.com/keras-team/keras/pull/9965#issuecomment-591748742

everybody is mentioning that @faustomorales's solution works but i can not see that solution.

@Shahidul1004 see https://github.com/keras-team/keras/pull/9965#issuecomment-501933060

Hi, @datumbox, thanks for your contribution. I'm a noob in Keras. Can you tell me what changes you made on the code side? At least some top-overview(of course apart from what you have mentioned on the blog)

Hi, @datumbox, thanks for your contribution. Can you tell me what changes you made on the code side? At least some top-overview(of course apart from what you have mentioned on the blog).
Also, you have written in the blog "Thankfully starting from version 2.1.3, when a BN layer is frozen it no longer updates its statistics. But is that enough? Not if you are using Transfer Learning." What I understood from this is that since Keras 2.1.3 BN layer will be locked up when frozen. So, is there no need for the patch you made if we use Keras 2.1.3?


Implement Lambda API InvokeWithResponseStream

Describe the feature
Lambda currently supports response streaming, but that's used through a separate Invoke API call called InvokeWithResponseStream. https://docs.aws.amazon.com/lambda/latest/api/API_InvokeWithResponseStream.html
Currently there doesn't seem to be any support for this API in the Rust SDK, even though it's supported in the Java, Python and JavaScript SDKs.
Use Case
I need to call Lambda functions that will return more than 6 MB as a response by utilizing the response streaming feature.
Proposed Solution
Implement the API.
Other Information
No response
Acknowledgements

[ ] I may be able to implement this feature request
[ ] This feature might incur a breaking change

A note for the community

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue, please leave a comment

The response is application/vnd.amazon.eventstream. Apparently this SDK already has some support for that stream type, but unfortunately it's specialized for the audio stream and not this InvokeWithResponseStream.
https://github.com/awslabs/aws-sdk-rust/blob/963342a44e7bd7661647c86623b8d2d2b823928b/sdk/transcribestreaming/src/event_stream_serde.rs#L57

The code generator already supports event streams for APIs using the Smithy RestJson protocol, so I think this Lambda API would work. We just need to test it and add it to the allow list to let the code generator put in the actual SDK.

Hi, thanks for getting this implemented so quickly! How often do new releases get pushed to crates.io? We'd love to start using this change with Lambda ASAP. I tried to just add the repository as a Cargo dependency but found that the size of this repo makes fetches in CI extremely slow, so we'll wait for an official release to crates.io to get started.

I am not one of the devs, but looking at the release frequency, I doubt it's going to be terribly many days since this is already in pending release: https://github.com/awslabs/aws-sdk-rust/releases

It'll go out in one of the daily releases the next time we release smithy-rs, which should be pretty soon.

This went out in yesterday's release.


index.html: load everything over https

demo: https://cben.github.io/codemirror-spell-checker/ (works)
vs https://nextstepwebs.github.io/codemirror-spell-checker/ (doesn't - "mixed content" errors)
(main audience is users of HTTPS Everywhere extension who automatically get the https demo.)

Good catch, thanks! I think it's best to change it from http:// to // so that it uses the matching protocol. Non-SSL connections are faster, so I'd like to utilize them when possible.

I can do that but then it won't work from a local checkout (file://).  One
would have to run a local server.
What do you prefer?
P.S. http://www.paulirish.com/2010/the-protocol-relative-url/ &
https://istlsfastyet.com/ argue that https-always is fast enough nowdays.
2015-07-21 17:05 GMT+03:00 Wes Cossick<EMAIL_ADDRESS>
Good catch, thanks! I think it's best to change it from http:// to // so
that it uses the matching protocol. Non-SSL connections are faster, so I'd
like to utilize them when possible.
—
Reply to this email directly or view it on GitHub
https://github.com/NextStepWebs/codemirror-spell-checker/pull/6#issuecomment-123321475
.

Since people may be checking it out and running it locally, I'll merge the https:// version.


Use separate required option for @cog.input()

The double behavior of default is not obvious.
This might want to be optional=True instead, so that inputs are required by default. If the inputs were not required by default, then users normally would make them required. If required inputs are not marked as required, this will cause breakage.

Closing, as @cog.input has been removed as of v0.1.0-alpha.


Update README.md

Added GW Security Cameras

"Not working yet"? Why you add it?

It works now.  I had to change the stream type from h.265 to h.264.  Please just delete that note.  I added it because I knew that it is correct.  It is correct.


vendor/ directory still owned by root in CI

I noticed yesterday that a router-pr job was stuck on Jenkins. A shell into the node showed that the workplace/router-pr/vendor/ directory was owned by root, a problem we'd seen before. Deleting the directory let the job continue to run, but it immediately recreated the owned-by-root problem.

See also https://github.com/deis/jenkins-jobs/issues/209

semi-resolved via https://github.com/deis/ci-manager/pull/8