how to use fit_generator with multiple image inputs

Hello, I'm trying to use a model with paired input images through (in their own similar directory trees), augmented through ImageDataGenerator using also flow_from_directory (so the method infers the labels by the folder structure). I'm getting an error because keras can't handle it in this way.
How can I combine the generators (using flow_from_directory) to be accepted by fit_generator?
Here is a sample code
Model definition ***
#two classic CNN blocks are defined before these lines and then cocatenated
#create model
model = Model([input_img_1,input_img_2], out, name='colliculus_proto')

#compile model
model.compile(optimizer=Adam(lr=1e-3),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

# image data generators for image inputs
def input_generator(train_dir,batchsize,img_height,img_width):
    train_generator = ImageDataGenerator(rescale = 1./255,
                                       shear_range = 0.2,
                                       zoom_range = 0.2,
                                       rotation_range=5.,
                                       horizontal_flip = True)
    training_set = train_generator.flow_from_directory(train_dir,
                                                 target_size = (img_height,img_width),
                                                 class_mode = 'categorical',
                                                 batch_size = batchsize,
                                                 shuffle=False)
    return training_set

def test_generator(test_dir,batchsize,img_height,img_width):
    test_gen = ImageDataGenerator(rescale = 1./255)
    test_set = test_gen.flow_from_directory(test_dir,
                                            target_size = (img_height,img_width),
                                            class_mode = 'categorical',
                                            batch_size = batchsize,
                                            shuffle=False)
    return test_set

#fit model
input1=input_generator(train_dir_1,batchsize,img_height,img_width)
input2=input_generator(train_dir_2,batchsize,img_height,img_width)
test1=test_generator(test_dir_1,batchsize,img_height,img_width)
test2=test_generator(test_dir_2,batchsize,img_height,img_width)

model.fit_generator([input1,input2],
                        steps_per_epoch=trainsetsize/batchsize,
                        epochs = epochs,
                        validation_data = [test1,test2],
                        validation_steps = testsetsize/batchsize,
                        use_multiprocessing=True,
                        shuffle=False)

The error I get is the following:
TypeError: Error when checking model input: data should be a Numpy array, or list/dict of Numpy arrays. Found: <keras.preprocessing.image.DirectoryIterator object at 0x7f824c5080f0>...

https://github.com/fchollet/keras/issues/3386
this might help

Thanks. Here is how I solved it following some ideas from issue 3386, maybe someone might find it useful

input_imgen = ImageDataGenerator(rescale = 1./255,
                                   shear_range = 0.2,
                                   zoom_range = 0.2,
                                   rotation_range=5.,
                                   horizontal_flip = True)

test_imgen = ImageDataGenerator(rescale = 1./255)

def generate_generator_multiple(generator,dir1, dir2, batch_size, img_height,img_width):
    genX1 = generator.flow_from_directory(dir1,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False,
                                          seed=7)

    genX2 = generator.flow_from_directory(dir2,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False,
                                          seed=7)
    while True:
            X1i = genX1.next()
            X2i = genX2.next()
            yield [X1i[0], X2i[0]], X2i[1]  #Yield both images and their mutual label

inputgenerator=generate_generator_multiple(generator=input_imgen,
                                           dir1=train_dir_1,
                                           dir2=train_dir_2,
                                           batch_size=batch_size,
                                           img_height=img_height,
                                           img_width=img_height)

testgenerator=generate_generator_multiple(test_imgen,
                                          dir1=train_dir_1,
                                          dir2=train_dir_2,
                                          batch_size=batch_size,
                                          img_height=img_height,
                                          img_width=img_height)

 history=model.fit_generator(inputgenerator,
                        steps_per_epoch=trainsetsize/batch_size,
                        epochs = epochs,
                        validation_data = testgenerator,
                        validation_steps = testsetsize/batch_size,
                        use_multiprocessing=True,
                        shuffle=False)

That was great. How would you find the class_indices for the inputgenerator and testgenerator? I'm not able to use this syntax: inputgenerator.class_indices() here since now, the generator works with multiple inputs.

@aendrs I would like to ask how to give the two inputs same pre-processing everytime? As in the function of input_generator,  input_imgen is called twice, for each time randomly selected preprocessing steps are applied on the input.

@aendrs @raaju-shiv @wangqianwen0418 @laukun @bmabey Could anyone of you guys kindly help me solve this problem https://github.com/keras-team/keras/issues/10499. I tried implementing the same generator as in this post but i dont seem  to figure out where is my mistake. any help is very much appreciated.

@laukun I would suggest write your own data generator like in this example:
class DataGenerator(keras.utils.Sequence):
    """Generates data for Keras."""
    def __init__(self, img_files, clinical_info, labels, ave=None, std=None, batch_size=32, dim=(300, 300), n_channels=3,
                 n_classes=2, shuffle=True):
        """Initialization.

        Args:
            img_files: A list of path to image files.
            clinical_info: A dictionary of corresponding clinical variables.
            labels: A dictionary of corresponding labels.
        """
        self.img_files = img_files
        self.clinical_info = clinical_info
        self.labels = labels
        self.batch_size = batch_size
        self.dim = dim
        if ave is None:
            self.ave = np.zeros(n_channels)
        else:
            self.ave = ave
        if std is None:
            self.std = np.zeros(n_channels) + 1
        else:
            self.std = std

        self.n_channels = n_channels
        self.n_classes = n_classes
        self.shuffle = shuffle
        self.on_epoch_end()

    def __len__(self):
        """Denotes the number of batches per epoch."""
        return int(np.floor(len(self.img_files) / self.batch_size))

    def __getitem__(self, index):
        """Generate one batch of data."""
        # Generate indexes of the batch
        indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]

        # Find list of IDs
        img_files_temp = [self.img_files[k] for k in indexes]

        # Generate data
        X, y = self.__data_generation(img_files_temp)

        return X, y

    def on_epoch_end(self):
        """Updates indexes after each epoch."""
        self.indexes = np.arange(len(self.img_files))
        if self.shuffle == True:
            np.random.shuffle(self.indexes)

    def __data_generation(self, img_files_temp):
        """Generates data containing batch_size samples."""
        # X : (n_samples, *dim, n_channels)
        # X = [np.empty((self.batch_size, self.dim[0], self.dim[1], self.n_channels))]
        X_img = []
        X_clinical = []
        y = np.empty((self.batch_size), dtype=int)

        # Generate data
        for i, img_file in enumerate(img_files_temp):
            # Read image
            img = skimage.io.imread(img_file)

            # Resize
            img = skimage.transform.resize(img, output_shape=self.dim, mode='constant', preserve_range=True)

            # Normalization
            for ch in range(self.n_channels):
                img[:, :, ch] = (img[:, :, ch] - self.ave[ch])/self.std[ch]

            if self.shuffle:
                # Some image augmentation codes
                ###### You can put your preprocessing codes here. #####

            X_img.append(img)
            X_clinical.append(self.clinical_info[img_file])
            y[i] = self.labels[img_file]
        X = [np.array(X_img), np.array(X_clinical)]
        return X, keras.utils.to_categorical(y, num_classes=self.n_classes)

And to call the generator:
train_datagen = DataGenerator(img_files=img_files, clinical_info=clinical_info, labels=labels, ave=ave, std=std, batch_size=batch_size)
val_datagen = DataGenerator(img_files=img_files, clinical_info=clinical_info, labels=labels, ave=ave, std=std, batch_size=batch_size)
hist = model.fit_generator(train_datagen,
                           steps_per_epoch=len(img_files) / batch_size,
                           epochs=num_epochs,
                           verbose=1,
                           validation_data=val_datagen,
                           validation_steps=1)

The different thing I'm doing here is using two different kinds of input, an image and a numpy array. I saved image paths as a list, and created two dictionaries whose key words are the image paths. When you want to do the preprocessing, you can easily apply them right after reading the image using the image paths.
But this way, the speed will be definitely slower than loading data directly from .npy

I'm trying to train a two streams CNN architecture, but unfortunately I got an unexpected result whitch is almost the same as one of the two streams (I have already trained every one separately):
here is my code, please tell me what I'm doing wrong :
@aendrs @sdw95927  @wangkechn
`
Arch
def fusionNet(nb_classes, inputs=(3, 224, 224)):
    input_img = Input(shape=inputs, name='RGB_input')
    conv1 = Conv2D(96, (7, 7), activation="relu", name="conv1", strides=(2, 2), padding="same",
    ## ....
    ## ....
    ## ....
    firstCNN_output = BatchNormalization(name='firstCNN_output_BN')(last_dropout)
    ##
    print(firstCNN_output._keras_shape) # I got here (None, 512, 13, 13)

    input_img2 = Input(shape=inputs, name='depth_input')
    conv12 = Conv2D(96, (7, 7), activation="relu", name="conv12", strides=(2, 2), padding="same",             kernel_initializer="glorot_uniform")(input_img2)
   #....
   #....
   #...
## BN
    SecondCNN_output = BatchNormalization(name='SecondCNN_output_BN2')(last_dropout2)
    print(SecondCNN_output._keras_shape) # I got here (None, 512, 13, 13)
    #

    merge2CNN = merge([firstCNN_output, SecondCNN_output], mode='concat', concat_axis=1)
    ### got here (None, 1024, 13, 13)
    conv10 = Conv2D(
    nb_classes, (1, 1),init='glorot_uniform',
    border_mode='valid', name='conv10')(merge2CNN)
   avgpool10 = AveragePooling2D((13, 13), name='avgpool10')(conv10)

    flatten = Flatten(name='flatten')(avgpool10)
   softmax = Activation("softmax", name='softmax')(flatten)

    return Model(inputs=[input_img,input_img2], output=softmax)

`
Training
sn = fusionNet(nb_classes=24)
sgd = SGD(lr=0.001, decay=0.0002, momentum=0.9, nesterov=True)

sn.compile( optimizer=sgd, loss='categorical_crossentropy', metrics=['accuracy')

train_datagen = ImageDataGenerator(
    rescale=1./255,
    zoom_range=0.01,
    rotation_range=2.,
    featurewise_center=True,
    featurewise_std_normalization=True,
)
#train_datagen = ImageDataGenerator(rescale=1./255)

test_datagen = ImageDataGenerator(rescale=1./255)

def generate_generator_multiple(generator,dir1, dir2, batch_size, img_height,img_width):
genX1 = generator.flow_from_directory(dir1,
                                  target_size = (img_height,img_width),
                                  class_mode = 'categorical',
                                  batch_size = batch_size,
                                  shuffle=False)

genX2 = generator.flow_from_directory(dir2,
                                  target_size = (img_height, img_width),
                                  class_mode = 'categorical',
                                  batch_size = batch_size)
while True:
    X1i = genX1.next()
    X2i = genX2.next()
    yield [X1i[0], X2i[0]], X2i[1]  #Yield both images and their mutual label

train_generator=generate_generator_multiple(generator=train_datagen,
                                   dir1=train_data_dir,
                                   dir2=train_data_dir_colored,
                                   batch_size=32,
                                   img_height=size,
                                   img_width=size)

validation_generator=generate_generator_multiple(test_datagen,
                                  dir1=validation_data_dir,
                                  dir2=validation_data_dir_colored,
                                  batch_size=batch_size,
                                  img_height=size,
                                  img_width=size)

tensorboardcb = keras.callbacks.TensorBoard(log_dir='/workspace/sqn_224/fusion', histogram_freq=0,         batch_size=32, write_graph=True)
sn.fit_generator(
        train_generator,
    steps_per_epoch=nb_train_samples//32,
    nb_epoch=nb_epoch,
    validation_data=validation_generator,
    validation_steps = nb_validation_samples//32,
    callbacks=[tensorboardcb])

I would like to merge the input data first, which might be a better idea.

@scstu I would like to merge the input data first, which might be a better idea.

@scstu that can be done in different ways. but keep in mind the number of learned parameters differ depending on the way you merge the input images. so if youre merging in a multispectral-like shape (for 2RGB images you can merge in two ways as far as i know. so it would be something like: hw6 presuming that your concatenating the images, or it could be hw*3 if youre finding a transformation of the 2 RGB images such as multiplication, add, sub etc..) in either case be careful with the input size for your model and remember that the number of the learned parameters will differ. to know whats better you can only try and compare the results. it also will affect the extracted feature by the convolution differently. i worked on a similar problem and i used multiple branch CNN and 2 input images where each images passes a branch from the CNN and then the features extracted by the two CNN branches are merged in a merging layer before passing it to the top layers.

@MjdMahasneh  I would really appreciate it if you took a look at the question I posted in SO. from reading your comment, I believe that what I am working on is quite similar to what you wrote about. Part of the question is related to my custom generator (the concerned code is written there).
Of course, anyone else is invited to take a look. I'd really be thankful if someone could provide some guidance.

@aendrs thank you very much on your algorithm. He is fantastic.

@aendrs it was really good work to generate multi-streams input. i'm new to all of these please help me out. i'm confused how we will define the input shape my data images are RGB and i'm taking two streams input using your generator code then getting following error:
ValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 1 array(s), but instead got the following list of 2 arrays: [array([[[[ 0.46274513,  0.08627451,  0.72156864],
[ 0.53333336,  0.10196079,  0.72549021],
[ 0.5529412 ,  0.1254902 ,  0.71764708],
...,
[ 0.56078434,  0.07058824...
the code is as following
img_width, img_height = 256, 58
train_data_dir_1 = 'image_data/training_set_1'
train_data_dir_2 = 'image_data/training_set_2'
validation_data_dir_1 = 'image_data/validation_set_1'
validation_data_dir_2 = 'image_data/validation_set_2'
nb_train_samples = 686
nb_validation_samples = 147
epochs = 25
batch_size = 12
if K.image_data_format() == 'channels_first':
input_shape = (3, img_width, img_height)
else:
input_shape = (img_width, img_height, 5)
model = Sequential()
model.add(Conv2D(32, (3, 3), input_shape=input_shape))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(64, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(128, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(256, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dense(128))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(1))
model.add(Activation('sigmoid'))
model.compile(loss='categorical_crossentropy',
optimizer='rmsprop',
metrics=['accuracy'])

if i do like this ( use_multi_processing=True) in this code
history=model.fit_generator(inputgenerator,
steps_per_epoch=trainsetsize/batch_size,
epochs = epochs,
validation_data = testgenerator,
validation_steps = testsetsize/batch_size,
use_multiprocessing=True,
shuffle=False)
then i get the following error
Using a generator with use_multiprocessing=True is not supported on Windows (no marshalling of generators across process boundaries). Instead, use single thread/process or multithreading.

Anybody help please...I'm new to all of these please help me out. i'm confused how we will define the input shape my data images are RGB and i'm taking two streams input using your generator code then getting following error:
ValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 1 array(s), but instead got the following list of 2 arrays: [array([[[[ 0.46274513,  0.08627451,  0.72156864],
[ 0.53333336,  0.10196079,  0.72549021],
[ 0.5529412 ,  0.1254902 ,  0.71764708],
...,
[ 0.56078434,  0.07058824...
the code is as following
img_width, img_height = 256, 58
train_data_dir_1 = 'image_data/training_set_1'
train_data_dir_2 = 'image_data/training_set_2'
validation_data_dir_1 = 'image_data/validation_set_1'
validation_data_dir_2 = 'image_data/validation_set_2'
nb_train_samples = 686
nb_validation_samples = 147
epochs = 25
batch_size = 12
if K.image_data_format() == 'channels_first':
input_shape = (3, img_width, img_height)
else:
input_shape = (img_width, img_height, 5)
model = Sequential()
model.add(Conv2D(32, (3, 3), input_shape=input_shape))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(64, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(128, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(256, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dense(128))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(1))
model.add(Activation('sigmoid'))
model.compile(loss='categorical_crossentropy',
optimizer='rmsprop',
metrics=['accuracy'])

@aendrs could you please share your portion of code for concatenation and inputs manipulation

I believe your comment is interesting about trainings and parameters. I have a doubt. I will train input sets on the same network, for example, model1 receives input X1 (three folders containing classes and each class has the training, validation, and test data) and model2 receives input X2 (three folders containing three classes and each class has the training, validation and test data). Then I will concatenate a convolution of model X1 with one of model X2. So I have two input data, two validation data and two test data. My question is about the following command: steps_per_epoch = nb_train_samples // batchsize. I would like to know if my nb_train_samples is the sum of only the training_class1_X1 + training_class2_X1 + training_class3_X1 or if nb_train_samples is the sum of (training_class1_X1 + training_class2_X1 + training_class3_X1) + (training_class1_X2 + training_class2_X2 + training_class3_X2).

@scstu that can be done in different ways. but keep in mind the number of learned parameters differ depending on the way you merge the input images. so if youre merging in a multispectral-like shape (for 2xRGB images you can merge in two ways as far as i know. so it would be something like: HxWx6 presuming that your concatenating the images, or it could be HxWx3 if youre finding a transformation of the 2 RGB images such as multiplication, add, sub etc..) in either case be careful with the input size for your model and remember that the number of the learned parameters will differ. to know whats better you can only try and compare the results. it also will affect the extracted feature by the convolution differently. i worked on a similar problem and i used multiple branch CNN and 2 input images where each images passes a branch from the CNN and then the features extracted by the two CNN branches are merged in a merging layer before passing it to the top layers.

@MjdMahasneh I believe your comment is interesting. Could you help me with a doubt? I will train input sets on the same network, for example, model1 receives input X1 (three folders containing classes and each class has the training, validation, and test data) and model2 receives input X2 (three folders containing three classes and each class has the training, validation and test data). Then I will concatenate a convolution of model X1 with one of model X2. So I have two input data, two validation data and two test data. My question is about the following command: steps_per_epoch = nb_train_samples // batchsize. I would like to know if my nb_train_samples is the sum of only the training_class1_X1 + training_class2_X1 + training_class3_X1 or if nb_train_samples is the sum of (training_class1_X1 + training_class2_X1 + training_class3_X1) + (training_class1_X2 + training_class2_X2 + training_class3_X2).

I'm trying to train a two streams CNN architecture, but unfortunately I got an unexpected result whitch is almost the same as one of the two streams (I have already trained every one separately):
here is my code, please tell me what I'm doing wrong :
@aendrs @sdw95927 @wangkechn
`
Arch
def fusionNet(nb_classes, inputs=(3, 224, 224)):
    input_img = Input(shape=inputs, name='RGB_input')
    conv1 = Conv2D(96, (7, 7), activation="relu", name="conv1", strides=(2, 2), padding="same",
    ## ....
    ## ....
    ## ....
    firstCNN_output = BatchNormalization(name='firstCNN_output_BN')(last_dropout)
    ##
    print(firstCNN_output._keras_shape) # I got here (None, 512, 13, 13)

    input_img2 = Input(shape=inputs, name='depth_input')
    conv12 = Conv2D(96, (7, 7), activation="relu", name="conv12", strides=(2, 2), padding="same",             kernel_initializer="glorot_uniform")(input_img2)
   #....
   #....
   #...
## BN
    SecondCNN_output = BatchNormalization(name='SecondCNN_output_BN2')(last_dropout2)
    print(SecondCNN_output._keras_shape) # I got here (None, 512, 13, 13)
    #

    merge2CNN = merge([firstCNN_output, SecondCNN_output], mode='concat', concat_axis=1)
    ### got here (None, 1024, 13, 13)
    conv10 = Conv2D(
    nb_classes, (1, 1),init='glorot_uniform',
    border_mode='valid', name='conv10')(merge2CNN)
   avgpool10 = AveragePooling2D((13, 13), name='avgpool10')(conv10)

    flatten = Flatten(name='flatten')(avgpool10)
   softmax = Activation("softmax", name='softmax')(flatten)

    return Model(inputs=[input_img,input_img2], output=softmax)

`
Training
sn = fusionNet(nb_classes=24)
sgd = SGD(lr=0.001, decay=0.0002, momentum=0.9, nesterov=True)

sn.compile( optimizer=sgd, loss='categorical_crossentropy', metrics=['accuracy')

train_datagen = ImageDataGenerator(
    rescale=1./255,
    zoom_range=0.01,
    rotation_range=2.,
    featurewise_center=True,
    featurewise_std_normalization=True,
)
#train_datagen = ImageDataGenerator(rescale=1./255)

test_datagen = ImageDataGenerator(rescale=1./255)

def generate_generator_multiple(generator,dir1, dir2, batch_size, img_height,img_width):
genX1 = generator.flow_from_directory(dir1,
                                  target_size = (img_height,img_width),
                                  class_mode = 'categorical',
                                  batch_size = batch_size,
                                  shuffle=False)

genX2 = generator.flow_from_directory(dir2,
                                  target_size = (img_height, img_width),
                                  class_mode = 'categorical',
                                  batch_size = batch_size)
while True:
    X1i = genX1.next()
    X2i = genX2.next()
    yield [X1i[0], X2i[0]], X2i[1]  #Yield both images and their mutual label

train_generator=generate_generator_multiple(generator=train_datagen,
                                   dir1=train_data_dir,
                                   dir2=train_data_dir_colored,
                                   batch_size=32,
                                   img_height=size,
                                   img_width=size)

validation_generator=generate_generator_multiple(test_datagen,
                                  dir1=validation_data_dir,
                                  dir2=validation_data_dir_colored,
                                  batch_size=batch_size,
                                  img_height=size,
                                  img_width=size)

tensorboardcb = keras.callbacks.TensorBoard(log_dir='/workspace/sqn_224/fusion', histogram_freq=0,         batch_size=32, write_graph=True)
sn.fit_generator(
        train_generator,
    steps_per_epoch=nb_train_samples//32,
    nb_epoch=nb_epoch,
    validation_data=validation_generator,
    validation_steps = nb_validation_samples//32,
    callbacks=[tensorboardcb])

Could you help me with a doubt? I will train input sets on the same network, for example, model1 receives input X1 (three folders containing classes and each class has the training, validation, and test data) and model2 receives input X2 (three folders containing three classes and each class has the training, validation and test data). Then I will concatenate a convolution of model X1 with one of model X2. So I have two input data, two validation data and two test data. My question is about the following command: steps_per_epoch = nb_train_samples // batchsize. I would like to know if my nb_train_samples is the sum of only the training_class1_X1 + training_class2_X1 + training_class3_X1 or if nb_train_samples is the sum of (training_class1_X1 + training_class2_X1 + training_class3_X1) + (training_class1_X2 + training_class2_X2 + training_class3_X2).

I am trying similar things, but I am getting stuck with the input of the model.
`
def generate_generator_multiple(generator,dir1, dir2,dir3, batch_size, img_size):
      genX1 = generator.flow_from_directory(dir1,
                                      target_size = (img_size,img_size),
                                      class_mode = 'categorical',
                                      batch_size = batch_size,
                                      shuffle=False,
                                      seed=7,subset='training')

      genX2 = generator.flow_from_directory(dir2,
                                      target_size = (img_size,img_size),
                                      class_mode = 'categorical',
                                      batch_size = batch_size,
                                      shuffle=False,
                                      seed=7,subset='training')

      genX3 = generator.flow_from_directory(dir3,
                                  target_size = (img_size,img_size),
                                  class_mode = 'categorical',
                                  batch_size = batch_size,
                                  shuffle=False,
                                  seed=7,subset='training')
       while True:
              X1i = genX1.next()
              X2i = genX2.next()
              X3i = genX3.next()
       yield [X1i[0], X2i[0],X3i[0]], X3i[1]  #Yield both images and their mutual label

train_datagen = ImageDataGenerator(validation_split=VAL_SPLIT,rescale=1./255)
inputgenerator=generate_generator_multiple(generator=train_datagen,
                                       dir1='mergedata',
                                       dir2='mergedata',
                                       dir3='mergedata',
                                       batch_size=BATCH_SIZE,
                                       img_size=IMG_SIZE)

modelA = load_model('/content/8-base_model_1.h5')
modelA.load_weights('/content/8-weights.002-0.991-0.406.hdf5')

modelB = load_model('/content/30-base_model_1.h5')
modelB.load_weights('/content/30-weights.010-0.499-0.374.hdf5')

modelC = load_model('/content/80-base_model_1.h5')
modelC.load_weights('/content/80-weights.002-1.064-0.305.hdf5')
for layer in modelA.layers:
    layer.trainable = False
    layer.name = layer.name + str("_1")

for layer in modelB.layers:
    layer.trainable = False
    layer.name = layer.name + str("_2")

for layer in modelC.layers:
    layer.trainable = False
    layer.name = layer.name + str("_3")

mod_1 = concatenate([modelA.layers[-1].output,  modelB.layers[-1].output,modelC.layers[-1].output])
mod_1 = Dropout(0.2)(mod_1)
mod_1 = Dense(15, activation='relu')(mod_1)
predictions = Dense(9, activation='softmax', name='pred_age')(mod_1)

top_model = Model(inputs=[modelA.input, modelB.input, modelC.input], outputs=predictions)

opt = get_optimizer('adam', lr_rate)
top_model.compile(loss=['categorical_crossentropy'],
          optimizer=opt,
          metrics=['accuracy','mae'])

model.fit_generator(inputgenerator,
                           epochs=EPOCHS,
                           steps_per_epoch = STEPS_PER_EPOCH,
                           validation_data=validgenerator,
                           validation_steps = VALIDATION_STEPS,
                           verbose=1,
                           callbacks=callbacks)`

I first trained 3 different models:

model 1 = 3 labels (8,9,10)
model 2 = 2 labels (30,31)
model 3 = 4 labels (80,81,82,83)

I would like to combine these 3 models into a final model with 1 output containing 9 labels (8,9,10,30,31,80,81,82,83). The final input needs to get 1 input image instead of 3 images. But I still get stuck. I am building a kind of hierarchy here to improve the accuracy.

Thanks. Here is how I solved it following some ideas from issue 3386, maybe someone might find it useful
input_imgen = ImageDataGenerator(rescale = 1./255,
                                   shear_range = 0.2,
                                   zoom_range = 0.2,
                                   rotation_range=5.,
                                   horizontal_flip = True)

test_imgen = ImageDataGenerator(rescale = 1./255)

def generate_generator_multiple(generator,dir1, dir2, batch_size, img_height,img_width):
    genX1 = generator.flow_from_directory(dir1,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False,
                                          seed=7)

    genX2 = generator.flow_from_directory(dir2,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False,
                                          seed=7)
    while True:
            X1i = genX1.next()
            X2i = genX2.next()
            yield [X1i[0], X2i[0]], X2i[1]  #Yield both images and their mutual label

inputgenerator=generate_generator_multiple(generator=input_imgen,
                                           dir1=train_dir_1,
                                           dir2=train_dir_2,
                                           batch_size=batch_size,
                                           img_height=img_height,
                                           img_width=img_height)

testgenerator=generate_generator_multiple(test_imgen,
                                          dir1=train_dir_1,
                                          dir2=train_dir_2,
                                          batch_size=batch_size,
                                          img_height=img_height,
                                          img_width=img_height)

 history=model.fit_generator(inputgenerator,
                        steps_per_epoch=trainsetsize/batch_size,
                        epochs = epochs,
                        validation_data = testgenerator,
                        validation_steps = testsetsize/batch_size,
                        use_multiprocessing=True,
                        shuffle=False)

How is that that the test generator and train generator(input generator) have the same directories??
Shouldn't it be different??
(dir1 and dir2 should be different)
Please correct me if I'm wrong.

@laukun I would suggest write your own data generator like in this example:
class DataGenerator(keras.utils.Sequence):
    """Generates data for Keras."""
    def __init__(self, img_files, clinical_info, labels, ave=None, std=None, batch_size=32, dim=(300, 300), n_channels=3,
                 n_classes=2, shuffle=True):
        """Initialization.

        Args:
            img_files: A list of path to image files.
            clinical_info: A dictionary of corresponding clinical variables.
            labels: A dictionary of corresponding labels.
        """
        self.img_files = img_files
        self.clinical_info = clinical_info
        self.labels = labels
        self.batch_size = batch_size
        self.dim = dim
        if ave is None:
            self.ave = np.zeros(n_channels)
        else:
            self.ave = ave
        if std is None:
            self.std = np.zeros(n_channels) + 1
        else:
            self.std = std

        self.n_channels = n_channels
        self.n_classes = n_classes
        self.shuffle = shuffle
        self.on_epoch_end()

    def __len__(self):
        """Denotes the number of batches per epoch."""
        return int(np.floor(len(self.img_files) / self.batch_size))

    def __getitem__(self, index):
        """Generate one batch of data."""
        # Generate indexes of the batch
        indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]

        # Find list of IDs
        img_files_temp = [self.img_files[k] for k in indexes]

        # Generate data
        X, y = self.__data_generation(img_files_temp)

        return X, y

    def on_epoch_end(self):
        """Updates indexes after each epoch."""
        self.indexes = np.arange(len(self.img_files))
        if self.shuffle == True:
            np.random.shuffle(self.indexes)

    def __data_generation(self, img_files_temp):
        """Generates data containing batch_size samples."""
        # X : (n_samples, *dim, n_channels)
        # X = [np.empty((self.batch_size, self.dim[0], self.dim[1], self.n_channels))]
        X_img = []
        X_clinical = []
        y = np.empty((self.batch_size), dtype=int)

        # Generate data
        for i, img_file in enumerate(img_files_temp):
            # Read image
            img = skimage.io.imread(img_file)

            # Resize
            img = skimage.transform.resize(img, output_shape=self.dim, mode='constant', preserve_range=True)

            # Normalization
            for ch in range(self.n_channels):
                img[:, :, ch] = (img[:, :, ch] - self.ave[ch])/self.std[ch]

            if self.shuffle:
                # Some image augmentation codes
                ###### You can put your preprocessing codes here. #####

            X_img.append(img)
            X_clinical.append(self.clinical_info[img_file])
            y[i] = self.labels[img_file]
        X = [np.array(X_img), np.array(X_clinical)]
        return X, keras.utils.to_categorical(y, num_classes=self.n_classes)

And to call the generator:
train_datagen = DataGenerator(img_files=img_files, clinical_info=clinical_info, labels=labels, ave=ave, std=std, batch_size=batch_size)
val_datagen = DataGenerator(img_files=img_files, clinical_info=clinical_info, labels=labels, ave=ave, std=std, batch_size=batch_size)
hist = model.fit_generator(train_datagen,
                           steps_per_epoch=len(img_files) / batch_size,
                           epochs=num_epochs,
                           verbose=1,
                           validation_data=val_datagen,
                           validation_steps=1)

The different thing I'm doing here is using two different kinds of input, an image and a numpy array. I saved image paths as a list, and created two dictionaries whose key words are the image paths. When you want to do the preprocessing, you can easily apply them right after reading the image using the image paths.
But this way, the speed will be definitely slower than loading data directly from .npy

@laukun I would suggest write your own data generator like in this example:
class DataGenerator(keras.utils.Sequence):
    """Generates data for Keras."""
    def __init__(self, img_files, clinical_info, labels, ave=None, std=None, batch_size=32, dim=(300, 300), n_channels=3,
                 n_classes=2, shuffle=True):
        """Initialization.

        Args:
            img_files: A list of path to image files.
            clinical_info: A dictionary of corresponding clinical variables.
            labels: A dictionary of corresponding labels.
        """
        self.img_files = img_files
        self.clinical_info = clinical_info
        self.labels = labels
        self.batch_size = batch_size
        self.dim = dim
        if ave is None:
            self.ave = np.zeros(n_channels)
        else:
            self.ave = ave
        if std is None:
            self.std = np.zeros(n_channels) + 1
        else:
            self.std = std

        self.n_channels = n_channels
        self.n_classes = n_classes
        self.shuffle = shuffle
        self.on_epoch_end()

    def __len__(self):
        """Denotes the number of batches per epoch."""
        return int(np.floor(len(self.img_files) / self.batch_size))

    def __getitem__(self, index):
        """Generate one batch of data."""
        # Generate indexes of the batch
        indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]

        # Find list of IDs
        img_files_temp = [self.img_files[k] for k in indexes]

        # Generate data
        X, y = self.__data_generation(img_files_temp)

        return X, y

    def on_epoch_end(self):
        """Updates indexes after each epoch."""
        self.indexes = np.arange(len(self.img_files))
        if self.shuffle == True:
            np.random.shuffle(self.indexes)

    def __data_generation(self, img_files_temp):
        """Generates data containing batch_size samples."""
        # X : (n_samples, *dim, n_channels)
        # X = [np.empty((self.batch_size, self.dim[0], self.dim[1], self.n_channels))]
        X_img = []
        X_clinical = []
        y = np.empty((self.batch_size), dtype=int)

        # Generate data
        for i, img_file in enumerate(img_files_temp):
            # Read image
            img = skimage.io.imread(img_file)

            # Resize
            img = skimage.transform.resize(img, output_shape=self.dim, mode='constant', preserve_range=True)

            # Normalization
            for ch in range(self.n_channels):
                img[:, :, ch] = (img[:, :, ch] - self.ave[ch])/self.std[ch]

            if self.shuffle:
                # Some image augmentation codes
                ###### You can put your preprocessing codes here. #####

            X_img.append(img)
            X_clinical.append(self.clinical_info[img_file])
            y[i] = self.labels[img_file]
        X = [np.array(X_img), np.array(X_clinical)]
        return X, keras.utils.to_categorical(y, num_classes=self.n_classes)

And to call the generator:
train_datagen = DataGenerator(img_files=img_files, clinical_info=clinical_info, labels=labels, ave=ave, std=std, batch_size=batch_size)
val_datagen = DataGenerator(img_files=img_files, clinical_info=clinical_info, labels=labels, ave=ave, std=std, batch_size=batch_size)
hist = model.fit_generator(train_datagen,
                           steps_per_epoch=len(img_files) / batch_size,
                           epochs=num_epochs,
                           verbose=1,
                           validation_data=val_datagen,
                           validation_steps=1)

The different thing I'm doing here is using two different kinds of input, an image and a numpy array. I saved image paths as a list, and created two dictionaries whose key words are the image paths. When you want to do the preprocessing, you can easily apply them right after reading the image using the image paths.
But this way, the speed will be definitely slower than loading data directly from .npy

Can you give me an example of clinical_info

def generate_generator_multiple(generator,dir1, dir2, batch_size, img_height,img_width):
genX1 = generator.flow_from_directory(dir1,
target_size = (img_height,img_width),
class_mode = 'categorical',
batch_size = batch_size,
shuffle=False,
seed=7)
genX2 = generator.flow_from_directory(dir2,
                                      target_size = (img_height,img_width),
                                      class_mode = 'categorical',
                                      batch_size = batch_size,
                                      shuffle=False,
                                      seed=7)
while True:
        X1i = genX1.next()
        X2i = genX2.next()
        yield [X1i[0], X2i[0]], X2i[1]  #Yield both images and their mutual label

In this method shown,
I have a situation where I am returning two ground truth masks, and one image using a custom data_generator as shown above.
However, how do I incorporate it into a custom loss function.
def custom_loss(y_true1,ytrue2, y_pred):
    #compute loss here
    return loss

Hi, I am using the train generator for two inputs as @aendrs wrote.
The problem is that the two generators have two different orders, and this is not intended because we have mutual labels. Is there a way to use order of the first generator in the second as well? Thank you!

To reply to my comment, I made my own class of generator that takes as input 2 images and allows multiple labels:
import numpy as np
import keras
import imageio
class DataGenerator(keras.utils.Sequence):
'Generates data for Keras'
def init(self, list_IDs1, list_IDs2, labels, batch_size=32, dim=(32,32,32), n_channels=1,
n_classes=10, shuffle=True):
'Initialization'
self.batch_size = batch_size
self.labels = labels
self.list_IDs1 = list_IDs1
self.list_IDs2 = list_IDs2
self.n_channels = n_channels
self.dim=dim
self.n_classes = n_classes
self.shuffle = shuffle
self.on_epoch_end()
def len(self):
'Denotes the number of batches per epoch'
return int(np.floor(len(self.list_IDs1) / self.batch_size))
def on_epoch_end(self):
'Updates indexes after each epoch'
self.indexes = np.arange(len(self.list_IDs1))
if self.shuffle == True:
np.random.shuffle(self.indexes)
def getitem(self, index):
'Generate one batch of data'
# Generate indexes of the batch
indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]
# Find list of IDs
list_IDs_temp1 = [self.list_IDs1[k] for k in indexes]
list_IDs_temp2 = [self.list_IDs2[k] for k in indexes]
# Generate data
[X1,X2], y = self.__data_generation(list_IDs_temp1,list_IDs_temp2,indexes)
return [X1,X2], y
def __data_generation(self, list_IDs_temp1,list_IDs_temp2,indexes):
'Generates data containing batch_size samples' # X : (n_samples, *dim, n_channels)
# Initialization
X1 = np.empty((self.batch_size, *self.dim, self.n_channels))
X2 = np.empty((self.batch_size, *self.dim, self.n_channels))
y = [np.empty(self.batch_size,dtype=np.float32) for _ in range(len(self.labels))]
# Generate data
for i, ID in enumerate(list_IDs_temp1):
# Store sample
X1[i,] = np.stack((imageio.imread(ID),) * 3, axis=-1)
# Store class
for list_id in range(len(self.labels)):
y[list_id][i] = self.labels[list_id][indexes[i]]
for i, ID in enumerate(list_IDs_temp2):
X2[i,] = np.stack((imageio.imread(ID),) * 3, axis=-1)
return [X1,X2], y
This way the order of the two batches of data is the same.

@enviz
Thanks for your guide on multiple inputs, very much appreciated.
I have a similar issue which I used your shared code. However, while executing, I have the TypeError: can't pickle generator objects
Kindly provide guidance to resolve this issue.
Thank you

@enviz
Thanks for your guide on multiple inputs, very much appreciated.
I have a similar issue which I used your shared code. However, while executing, I have the TypeError: can't pickle generator objects
Kindly provide guidance to resolve this issue.
Thank you

@aendrs @enviz
Please, I have the TypeError: can't pickle generator objects
Kindly provide guidance to resolve this issue.
Thank you

@aendrs @enviz
Please, I have the TypeError: can't pickle generator objects
Kindly provide guidance to resolve this issue.
Thank you

I experience a similar problem.
I wanted to combine two input streams: 1 is an image and 2 is numerical data.
I found that the best solution is to manipulate the keras.utils.Sequence.TimeseriesGenerator functionality for your own purpose here.
For my problem [x1, x2], y this is a working generator:
import numpy as np
import json
import keras
class TimeseriesGenerator(keras.utils.Sequence):
def __init__(self, img, data, targets, length,
             sampling_rate=1,
             stride=1,
             start_index=0,
             end_index=None,
             shuffle=False,
             reverse=False,
             batch_size=128):

    if len(data) != len(targets):
        raise ValueError('Data and targets have to be' +
                         ' of same length. '
                         'Data length is {}'.format(len(data)) +
                         ' while target length is {}'.format(len(targets)))

    self.img = img
    self.data = data
    self.targets = targets
    self.length = length
    self.sampling_rate = sampling_rate
    self.stride = stride
    self.start_index = start_index + length
    if end_index is None:
        end_index = len(data) - 1
    self.end_index = end_index
    self.shuffle = shuffle
    self.reverse = reverse
    self.batch_size = batch_size

    if self.start_index > self.end_index:
        raise ValueError('`start_index+length=%i > end_index=%i` '
                         'is disallowed, as no part of the sequence '
                         'would be left to be used as current step.'
                         % (self.start_index, self.end_index))

def __len__(self):
    return (self.end_index - self.start_index +
            self.batch_size * self.stride) // (self.batch_size * self.stride)

def __getitem__(self, index):
    if self.shuffle:
        rows = np.random.randint(
            self.start_index, self.end_index + 1, size=self.batch_size)
    else:
        i = self.start_index + self.batch_size * self.stride * index
        rows = np.arange(i, min(i + self.batch_size *
                                self.stride, self.end_index + 1), self.stride)

    samples2 = np.array([self.img[row - self.length:row:self.sampling_rate]
                        for row in rows])
    samples = np.array([self.data[row - self.length:row:self.sampling_rate]
                        for row in rows])
    targets = np.array([self.targets[row] for row in rows])

    if self.reverse:
        return samples[:, ::-1, ...], targets
    return [samples2, samples], targets

def get_config(self):
    '''Returns the TimeseriesGenerator configuration as Python dictionary.
    # Returns
        A Python dictionary with the TimeseriesGenerator configuration.
    '''
    img = self.img
    if type(self.img).__module__ == np.__name__:
        img = self.img.tolist()
    try:
        json_img = json.dumps(img)
    except TypeError:
        raise TypeError('Data not JSON Serializable:', img)

    data = self.data
    if type(self.data).__module__ == np.__name__:
        data = self.data.tolist()
    try:
        json_data = json.dumps(data)
    except TypeError:
        raise TypeError('Data not JSON Serializable:', data)

    targets = self.targets
    if type(self.targets).__module__ == np.__name__:
        targets = self.targets.tolist()
    try:
        json_targets = json.dumps(targets)
    except TypeError:
        raise TypeError('Targets not JSON Serializable:', targets)

    return {
        'img': json_img,
        'data': json_data,
        'targets': json_targets,
        'length': self.length,
        'sampling_rate': self.sampling_rate,
        'stride': self.stride,
        'start_index': self.start_index,
        'end_index': self.end_index,
        'shuffle': self.shuffle,
        'reverse': self.reverse,
        'batch_size': self.batch_size
    }

def to_json(self, **kwargs):
    """Returns a JSON string containing the timeseries generator
    configuration. To load a generator from a JSON string, use
    `keras.preprocessing.sequence.timeseries_generator_from_json(json_string)`.
    # Arguments
        **kwargs: Additional keyword arguments
            to be passed to `json.dumps()`.
    # Returns
        A JSON string containing the tokenizer configuration.
    """
    config = self.get_config()
    timeseries_generator_config = {
        'class_name': self.__class__.__name__,
        'config': config
    }
    return json.dumps(timeseries_generator_config, **kwargs)

def timeseries_generator_from_json(json_string):
"""Parses a JSON timeseries generator configuration file and
returns a timeseries generator instance.
# Arguments
json_string: JSON string encoding a timeseries
generator configuration.
# Returns
A Keras TimeseriesGenerator instance
"""
full_config = json.loads(json_string)
config = full_config.get('config')
img = json.loads(config.pop('img'))
config['img'] = img
data = json.loads(config.pop('data'))
config['data'] = data
targets = json.loads(config.pop('targets'))
config['targets'] = targets

return TimeseriesGenerator(**config)

Thanks. Here is how I solved it following some ideas from issue 3386, maybe someone might find it useful
input_imgen = ImageDataGenerator(rescale = 1./255,
                                   shear_range = 0.2,
                                   zoom_range = 0.2,
                                   rotation_range=5.,
                                   horizontal_flip = True)

test_imgen = ImageDataGenerator(rescale = 1./255)

def generate_generator_multiple(generator,dir1, dir2, batch_size, img_height,img_width):
    genX1 = generator.flow_from_directory(dir1,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False,
                                          seed=7)

    genX2 = generator.flow_from_directory(dir2,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False,
                                          seed=7)
    while True:
            X1i = genX1.next()
            X2i = genX2.next()
            yield [X1i[0], X2i[0]], X2i[1]  #Yield both images and their mutual label

inputgenerator=generate_generator_multiple(generator=input_imgen,
                                           dir1=train_dir_1,
                                           dir2=train_dir_2,
                                           batch_size=batch_size,
                                           img_height=img_height,
                                           img_width=img_height)

testgenerator=generate_generator_multiple(test_imgen,
                                          dir1=train_dir_1,
                                          dir2=train_dir_2,
                                          batch_size=batch_size,
                                          img_height=img_height,
                                          img_width=img_height)

 history=model.fit_generator(inputgenerator,
                        steps_per_epoch=trainsetsize/batch_size,
                        epochs = epochs,
                        validation_data = testgenerator,
                        validation_steps = testsetsize/batch_size,
                        use_multiprocessing=True,
                        shuffle=False)

This code is almost perfect, but for me I have the problem that it just runs forever. If I replace the while true I could get it to work as I expected.

Thanks. Here is how I solved it following some ideas from issue 3386, maybe someone might find it useful
input_imgen = ImageDataGenerator(rescale = 1./255,
                                   shear_range = 0.2,
                                   zoom_range = 0.2,
                                   rotation_range=5.,
                                   horizontal_flip = True)

test_imgen = ImageDataGenerator(rescale = 1./255)

def generate_generator_multiple(generator,dir1, dir2, batch_size, img_height,img_width):
    genX1 = generator.flow_from_directory(dir1,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False,
                                          seed=7)

    genX2 = generator.flow_from_directory(dir2,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False,
                                          seed=7)
    while True:
            X1i = genX1.next()
            X2i = genX2.next()
            yield [X1i[0], X2i[0]], X2i[1]  #Yield both images and their mutual label

inputgenerator=generate_generator_multiple(generator=input_imgen,
                                           dir1=train_dir_1,
                                           dir2=train_dir_2,
                                           batch_size=batch_size,
                                           img_height=img_height,
                                           img_width=img_height)

testgenerator=generate_generator_multiple(test_imgen,
                                          dir1=train_dir_1,
                                          dir2=train_dir_2,
                                          batch_size=batch_size,
                                          img_height=img_height,
                                          img_width=img_height)

 history=model.fit_generator(inputgenerator,
                        steps_per_epoch=trainsetsize/batch_size,
                        epochs = epochs,
                        validation_data = testgenerator,
                        validation_steps = testsetsize/batch_size,
                        use_multiprocessing=True,
                        shuffle=False)

This code is almost perfect, but for me I have the problem that it just runs forever. If I replace the while true I could get it to work as I expected.

in fit.generator..what is trainsetsize and testsetsize???where it is defined??

Thanks. Here is how I solved it following some ideas from issue 3386, maybe someone might find it useful
input_imgen = ImageDataGenerator(rescale = 1./255,
                                   shear_range = 0.2,
                                   zoom_range = 0.2,
                                   rotation_range=5.,
                                   horizontal_flip = True)

test_imgen = ImageDataGenerator(rescale = 1./255)

def generate_generator_multiple(generator,dir1, dir2, batch_size, img_height,img_width):
    genX1 = generator.flow_from_directory(dir1,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False,
                                          seed=7)

    genX2 = generator.flow_from_directory(dir2,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False,
                                          seed=7)
    while True:
            X1i = genX1.next()
            X2i = genX2.next()
            yield [X1i[0], X2i[0]], X2i[1]  #Yield both images and their mutual label

inputgenerator=generate_generator_multiple(generator=input_imgen,
                                           dir1=train_dir_1,
                                           dir2=train_dir_2,
                                           batch_size=batch_size,
                                           img_height=img_height,
                                           img_width=img_height)

testgenerator=generate_generator_multiple(test_imgen,
                                          dir1=train_dir_1,
                                          dir2=train_dir_2,
                                          batch_size=batch_size,
                                          img_height=img_height,
                                          img_width=img_height)

 history=model.fit_generator(inputgenerator,
                        steps_per_epoch=trainsetsize/batch_size,
                        epochs = epochs,
                        validation_data = testgenerator,
                        validation_steps = testsetsize/batch_size,
                        use_multiprocessing=True,
                        shuffle=False)

This code is almost perfect, but for me I have the problem that it just runs forever. If I replace the while true I could get it to work as I expected.

in fit.generator..what is trainsetsize and testsetsize???where it is defined??

https://keras.io/api/models/model_training_apis/
https://www.pyimagesearch.com/2018/12/24/how-to-use-keras-fit-and-fit_generator-a-hands-on-tutorial/

yield [X1i[0], X2i[0]]

I am using a similar generator for prediction purposes. But, it is running infinitely. How can I resolve this?

Thanks. Here is how I solved it following some ideas from issue 3386, maybe someone might find it useful
input_imgen = ImageDataGenerator(rescale = 1./255,
                                   shear_range = 0.2,
                                   zoom_range = 0.2,
                                   rotation_range=5.,
                                   horizontal_flip = True)

test_imgen = ImageDataGenerator(rescale = 1./255)

def generate_generator_multiple(generator,dir1, dir2, batch_size, img_height,img_width):
    genX1 = generator.flow_from_directory(dir1,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False,
                                          seed=7)

    genX2 = generator.flow_from_directory(dir2,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False,
                                          seed=7)
    while True:
            X1i = genX1.next()
            X2i = genX2.next()
            yield [X1i[0], X2i[0]], X2i[1]  #Yield both images and their mutual label

inputgenerator=generate_generator_multiple(generator=input_imgen,
                                           dir1=train_dir_1,
                                           dir2=train_dir_2,
                                           batch_size=batch_size,
                                           img_height=img_height,
                                           img_width=img_height)

testgenerator=generate_generator_multiple(test_imgen,
                                          dir1=train_dir_1,
                                          dir2=train_dir_2,
                                          batch_size=batch_size,
                                          img_height=img_height,
                                          img_width=img_height)

 history=model.fit_generator(inputgenerator,
                        steps_per_epoch=trainsetsize/batch_size,
                        epochs = epochs,
                        validation_data = testgenerator,
                        validation_steps = testsetsize/batch_size,
                        use_multiprocessing=True,
                        shuffle=False)

I have 2 inputs and each one have 5 classes, 30 images. And I use this code but only get
Found 150 images belonging to 5 classes. ---> training data 1
Found 150 images belonging to 5 classes. ---> validation data 1

But i think it should be like this.
Found 150 images belonging to 5 classes. ---> training data 1
Found 150 images belonging to 5 classes. ---> training data 2
Found 150 images belonging to 5 classes. ---> validation data 1
Found 150 images belonging to 5 classes. ---> validation data 2

How can I resolve this?


make hass-cli available in shell as an command

Problem/Motivation
would be awesome if the ide container had hass-cli available similar to addon-ssh.
then i wouldn't need to jump between tabs and could end up doing some nice demos
and find some interesting usecases for hass-cli template.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

Can't be done at this point, see PR #36.


Update sbt to 1.5.5

Updates org.scala-sbt:sbt from 1.5.4 to 1.5.5.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.scala-sbt", artifactId = "sbt" } ]

labels: library-update, semver-patch

Codecov Report

Merging #191 (34d1e28) into master (4696822) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #191   +/-   ##
=======================================
  Coverage   41.66%   41.66%
=======================================
  Files           1        1
  Lines          36       36
  Branches        1        1
=======================================
  Hits           15       15
  Misses         21       21

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4696822...34d1e28. Read the comment docs.


No entity: weather.astroweather_LONGITUTE_LATITUDE

Says to create a custom card with the below, however no device exists called entity: weather.astroweather_LONGITUTE_LATITUDE
Long & Lat are separate entities.  Believe the documentation needs to be updated most likely to use weather.astroweather_ds5ss6g7njg1

During the initial configuration of the integration, the weather entity is created with the above name. LONGITUDE and LATITUDE are replaced with the values you have configured for your location

resolved

Sorry for "abusing" of an old issue, but I too don't understand what entity name I have to give. In the entities list, if I search for weather.astrow it won't find anything. Then whether I use weather.astroweather_45.863400_9.382268 or weather.astroweather_45863400_9382268 it still won't find the entity. I cannot figure out what I'm doing wrong, if anybody can help... Thank you

You did install the AstroWeather custom integration, or?

You did install the AstroWeather custom integration, or?

It seems so

then you should have a weather.astroweather entity in your states list. Did you check this?

I don't have it

which version of home assistant are you running? You should be on 2023.9+.
Otherwise enable debug logging for astroweather please and post the logs

2023.8.0 😓

Upgraded and worked, thank you! Sorry for bugging

🎉


Spanish translation

Hello @ratiw, I would love to collaborate with the community translating the vuetable tutorial to spanish language.

@sigfriedCub1990 That's really great to hear. I'm quite occupied with other work at the moment, but please let me know what you need to get started.

Thanks @ratiw, for the quick response :), I'll fork to repository and I'll start to translate the tutorials. I'll keep you informed.

Unable to work on this for now.


chore: deprecated for username on user

Do we need to also expose the alternative Handle field for folks to switch to now, or is it ok to deprecate without the alt for now?

I think it's ok to deprecate for now: we don't have a lot of API user so at this stage this is more a self documentation for us than something we're communicating to external developers. Eventually when we deprecated API in the future it might make sense to already have an alternative.
I was also trying to understand how to expose an property in our graphql is deprecated because we never have done so in the past.


Support running MC in private subnet in AWS without disabling webhook validation

When machine-controller is running in a private subnet in AWS without NAT gateway, it is possible to rely on amazon api gateway in order to access EC2 API. The same is not possible for IAM API, and this makes the machine and machinedeployment validation fail because the instance profile existence check relies on it (see https://github.com/kubermatic/machine-controller/blob/56a7f0e/pkg/cloudprovider/provider/aws/provider.go#L446).
The goal of this task is to define and implement a strategy to support this use-case without disabling the validation.
Some possible options:

Introduce a flag to disable instance profile check
Make it best effort e.g. in case of connection timeout when contacting IAM API don't consider as a failure
Reduce scope of the check and only verify that is present

/reopen

During the creation of offline AWS Networks this issue is sill showing up, see https://github.com/kubermatic/offline-reference-setup/issues/78


Reducing redundant code

In DepCleanMojo.java file while printing dependencies on console initial code was repeating same amount of lines so just added a method for it.
Also want to know if we could remove sizeOfDependencies from the parameters as it is globally accessible by all the methods.
Thanks and Regards
Dhruvil Shah

Hey! @dhruvil-shah,
All LGTM but I think you didn't follow the checkstyle ruleset.
Run mvn checkstyle:checkstyle
Open /target/site/checkstyle.html in any browser
Solve all the errors for your code and then try again.
Thank you
Abhay

Hi @dhruvil-shah, thanks for the refactoring suggestion. Please do as @ABHAY0O7 says, just follow the Google Java style conventions. Your PR is good.

Hey @ABHAY0O7 Which ide you're using because I'm not able to configure checkstyle with VS Code

Hey! @dhruvil-shah, You didn't need to configure checkstyle it is already configured in the project's pom.xml file.
Just follow the steps and try to run these commands that I am mentioning here on your terminal.

Run mvn clean
Run mvn compile
Run mvn site
Run mvn checkstyle:checkstyle
Note - If these above commands are not running successfully i.e. BUILD FAILED then you got some problems with the project setup.
Now go to depclean-maven-plugin/target/site/checkstyle.html and open this file in any browser
There you will found lots of errors but you have to solve only those errors which are coming from your code.
After solving errors push the changes to the remote.

And in the end, I have a suggestion for you i.e. try to use IntelliJ IDEA ( I use this one ) or Eclipse IDE because these IDEs are specially designed for java based projects and related development.
Correct me @cesarsotovalero if I am wrong somewhere.
Thank you
Abhay.

Thanks much man @ABHAY0O7

Done updatedPR
Thanks @cesarsotovalero @ABHAY0O7 for the guidelines.

Hey @ABHAY0O7
When I make PR without that files it fails build checks c69a6cb.
I saw your PR got accepted, so Is there any thing I'm missing.
Because on my local system the build's are successfull.
Thanks and Regards
Dhruvil Shah

Correct me @cesarsotovalero if I am wrong somewhere.

You're totally right @ABHAY0O7. Thanks for your very concrete explanation of the setup. As you mentioned, it is very easy configuring Chekstyle in IntelliJ, there's even a very useful plugin for that that allows displaying the issues in your working file.

Hey @ABHAY0O7
When I make PR without that files it fails build checks c69a6cb.
I saw your PR got accepted, so Is there any thing I'm missing.
Because on my local system the build's are successfull.
Thanks and Regards
Dhruvil Shah

Hey @dhruvil-shah,
The PR you have mentioned is already closed therefore I am not able to see where your build fails.
Or I will also give you advice that for one particular change you already created 3 PRs and this one is also still open.
So I will suggest you stay on one PR and try to commit to it not make different PRs for a single issue.
And close those PRs on which you are not working.
Thank you
Abhay

Hey @dhruvil-shah and @cesarsotovalero,
I want to know that, anyone from you is working on this PR?
If no then may I complete this?
Thank you

Hi @dhruvil-shah, I just fixed the Checkstyle error and merged your changes, see https://github.com/castor-software/depclean/pull/69. Looking forward to your new contribution.


WebHooks for services don't work

Rancher Versions: 4/14
Steps to Reproduce:

Create a scaling group and add a container
Try to add a webhook to the service

Results: 	Can only create webhooks for Services. The supplied service is of type scalingGroup

@deniseschannon "service" is not changing to "scalingGroup" anymore


Add preview in the front-end

Add the ability to preview the gravatars in the front-end.

Resolved by ea1c88f1e4a5d884d30d5118024b83810a9b582d


Tag for 2.5.0

Hi,
Would it be possible to create a tag and a release on github for wsmancli version 2.5.0 ?
Thanks.

Done. Sorry for missing that.


[VL] Support columnar collect limit

What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
(Fixes: #ISSUE-ID)
How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

@ulysses-you I noticed that there is an existing PR about collect limit support but may have some different coverage. You may can take a look to avoid some duplicated work. FYI.

@ulysses-you I noticed that there is an existing PR about collect limit support but may have some different coverage. You may can take a look to avoid some duplicated work. FYI.

Seems to be that this PR was opened a little bit earlier than https://github.com/apache/incubator-gluten/pull/5269.
So @ulysses-you @guixiaowen thank both of you for taking this part of work. Do you think we should decide using which PR before continuing on this?

@ulysses-you I noticed that there is an existing PR about collect limit support but may have some different coverage. You may can take a look to avoid some duplicated work. FYI.

Seems to be that this PR was opened a little bit earlier than #5269.
So @ulysses-you @guixiaowen thank both of you for taking this part of work. Do you think we should decide using which PR before continuing on this?

@zhztheplayer  @ulysses-you  hi both, I'm very sorry for not paying attention to this review. I replied a bit late. I mentioned issues before and didn't notice that @ulysses-you  had already been submitted before me. If the evaluation criteria were based on the order of submission, @ulysses-you's  should be used.  @zhztheplayer  Thank you for your review. I will close my pr later.

If the evaluation criteria were based on the order of submission, @ulysses-you's should be used.

@guixiaowen We don't have strong criteria on this, so don't worry. Let's just decide using which one first so we can avoid duplicating work so wasting contributor's time.

@guixiaowen I just confirmed with @ulysses-you, let’s continue with your pr #5269. So please proceed. Thank you!

@guixiaowen I just confirmed with @ulysses-you, let’s continue with your pr #5269. So please proceed. Thank you!

@zhztheplayer Ok. Thank you  for your review.

thank you @zhztheplayer and @guixiaowen, I'm on vacation so please go ahead with your pr.


Feature/support listen to multiple netns

This PR come to solve the problem introduce by mtls implemented by various Service Mash solution such as Istio.
With mtls enabled the tapper see encrypted traffic, preventing it from showing the user its traffic.
Service mash implements mtls by running a sidecar container inside a POD, which in turn responsible for the encryption of the data. It then passes the decrypted data to the real container which run in the same network namespace, The network namespace isolating the traffic from the host machine, preventing the tapper from capturing the decrypted packets.
In this PR we introduce a way to listen to a traffic in an isolated network namespace, future PRs, will discover Envoy proxy automatically and configure the tapper accordingly.

@dvdlevanon please add a PR description that elaborates these questions:

Why is there a need for listening different Linux network namespaces? Why just opening the any device is not enough?
What's the end goal of this PR?

Are we currently not able to capture the traffic in certain Kubernetes cluster configurations? If so please give examples.
or is it about TLS deciphering?

If the PR is about TLS deciphering, how is it relevant? Is it about not having a cooked capture header like DLT_LINUX_SLL but instead having DLT_EN10MB?

@dvdlevanon please make the PR title a little bit more descriptive and write it in imperative mood like: Add support of ...

@dvdlevanon please add a PR description that at least elaborates these questions:

Why is there a need for listening different Linux network namespaces? Why just opening the any device is not enough?

What's the end goal of this PR?

Are we currently not able to capture the traffic in certain Kubernetes cluster configurations? If so, please give examples. (test cases)
or is it about TLS deciphering?

If the PR is about TLS deciphering, how is it relevant? Is it about not having a cooked capture header like DLT_LINUX_SLL but instead having DLT_EN10MB?

@mertyildiran Added a description to the PR.

Are we currently not able to capture the traffic in certain Kubernetes cluster configurations? Yes. Just run a cluster with istio and see it
If the PR is about TLS deciphering, how is it relevant? Please see the description, I never heard about DLT_LINUX_SLL or DLT_EN10MB

@mertyildiran

Conventional Commits is not followed by this project but Feature/ prefix is also not Conventional Commits.

I'm not sure I understand what it means, please let me know and I'll follow the convention.

@mertyildiran

Conventional Commits is not followed by this project but Feature/ prefix is also not Conventional Commits.

I'm not sure I understand what it means, please let me know and I'll follow the convention.

No, you don't need to follow the Conventional Commits. The PR title seems OK now.

Please resolve the conflicts.


adding my university

can you please add my university to get a free license jetbrains
institut supérieur d'informatique et de technique de communication

@souleimagara I am sorry, your request requires changing too many files. Please make a new request, which includes just one file, which corresponds your university only. Thank you.
I have to close his request due to incorrect content.


new fields photo, description and invite_link to the Chat object.

Added the new fields photo, description and invite_link to the Chat object.

Implemented in ce0d192ed314504d6d7088fc04f9a0f28fa7a826


Broken link in the 4.15.1 docs

Please update the main docs :
deb http://server.url/cloudstack/repo/binary ./
wget -q -O - http://server.url/cloudstack/repo/binary/KEY.gpg | sudo apt-key add -
Url : http://docs.cloudstack.apache.org/en/<IP_ADDRESS>/installguide/building_from_source.html

@abdelouahabb This is when the user builds packages from source and signs it with his / her custom key. The key is to be uploaded to a their custom webserver
http://docs.cloudstack.apache.org/en/<IP_ADDRESS>/installguide/building_from_source.html#repository-signing

cc @wido @GabrielBrascher can you check this issue and specific docs URL? Do we need to update that wrt Ubuntu?

Please update the main docs :
deb http://server.url/cloudstack/repo/binary ./
wget -q -O - http://server.url/cloudstack/repo/binary/KEY.gpg | sudo apt-key add -
Url : http://docs.cloudstack.apache.org/en/<IP_ADDRESS>/installguide/building_from_source.html

I don't see a broken link, please specify

this makes a confusion, maybe it should be written <http://server.url/> to make it clear that this is a variable and it is used to make a custom repo

@abdelouahabb I have created a documentation PR to address this issue, please review: https://github.com/apache/cloudstack-documentation/pull/235 cc. @DaanHoogland @rhtyd

Docs PR merged https://github.com/apache/cloudstack-documentation/pull/235


Initialize Repository: Integrate with Drizzle & GraphQL

[ ] Initialize NestJS project. (in docs)
[ ] Integrate with GraphQL. (in docs)
[ ] Integrate with Drizzle. (probably not in docs)

Didn't Khail say that he would do this?

@Holo-xy Mention so we can get a notification.
I think @alikehel is initializing frontend & monorepo only. He might do the first step  Initialize NestJS project. (in docs) in the process.
أو إلحقه قبل ما يعملها 😁

He might do the first step Initialize NestJS project. (in docs) in the process.

Yes I was gonna do this lol.

He might do the first step Initialize NestJS project. (in docs) in the process.

Yes I was gonna do this lol.

Ok do it and @Holo-xy will continue with the remaining steps.

Hello @Holo-xy, @alikehel is finished with setting up the repo, you can work on the remaining tasks.


contrib/raftexample: save snapshot order

The documentation for the etcd WAL package says of wal.Open's snapshot argument:

The snapshot must have been written to the WAL.

While looking over the example Raft use in contrib/raftexample, I noticed these lines for saving a new snapshot:
func (rc *raftNode) saveSnap(snap raftpb.Snapshot) error {
	if err := rc.snapshotter.SaveSnap(snap); err != nil {
		return err
	}
	walSnap := walpb.Snapshot{
		Index: snap.Metadata.Index,
		Term:  snap.Metadata.Term,
	}
	if err := rc.wal.SaveSnapshot(walSnap); err != nil {
		return err
	}
	return rc.wal.ReleaseLockTo(snap.Metadata.Index)
}

https://github.com/coreos/etcd/blob/933aa09b73ef0b3556a1c663f9bd2bd92b714be9/contrib/raftexample/raft.go#L109-L121
For my understanding of the code, the wal.Open invariant can be violated if the process crashes between saving the snapshot to the file system and saving the snapshot index to the WAL. On the next startup, the snapshotter would load the most recent snapshot which was never saved to the WAL.
Looking at etcdserver's code, etcd seems to avoid this by writing the index to the WAL before persisting the snapshot. Is that the correct sequence of persisting a snapshot? If so, I'd be happy to submit a PR updating the raft example.

The ordering in etcdserver does seem to be deliberate given 1231f82f2243d951be8bcfade00c87779f71e471 (from #2597).


Fix control flow evaluation for empty functions

Fixed an exception while evaluating control flow for empty functions. e.g.
function add(a: int, b: int): int
{
}

When the body of the function is empty, the BasicBlock for the function has no statements. When AllPathsReturn tries to get the last statement to check it is a BoundNodeKind.ReturnStatement, an exception is thrown because the sequence is empty.
Unhandled Exception: System.InvalidOperationException: Sequence contains no elements

Don't know why the check failed. It says:

/usr/share/dotnet/sdk/2.2.105/NuGet.targets(499,5): error : Could not find a part of the path '/tmp/NuGetScratch/f9f00627-b75d-4d70-be29-4d6565f6ecd1'. [/home/vsts/work/1/s/src/mi/mi.csproj]

I'm sorry I don't have experience with Azure Pipelines. How can I fix it?


demo is not working

http://thecodinghouse.in/themes/ returns 404

Demo site is no longer available. I have removed the link from the read me.


Error when timestamp casting for time unit millisecond or microsecond

I am implementing the support for the following expression in polars using the cast function under the compute module of this library, but I am getting the following error when using millisecond or microsecond time units.

import polars as pl

 df = pl.DataFrame(
        {"x1": ["2021-12-19T16:39:57-02:00"]}
    ).with_columns(
        **{
            "x1-datetime-ms": pl.col("x1").cast(pl.Datetime(time_unit="ms")),
            "x1-datetime-us": pl.col("x1").cast(pl.Datetime(time_unit="us")),
        }
    )

Output:
cast failed: NotYetImplemented("Casting from Utf8 to Timestamp(Millisecond, Some(\"-02:00\")) not supported")
I would expect them to cast the date time without issues. I can provide a proposal PR to support this if you are ok with it.
Also the following date time format is not supported '2021-12-19 16:39:57', When I try to cast using it returns None. Should I open a new issue for this or could we handle it here?

Already provided a PR. I Also need to support different timestamp formats cause the only one supported right now is %Y-%m-%dT%H:%M:%S%.f%:z but I guest I have to put those changes on a different PR.


feat(396): have streetview modal take full page

Description
This PR corrects the Street View modal to have it take the full screen, overlaying the navigation bar as well
Demo
https://github.com/CodeForPhilly/vacant-lots-proj/assets/8061917/88d7e8ef-16da-4230-a776-1d7dabf79f5f
Closes #396

Looks great!


Enable mcux rtc driver on all Kinetis SoCs/boards

All in-tree Kinetis SoCs have the same rtc hardware as kw41, so we can extend the mcux rtc driver from #7182 to work on all Kinetis boards. Make sure tests/drivers/rtc/rtc_basic_api builds and passes

As far as I can tell, the following in-tree SoCs already enables support for the RTC:

k64f
k66f
k8xf
ke1xf
kw41z

The following in-tree SoCs should support it, but does not enable it yet:

k22f
kw22d
kw24
kl25z
kw40z


Not working as expected in Vaadin 7.6.3(+)?

I like your add-on but I have problems getting it to work as expected. I tried several things to get it running:

Placed under TabSheet
No lazy loading is done and it gives the JavaScript warning:
Sun Jan 08 17:05:26 GMT+100 2017 org.vaadin.alump.lazylayouts.client.LazyVerticalLayoutConnector
SEVERE: Failed to connect to scrolling events of com.vaadin.client.ui.tabsheet.TabsheetConnector

Placed under Panel
This loads next content over and over again without waiting for the user to scroll down.

Placed under Window
This resets the scroll position to top when content was lazy loaded.

I tried Vaadin versions 7.6.3 and 7.7.6, both with the same issues. Would be nice if you can have a look at it. Maybe something has changed in new Vaadin versions.
Have a nice day,
Steffen

I need to have look at this soon, and also verify functionality with Vaadin 8.

I am trying to use this addon with Vaadin 8.1.0 within a normal Panel (which is in an other VerticalSplitPanel as second component) and the first loading of the contents seems to work fine.
But then it doesn´t reload the next content if the user scrolls down. The loading animation is shown but nothing happens. The listener onLazyComponentRequest is never called again after the first time.

https://github.com/alump/LazyLayouts/blob/master/lazylayouts-demo/src/main/java/org/vaadin/alump/lazylayouts/demo/Issue7View.java - I made this test with 8.1.1 and it worked.

Check that you didn't define height of LazyVerticalLayout inside the panel. Also the panel should have it's height defined (usually 100% if you want it to take all available space in half of SplitPanel).

Closing as no response.


Add ability to redact operations and results

Is your feature request related to a problem?
I would like to log GraphQL operations sent to an API and responses from the API for operational purposes and data analytics. However, some arguments and fields are sensitive, so the values should be redacted prior to being logged. I would like a way to mark arguments and fields as "sensitive" on an individual basis and have a rewriter which redacts all the sensitive data in a query/mutation.
I have tried to do this with a custom directive but ran in to problems. When I extend QuerySyntaxRewriter and apply it to IRequestContext.Operation.Definition, the latter has no schema information, so when it encounters a syntax node, there is no way to check whether that syntax node corresponds to a field/argument with a "sensitive" directive. The only mildly useful information is argument/field name, but redacting on this basis is very coarse and would result in false positives. It would also mean the code deciding whether to redact a field/argument is very distant from the field/argument definition itself, reducing maintainability.
The solution you'd like
The ideal solution would be either a "Sensitive" GraphQL directive or a CLR attribute which can be applied to, at the very least, fields and arguments of type String or String!. There would be a syntax rewriter which could rewrite queries/mutations to redact the sensitive data:

Any sensitive field/argument whose value is a literal would be redacted to a fixed string, e.g. ***.
If a variable is passed to a sensitive field/argument, then that variable's value would be redacted.

There would also be (with lower priority) a rewriter for results. Sensitive fields would have their returned values redacted.
Any application of the Sensitive directive/attribute to a schema element where it has no effect would result in a warning/error, to guard against accidentally leaking sensitive data.
Product
Hot Chocolate

I think this belongs into a community package or into userland not into the core library as I can think of a dozen variations on how people want to have that. There are also other areas this leads into like auditing of data that has be exposed to certain users and so on.
I am closing this issue as we will not pick this up anytime soon.

In that case, do you have any suggestions on how one would implement something like this? The logging is happening inside a custom ExecutionDiagnosticEventListener subclass, so we have access to IRequestContext. Here are the things I tried, all with variants of the Sensitive directive described above:

Traversing the structure of IRequestContext.Operation.GetRootSelectionSet(). Unfortunately there is no OOTB visitor/walker for this, and traversing the structure requires me to provide a lot of type information which is not easily accessible at this point. Continuing with this approach feels like it would be duplicating a lot of the core query processing logic. There is a method Operation.Print which might work if there was a way to inject custom rewriting/serialization into it, but currently there isn't.
Traversing IRequestContext.Operation.Definition. There are OOTB visitors/walkers for this, but using it is impossible because there is no schema information here, only syntax. This would be the best approach if it worked, because it does granular redaction (unlike (3)) and operates on the original operation, not a compiled form (unlike (2))
Field middleware which detects whether some part of the operation has sensitive data so we can at least redact the whole operation in that case. This is impossible because I want to apply it to arguments and input fields, and field middleware doesn't work for those.

Regarding arguments there are a couple of ways:
https://www.youtube.com/watch?v=yJ-2kxrs2TU
https://www.youtube.com/watch?v=4n0dYBJ4SDA
Each video shows a different approach.
To implement this are a couple of pieces. But all possible, you can extend each part of Hot Chocolate.
I think I would first identify in which area sensitive information can be, as I would not want to cause perf degradation for  things that cannot have such an issue.
Next you could use descriptor attributes or type interceptors to annotate to these things middleware or input formatter.
You could also traverse and rewrite the result structure ... not sure if I would do that actually... really depends on a lot of factors. If you want to access the request context in a mutable way use a request middleware.
https://www.youtube.com/watch?v=Ut33sSTYmgw

Walking you through on how to implement such a thing is a lot of work.  You can join us on slack.chillicream.com and ask questions there. The links I posted are to our youtube channel where I show a lot of low level APIs of HC to change the behavior.
If all the free resources are not enough, we are also offering consulting:
https://chillicream.com/services/advisory

BTW ... in Hot Chocolate 13 we have the new ResultFormatterDefinition which essentially is a sync middleware. Meaning you can use those on fields that are inlinable to add formatting logic like yours. This has a lot less impact on perf since these resolvers can be folded into their parent.


Issue #66: Batch save should return the failed entities

Mapping back is quite hard - and test cases are missing

Codecov Report

Merging #104 into master will decrease coverage by 0.17%.
The diff coverage is 34.78%.

@@             Coverage Diff              @@
##             master     #104      +/-   ##
============================================
- Coverage     40.92%   40.74%   -0.18%
+ Complexity      338      334       -4
============================================
  Files            64       64
  Lines          1757     1750       -7
  Branches        344      340       -4
============================================
- Hits            719      713       -6
- Misses          927      930       +3
+ Partials        111      107       -4

Impacted Files
Coverage Δ
Complexity Δ

...pository/support/SimpleDynamoDBCrudRepository.java
30% <0%> (-3.34%)
5 <0> (ø)

...in/spring/data/dynamodb/core/DynamoDBTemplate.java
24.69% <57.14%> (-2.68%)
6 <0> (-4)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update be2a72e...4fd079c. Read the comment docs.


Adds static maps export dialog

This PR introduces the new Static maps export dialog, that allows to request an image of the map the user is working on.
Important files

static_image_dialog.js
static_image_dialog.css.scss
mapview.js

Screenshots

Frontend tests were OK :+1: (details)

Frontend tests were OK :+1: (details)

Frontend tests were OK :+1: (details)

Frontend tests were OK :+1: (details)

Frontend tests were OK :+1: (details)

Frontend tests were OK :+1: (details)

Could you please review it, @CartoDB/frontend & @CartoDB/design?

Could you please review it, @CartoDB/frontend & @CartoDB/design?

+1

Code wise looks legit :+1:
Only one question, Don't forget to give proper attribution:, where/how should one "give proper attribution"?

Thanks, @viddo!
Regarding the attribution: I guess it's up to them (they could add it on top of the map, next to it, in an index, etc.), but maybe we could clarify the importance or the need to do that or extend a little the information we give them. What do you think, @saleiva, @jatorre, @javisantana, @andrewxhill?

No, I think it is enough!

:es: please, update news if needed and merge

Can we prioritize this?

Hey @andrewxhill we might want to do a blogpost, right? We take care of the assets.

totally!
On Tue, Feb 17, 2015 at 10:14 AM, saleiva<EMAIL_ADDRESS>wrote:

Hey @andrewxhill https://github.com/andrewxhill we might want to do a
blogpost, right? We take care of the assets.
—
Reply to this email directly or view it on GitHub
https://github.com/CartoDB/cartodb/pull/2199#issuecomment-74682962.

--
andrewxhill http://twitter.com/andrewxhill
http://cartodb.com

@saleiva: I'm going to release a new version of cartodb.js, release to production and then I'll take care of this.

Frontend tests were OK :+1: (details)

Frontend tests were OK :+1: (details)

Frontend tests were OK :+1: (details)

Frontend tests were OK :+1: (details)

Released!

Frontend tests were OK :+1: (details)

\o/


Lazy loading of stories

Component & pages .exs files are compiled as a whole at compilation time (triggered by the compilation of the storybook backend module defined in the user application).
As the codebase grows, this could induce increased compilation times.
Components & pages exs files should instead:

be eagerly compiled in prod env
be lazily compiled at page load in any other environment

Regarding prod being eager, that's likely important but we need to be careful because most people likely won't run storybooks in prod?

I do run the storybook in prod. I think it is a very valuable tool to share a design system with UI/UX designers
But we can make it an opt-in setting (disabled in prod by default)


Add alternate language support

Not saying we have to do this for every language, but I think it would be good to add the framework to allow for easy translation of the hard-coded text in the app (instructions, labels, etc.)
The way I propose we do this is similar to how it is done in Android Studio: where all hard-coded text is kept in a separate file and imported at runtime depending on the language setting. (See android studio documentation on string resources: https://developer.android.com/guide/topics/resources/string-resource) This way, a translator can just add the translations in the strings file without having to know where every hard-coded string is in the code, making contributing translations much more accessible for someone who isn't familiar with coding.

 Following this guide now https://medium.com/simform-engineering/localize-your-apps-to-support-multiple-languages-ios-localization-in-swiftui-c72d891a3e9


New 3ds?

Does this work on the New 3ds?

Technically, because this is not a software hack, it's a hardware hack. All you need to do is find the right test points. So yes is the short answer, but no because it has a new circuit board.

Is there any mapping for the new test points? I don't have a New 3ds yet so I can't look.

I have a new 3DS XL and I can try to help if you point me in the right direction.  I'm currently trying to confirm if there are any difference in setup between the two before I start the mod.

Never mind, he includes the new as well in his google doc. https://docs.google.com/document/d/1-dCfkwnoIebxSwfW9gZ9NFpktMHbVTEVkIrduLGLTMM/mobilebasic?pli=1


TypeScript provides two identical tasks

create an empty folder
open the empty folder in VS Code
create a file test.ts
in the terminal run 'tsc --init'
run Tasks>Run Build Task...

The same tasks is shown twice

After running the build task once, then the next time only a single task is proposed:

(assigning to July, since this can be part of the getting started experience)

I believe this should have been fixed. I added a check to make sure we don't return the tsconfig at a given path more than once

Cannot reproduce the issue in 1.14.2, so like Matt says this must have been fixed in the previous iteration


Change transactions module already imported check

Purpose
Previous way we did the check for transaction module already imported is not sound when
ballerina build -a is used as there will be only one compiler context for all module in this build execution. Hence look at the import statement in pkgNode each time we need to determine whether transaction module is imported.
Fixes https://github.com/ballerina-platform/ballerina-lang/issues/19300
Approach

Describe how you are implementing the solutions along with the design details.

Samples

Provide high-level details about the samples related to this feature.

Remarks

List any other known issues, related PRs, TODO items, or any other notes related to the PR.

Check List

[ ] Read the Contributing Guide
[ ] Required Balo version update
[ ] Updated Change Log
[ ] Checked Tooling Support (#)
[ ] Added necessary tests

[ ] Unit Tests
[ ] Spec Conformance Tests
[ ] Integration Tests
[ ] Ballerina By Example Tests

[ ] Increased Test Coverage
[ ] Added necessary documentation

[ ] API documentation
[ ] Module documentation in Module.md files
[ ] Ballerina By Examples

Codecov Report

Merging #19301 into master will decrease coverage by 4.78%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #19301      +/-   ##
==========================================
- Coverage    20.5%   15.71%   -4.79%
==========================================
  Files          52       47       -5
  Lines        1395     1266     -129
  Branches      216      197      -19
==========================================
- Hits          286      199      -87
+ Misses       1096     1054      -42
  Partials       13       13

Impacted Files
Coverage Δ

composer/packages/tracing/src/ToolBar.tsx

composer/packages/tracing/src/TraceList.tsx

composer/packages/tracing/src/DetailView.tsx

composer/packages/tracing/src/TraceLogs.tsx

composer/packages/tracing/src/index.ts

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b0b2dc7...8ddb0a4. Read the comment docs.

@rdhananjaya Can you add a test case for this?

@rdhananjaya Can you add a test case for this?

done