What is fractionally-strided convolution layer? In paper Generating High-Quality Crowd Density Maps using Contextual Pyramid CNNs, in Section 3.4, it said

  Since, the aim of this work is to estimate high-resolution and
  high-quality density maps, F-CNN is constructed using a set of
  convolutional and fractionally-strided convolutional layers. The set
  of fractionally-strided convolutional layers help us to restore
  details in the output density maps. The following structure is used
  for F-CNN: CR(64,9)-CR(32,7)- TR(32)-CR(16,5)-TR(16)-C(1,1), where, C
  is convolutional layer, R is ReLU layer, T is fractionally-strided
  convolution layer and the first number inside every brace indicates
  the number of filters while the second number indicates filter size.
  Every fractionally-strided convolution layer increases the input
  resolution by a factor of 2, thereby ensuring that the output
  resolution is the same as that of input.

I would like to know the detail of fractionally-strided convolution layer.

Here is an  animation of fractionally-strided convolution (from this github project):

where the dashed white cells are zero rows/columns padded between the input cells (blue). These animations are visualizations of the mathematical formulas from the article below:

A guide to convolution arithmetic for deep learning

Here is a quote from the article:

  Figure [..] helps understand what fractional strides involve: zeros
  are inserted between input units, which makes the kernel move around
  at a slower pace than with unit strides [footnote: doing so is
  inefficient and real-world implementations avoid useless
  multiplications by zero, but conceptually it is how the transpose of a
  strided convolution can be thought of.]

Also, here is a post on this site asking "What are deconvolutional layers?" which is the same thing.

And here are two quotes from a post by Paul-Louis Pröve on different types of convolutions:

  Transposed Convolutions (a.k.a. deconvolutions or fractionally strided
  convolutions)

and

  Some sources use the name deconvolution, which is inappropriate
  because it’s not a deconvolution [..] An actual deconvolution reverts the process of a convolution.


TypeError: Image data of dtype object cannot be converted to float I am trying to convert files stored in a CSV file dataset to grayscale. I run the code which worked on my previous dataset but I now get this error:
TypeError: Image data of dtype object cannot be converted to float
The code I produced is below:
import pandas as pd
import numpy as np
from sklearn.datasets import load_digits
import matplotlib.pyplot as plt

data_path = "pe_section_headers.csv"

data1 = pd.read_csv(data_path);

data = data1

#data = data.drop("Malware", axis=1)
#data = data.drop("Name", axis=1)
data = data.values
data = data.reshape(data.shape[0], data.shape[1], 1)
data = np.tile(data, (1, data.shape[1]))

for i in range(data.shape[0]):
    plt.tick_params(left=False, bottom=False, labelleft=False, labelbottom=False)
    plt.imshow(data[i], cmap="gray")
    plt.savefig(f"output_image_{data1.iloc[i,0]}.png")
    plt.close()

#print(data[0].shape)

And the error message I receive is:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-8-9f9e3cc13a98> in <module>
     21 for i in range(data.shape[0]):
     22     plt.tick_params(left=False, bottom=False, labelleft=False, labelbottom=False)
---> 23     plt.imshow(data[i], cmap="gray")
     24     plt.savefig(f"output_image_{data1.iloc[i,0]}.png")
     25     plt.close()

~/opt/anaconda3/envs/jacksprojectnew/lib/python3.6/site-packages/matplotlib/pyplot.py in imshow(X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, filternorm, filterrad, resample, url, data, **kwargs)
   2728         filternorm=filternorm, filterrad=filterrad, resample=resample,
   2729         url=url, **({"data": data} if data is not None else {}),
-> 2730         **kwargs)
   2731     sci(__ret)
   2732     return __ret

~/opt/anaconda3/envs/jacksprojectnew/lib/python3.6/site-packages/matplotlib/__init__.py in inner(ax, data, *args, **kwargs)
   1445     def inner(ax, *args, data=None, **kwargs):
   1446         if data is None:
-> 1447             return func(ax, *map(sanitize_sequence, args), **kwargs)
   1448
   1449         bound = new_sig.bind(ax, *args, **kwargs)

~/opt/anaconda3/envs/jacksprojectnew/lib/python3.6/site-packages/matplotlib/axes/_axes.py in imshow(self, X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, filternorm, filterrad, resample, url, **kwargs)
   5521                               resample=resample, **kwargs)
   5522
-> 5523         im.set_data(X)
   5524         im.set_alpha(alpha)
   5525         if im.get_clip_path() is None:

~/opt/anaconda3/envs/jacksprojectnew/lib/python3.6/site-packages/matplotlib/image.py in set_data(self, A)
    701                 not np.can_cast(self._A.dtype, float, "same_kind")):
    702             raise TypeError("Image data of dtype {} cannot be converted to "
--> 703                             "float".format(self._A.dtype))
    704
    705         if self._A.ndim == 3 and self._A.shape[-1] == 1:

TypeError: Image data of dtype object cannot be converted to float

Could anyone suggest what this means? Thanks

The data you are trying to plot (data[i]) contains data of type object (i.e. a string) which of course cannot be plotted as they are not numbers. This is caused by the fact that one of the columns in your dataset is a string which you are trying to plot, likely the Name column. Try removing the Name column from the dataset and see if there are other column which contains strings.


How is a splitting point chosen for continuous variables in decision trees? I have two questions related to decision trees:

If we have a continuous attribute, how do we choose the splitting value?

Example: Age=(20,29,50,40....)
Imagine that we have a continuous attribute $f$ that have values in $R$. How can I  write an algorithm that finds the split point $v$, in order that when we split $f$ by $v$, we have a minimum gain for $f>v$?

In order to come up with a split point, the values are sorted, and the mid-points between adjacent values are evaluated in terms of some metric, usually information gain or gini impurity. For your example, lets say we have four examples and the values of the age variable are $(20, 29, 40, 50)$. The midpoints between the values $(24.5, 34.5, 45)$ are evaluated, and whichever split gives the best information gain (or whatever metric you're using) on the training data is used.

You can save some computation time by only checking split points that lie between examples of different classes, because only these splits can be optimal for information gain.


Should I use rescale parameters for data augmentation? I am using Keras library to build a CNN model. I want to use data augmentation for training data.

Should I use rescale parameters for data augmentation?

ImageDataGenerator(rescale=1./255)

The rescale argument byitself does not augment your data.

If you input pixels are in the range of [0,255], you can rescale to [0,1] using the code in your question. Just remember to do the same on all partitions of your dataset.

If you input is already in the range of [0,1], then obviously you should not rescale.


What is the Q function and what is the V function in reinforcement learning? It seems to me that the $V$ function can be easily expressed by the $Q$ function and thus the $V$ function seems to be superfluous to me. However, I'm new to reinforcement learning so I guess I got something wrong.

Definitions

Q- and V-learning are in the context of Markov Decision Processes. A MDP is a 5-tuple $(S, A, P, R, \gamma)$ with

$S$ is a set of states (typically finite)
$A$ is a set of actions (typically finite)
$P(s, s', a) = P(s_{t+1} = s' | s_t = s, a_t = a)$ is the probability to get from state $s$ to state $s'$ with action $a$.
$R(s, s', a) \in \mathbb{R}$ is the immediate reward after going from state $s$ to state $s'$ with action $a$. (It seems to me that usually only $s'$ matters).
$\gamma \in [0, 1]$ is called discount factor and determines if one focuses on immediate rewards ($\gamma = 0$), the total reward ($\gamma = 1$) or some trade-off.

A policy $\pi$, according to Reinforcement Learning: An Introduction by Sutton and Barto is a function $\pi: S \rightarrow A$ (this could be probabilistic).

According to Mario Martins slides, the $V$ function is
$$V^\pi(s) = E_\pi \{R_t | s_t = s\} = E_\pi \{\sum_{k=0}^\infty \gamma^k r_{t+k+1} | s_t = s\}$$
and the Q function is
$$Q^\pi(s, a) = E_\pi \{R_t | s_t = s, a_t = a\} = E_\pi \{\sum_{k=0}^\infty \gamma^k r_{t+k+1} | s_t = s, a_t=a\}$$

My thoughts

The $V$ function states what the expected overall value (not reward!) of a state $s$ under the policy $\pi$ is.

The $Q$ function states what the value of a state $s$ and an action $a$  under the policy $\pi$ is.

This means,
$$Q^\pi(s, \pi(s)) = V^\pi(s)$$

Right? So why do we have the value function at all? (I guess I mixed up something)

$V^\pi(s)$ is the state-value function of MDP (Markov Decision Process). It's the expected return starting from state $s$ following policy $\pi$.
In the expression
$$V^\pi(s) = E_{\pi} \{G_t \vert s_t = s\} $$
$G_t$ is the total DISCOUNTED reward from time step $t$, as opposed to $R_t$ which is an immediate return. Here you are taking the expectation of ALL actions according to the policy $\pi$.
$Q^\pi(s, a)$ is the action-value function. It is the expected return starting from state $s$, following policy $\pi$, taking action $a$. It's focusing on the particular action at the particular state.
$$Q^\pi(s, a) = E_\pi \{G_t | s_t = s, a_t = a\}$$
The relationship between $Q^\pi$ and $V^\pi$ (the value of being in that state) is
$$V^\pi(s) = \sum_{a ∈ A} \pi (a|s) * Q^\pi(s,a)$$
You sum every action-value multiplied by the probability to take that action (the policy $\pi(a|s)$).
If you think of the grid world example, you multiply the probability of (up/down/right/left) with the one step ahead state value of (up/down/right/left).


Model accuracy: how to determine it? I have some doubts regarding the approach to build a classifier such as Multinomial Naive Bayes or SVM. I will go through the steps to see if the approach is fine. I have not a lot of experience in model build, so any suggestions would be great!
My dataset has approx. 1115 obs having positive value (0) and 66 obs having negative value (1).
The distribution of the dependant variable is shown in the figure below.
I split the dataset into train (70) and test (30), using stratify (it should help in case of such discrepancy between classes, hopefully):
from sklearn.model_selection import train_test_split

y=df['Label']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.30, stratify=y)

Then I imported the sim model to create a SVM classifier:
from sklearn import svm

clf = svm.SVC(kernel='linear')

clf.fit(X_train, y_train)

And to Predict the response for test dataset I used the following:
y_pred = clf.predict(X_test)

For accuracy calculation, I used the following code:
from sklearn import metrics

print("Accuracy:",metrics.accuracy_score(y_test, y_pred))

print("Precision:",metrics.precision_score(y_test, y_pred))

print("Recall:",metrics.recall_score(y_test, y_pred))

getting different values every time I re-run it:
Accuracy: 0.75

                precision    recall  f1-score   support

           0       0.95      0.79      0.86       316
           1       0.08      0.30      0.13        20

    accuracy                           0.76       336
   macro avg       0.51      0.54      0.49       336
weighted avg       0.90      0.76      0.82       336

2nd re-run
Accuracy:  0.8005952380952381

                precision    recall  f1-score   support

           0       0.94      0.84      0.89       316
           1       0.07      0.20      0.11        20

    accuracy                           0.80       336
   macro avg       0.51      0.52      0.50       336
weighted avg       0.89      0.80      0.84       336

Confusion Matrix:
 [[265  51]
 [ 16   4]]

3rd re-run
Accuracy:  0.7797619047619048

                precision    recall  f1-score   support

           0       0.94      0.81      0.87       316
           1       0.08      0.25      0.12        20

    accuracy                           0.78       336
   macro avg       0.51      0.53      0.50       336
weighted avg       0.89      0.78      0.83       336

Confusion Matrix:
 [[257  59]
 [ 15   5]]

I have a couple of questions on these results and I hope to find answers to them:

Which value should I take into account for saying that my model has accuracy of ...?
Does it make sense to run a model where there are so a few values = 1 for the dependent variable?

Which value should I take into account for saying that my model has accuracy of ...?

None. Accuracy is practically meaningless in such class imbalanced settings; the metrics of interest here are precision, recall, and f1 score.
Now, it's true that the values of these metrics also fluctuate between runs, much similarly with the reported accuracy. But this is to be expected due to the small-sample effects - your sample is so small that even a difference in the classification of a couple of samples in your (even smaller) validation set is enough to give the observed discrepancies.

Does it make sense to run a model where there are so a few values = 1 for the dependent variable?

Indeed it does; there are plenty of applications where the positive values are a very small percentage of the whole dataset (thing of sick people versus the general population of healthy ones, or engine faults versus long running times where no fault is present). That's why this is a (large enough) sub-topic of machine learning called class imbalance or imbalanced classification, with its own specific approaches. I suggest you start googling ruthlessly...


Will this MAPE implementation work for multidimensional output? I'm currently working on a CNN problem where the output is a 60x59 array of numerical values. I want to verify if the mean absolute percentage error (MAPE) function I'm employing will properly consider the error by matching each corresponding point to the true value, as opposed to unexpected behaviour. Will this formulation work?

def percent_mean_absolute_error(y_true, y_pred):
    if not K.is_tensor(y_pred):
        y_pred = K.constant(y_pred)
    y_true = K.cast(y_true, y_pred.dtype)
    diff = K.mean(K.abs((y_pred - y_true)) / K.mean(K.clip(K.abs(y_true),
                                                           K.epsilon(),
                                                           None)))
    return 100. * K.mean(diff)

K here is keras.backend. All I want to know is if this function properly considers the error by mapping the function properly from point to point from the input to the output?

You can either directly use the mean absolute percentage error (MAPE) function in Keras API for TensorFlow or use that function to test your custom function.

tf.keras.losses.MAPE code is avaible in the TensorFlow GitHub repo:

def mean_absolute_percentage_error(y_true, y_pred):
  y_pred = ops.convert_to_tensor(y_pred)
  y_true = math_ops.cast(y_true, y_pred.dtype)
  diff = math_ops.abs(
      (y_true - y_pred) / K.maximum(math_ops.abs(y_true), K.epsilon()))
  return 100. * K.mean(diff, axis=-1)


Statistical test for machine learning I want to prove that my proposed machine learning algorithm (prop_ml) is better than other baseline algorithms (ml_1, ml_2, ml_3) when given a small number of data for training. What I've done is to split a dataset into train and test sets. Then, I've randomly selected small k samples (10, 20, 30, ... 100) from the train set and used them to train the classifiers and used the test set for testing. I've replicated this 5 times to make sure I got some reliable results.

Now, I want to evaluate the results. Any suggestions on a statistical test that I can use to prove that the proposed ml is better or not? Thanks.

I did something like that when I was at university. Professor asked me to implement and compare a variant of k-nearest neighbor with the original one.

I've applied the 10 fold cross validation over several datasets with both the algorithms and then I've applied the Friedman test with Holm post hoc over the accuracy means. You can use also simple paired t-test or others, it depends on your needs.

My professor also suggested me to use this graphical tool.
Hope this helps

SPOILER

It turned out that my k-NN variant wasn't better!


What are the approaches to aggregate categorical variables? I am working on a clickstream dataset. I have come up with the following example dataset to explain my problem:

ClickTimeStamp        | SessionID | ART_weekOfYear | PagenameClicked | TimeSpentPerSession | CustID | ContractID | ... | TARGET |
2017-01-04 16:48:00   | 1         | 1              | P1              | 1                   | abc    | xyz        |     | 1      |
2017-01-04 16:48:53   | 1         | 1              | P2              | 1                   | abc    | xyz        |     | 1      |
2017-01-11 10:09:57   | 2         | 2              | P1              | 2                   | abc    | xyz        |     | 1      |
2017-01-11 10:11:24   | 2         | 2              | P4              | 2                   | abc    | xyz        |     | 1      |
2017-01-27 13:22:39   | 3         | 4              | P1              | 2                   | abc    | mnp        |     | 0      |
2017-01-27 13:24:01   | 3         | 4              | P7              | 2                   | abc    | mnp        |     | 0      |

The above dataset has clicks on its each row and TARGET is (let's say) contract was retained (1) or not (0). Keep in mind the TARGET is at contract level.

Now, I aggregate the above dataset as per my need (i.e. aggregate on contractID) and training set looks like this:

CustID | ContractID | ... | SessionID_conct | ART_weekOfYear_conct | PagenameClicked  | TimeSpentPerSession_avg | TARGET |
abc    | xyz        |     | "1-2"           |"1-2"                 | "P1->P2->P1->P4" | 1.5                     | 1      |
abc    | mnp        |     | "3"             |"4"                   | "P1->P7"         | 2                       | 0      |

PROBLEM:
For numerical features I just took average (as for TimeSpentPerSession_avg) but for categorical features it is not straightforward. In reality, my categorical features have very high cardinality, such as "PagenameClicked". So I cannot simply convert my categorical features to dummy variables and then aggregate them as numerical features.

I would like to know possible solutions to treat categorical features in such a way that dimensionality doesn't explode and I can also aggregate new representation on the contractID.

I have tried Entity Embeddings and read this paper for details. I transformed each categorical feature to an embedding representation of 16 dimension. However, now I am stuck at aggregating these embedding vectors for each contractID. Kindly let me know if anyone has worked in this direction or has a better solution.

Thanks allot for reading this question. :)

You problem is essentially you have high cardinality in your features, right? This will be relative to your problem, but you can look for mean encodings. Essentially, you will replace categories by the mean on target variable, however, this is highly prone to overfitting and you should take care.

The following two videos will give an excellent explanation:

https://www.coursera.org/learn/competitive-data-science/lecture/b5Gxv/concept-of-mean-encoding
https://www.coursera.org/learn/competitive-data-science/lecture/LGYQ2/regularization

Another idea is to group similar sequences into one segment, for example, let's say we have:

A->B->C and B->C

If this makes sense in your application, this could be transformed into one variable only.


What is difference between transductive and inductive in GNN? It seems in GNN(graph neural network), in transductive situation, we input the whole graph and we mask the label of valid data and predict the label for the valid data.

But is seems in inductive situation, we also input the whole graph(but sample to batch) and mask the label of the valid data and predict the label for the valid data.

In inductive learning, during training you are unaware of the nodes used for testing. For the specific inductive dataset here (PPI), the test graphs are disjoint and entirely unseen by the GNN during training.


Slow keras fit method with 100x100 array, how can I make it faster? how can I make this training faster ?
when I call the fit method on a 100 x 100 matrix goes very slow
my model it's a sequential
h = self.model.fit(
        inputs,
        targets,
        epochs=epochs,
        batch_size=16,
        verbose=1,
)

this is my matrix
def build(n):
    mat=np.ones(N*N)
    return mat.reshape((N,N))

this is the of my Qtraining
qt = Qtraining(model,
               env,
               n_epoch=200,
               max_memory=500,
               data_size=100,
               name='model100')

This is the experience method
def get_data(self, data_size=10):
        env_size = self.memory[0][0].shape[1]  # env_state 1d size (1st element of episode)
        mem_size = len(self.memory)
        data_size = min(mem_size, data_size)
        inputs = np.zeros((data_size, env_size)) # metti Nsize righe di 0 , e envSize elementi 0
        targets = np.zeros((data_size, self.num_actions))#
        for i, j in enumerate(np.random.choice(range(mem_size), data_size, replace=False)):
            env_state, action, reward, next_env_state, game_over = self.memory[j]
            inputs[i] = env_state
            # There should be no target values for actions not taken.
            # Thou shalt not correct actions not taken #deep (quote by Eder Santana)
            targets[i] = self.predict(env_state)
            # Q_sa = derived policy = max quality env/action = max_a' Q(s', a')
            Q_sa = np.max(self.predict(next_env_state))
            if game_over:
                targets[i, action] = reward
            else:
                # reward + gamma * max_a' Q(s', a')
                targets[i, action] = reward + self.discount * Q_sa

        return inputs, targets

using a 50 x 50 matrix I get 2500 cells, and in the construction of the neural network I have a 2500x2500 parameters + 2500 for a total of 6252500.
I think that slows down operations.
and there are only 3 Danse layers, the last one is size 4 because of the possible actions that are 4.
Is it possible to reduce the time of operation by adding more Danse layers?
this is my model
def build_model(env, **opt):
loss = opt.get('loss', 'mse')
a = opt.get('alpha', 0.24)
model = Sequential()
esize = env.maze.size
model.add(Dense(esize, input_shape=(esize,)))
model.add(LeakyReLU(alpha=a))
model.add(Dense(esize))
model.add(LeakyReLU(alpha=a))
model.add(Dense(num_actions))
model.compile(optimizer='adam', loss='mse')
return model


Target data values are not evenly distributed Data nature:

I have features with 10 numeric type, and other 10 categorical, with a lot of  values, at the end, using one-hot encoding I got a matrix of 600 columns. My problem is with accuracy which is 0.7, knowing that other peers got more that 0.9.

Problem:

Target data is binary, and is not evenly distributed at all. Trying blindly after pre-processing from sklearn.linear_model import LogisticRegression and sklearn.svm scored using roc_auc_score:  .7 and .75.

Back to basics, I run this

train['cible'].value_counts() / train['cible'].count()

and got

1    0.970791
0    0.029209
Name: cible, dtype: float64

Quite interesting I think, but how can I improve accuracy. Any hints ?

Note: I will edit and add False Positive Rate and True Positive Rate as I lost output, after scaling, missing data imputation and retraining the model which takes couple of hours.

From scikitlearn LogisticRegression docs:

  class_weight : dict or ‘balanced’, default: None

  Weights associated with classes in the form {class_label: weight}. If
  not given, all classes are supposed to have weight one. The “balanced”
  mode uses the values of y to automatically adjust weights inversely
  proportional to class frequencies in the input data as n_samples /
  (n_classes * np.bincount(y)). Note that these weights will be
  multiplied with sample_weight (passed through the fit method) if
  sample_weight is specified. New in version 0.17:
  class_weight=’balanced’

So try to add class_weight='balanced'in your call to LogisticRegression()

Or maybe if this doesn't work, try to use as trainSet an evenly split dataset: where the number of samples of class 1 is equal to class 0.


Can we use pca for supervise classification? My questions are:

Can we use "pca feature selection" for supervised classification? What will happen to labels when we use dimension reduction?
If I understand it right when we use pca for
feature extraction  after do it we have old features but in new
dimension then can we say these are the new features or mixed
features?
Is "feature extraction" different with "feature
selection" in pca? I mean they are two things to do with different
ways.
How can select effective features with pca?
There is a lots of references on the internet but I couldn't find
any MATLAB sample code or video (on youtube) about pca feature
selection, I mean not in theory, I'm looking for sample code in any language(Paython, Matlab,etc).

Let $X\in\mathbb{R}^{n\times m}$ be your data and $Z=XT\in\mathbb{R}^{n\times k}$ be the PCA transformed data.

  Can we use "pca feature selection" for supervised classification? What will happen to labels when we use dimension reduction?

Yes. It is commonly used for feature selection and linear dimensionality reduction. Nothing happens to the labels. At the start you have training pairs $(x_i,y_i)$, and after PCA you have $(z_i,y_i)$ instead.

  If I understand it right when we use pca for feature extraction after do it we have old features but in new dimension then can we say these are the new features or mixed features?

You have new features which are simply linear combinations of the old features. Another way to see it is that you chose a new set of (orthogonal) axes in the data space (potentially throwing away a few of them), and then projected your data onto this new subspace. So $Z$ is (literally) a mixture of the features from $X$.

  Is "feature extraction" different with "feature selection" in pca? I mean they are two things to do with different ways.

It can be both or either. To me, feature extraction implies obtaining latent descriptors from raw data. So if PCA is run on raw data, then it is doing feature extraction. Otherwise, it is not. As for feature selection, I would say whenever you remove any principal components (i.e., performing linear dimensionality reduction), then you are doing feature selection because you are selecting some features but not others.

  How can select effective features with pca? There is a lots of references on the internet but I couldn't find any MATLAB sample code or video (on youtube) about pca feature selection, I mean not in theory, I'm looking for sample code in any language(Paython, Matlab,etc).

People tend to either (a) choose a fixed variance to explain based on the normalized singular values (e.g., 90% of the original variance is explained) or (b) treat the number of selected features ($k$) as a hyper-parameter, and cross-validate over a validation set with it. Some discussion on the explained variance and its connection to the reconstruction error are here or here.

As for practical code samples, the scikit-learn library has some simple examples:

from sklearn import decomposition, datasets

k = 3
iris = datasets.load_iris()
X = iris.data
Y = iris.target
pca = decomposition.PCA(n_components=k)
pca.fit(X)
Z = pca.transform(X)


Unable to make CNN model with same output and input dimensions I'm trying to develop a simple CNN model that takes in a RGB images and returns the same 3 channel as output.
Eg: 3 x 128 x 128 is what I'm giving as input
K.set_image_data_format('channels_first')
def resBlock(x, channels, kernel_size=[3, 3], scale=0.1):
    tmp = Conv2D(channels, kernel_size, kernel_initializer='he_uniform', padding='same')(x)
    tmp = Activation('relu')(tmp)
    tmp = Conv2D(channels, kernel_size, kernel_initializer='he_uniform', padding='same')(tmp)
    tmp = Lambda(lambda x: x * scale)(tmp)

    return Add()([x, tmp])

feature_size = 128

input_shape = (3, 128,128)
inputs = Input(shape=input_shape)

# x = Concatenate(axis=1)([inputs])
x = Conv2D(feature_size, kernel_size=(3,3), activation='relu', input_shape=input_shape, padding='same')(inputs)
for i in range(6):
    x = resBlock(x, feature_size)

x = Conv2D(feature_size, (3, 3), kernel_initializer='he_uniform', padding='same')(x)
x = Add()([x, inputs])
model = Model(inputs=inputs,  outputs = x)

This is what I have currently, however, it throws an error:
ValueError: Operands could not be broadcast together with shapes (128, 128, 128) (3, 128, 128)

It fails in this line: x = Add()([x, inputs])

The error indicates that you are trying to add tensors of incompatible dimensions, as x has 128 channels and inputs has 3 channels.
The reason why x has 128 channels is just in the line above, where you pass feature_size with value 128 as the number of output channels. Change the number of output channels in that line to 3 and it should work, like this:
x = Conv2D(3, (3, 3), kernel_initializer='he_uniform', padding='same')(x)


Pandas grouped data to Bokeh graph I'm having trouble graphing Pandas grouped data in Bokeh.

company_id    company_score    date_submitted    company_region
AA            .07              1/1/2017          NW
AB            .08              1/2/2017          NE
CD            .0003            1/18/2017         NW

I've successfully grouped the data by company_id, and calculated the simple moving average of company_score using 10 days / inputs.

score_by_company = df['company_score'].groupby(df['company_id']).rolling(10).mean()

company_id    index           SMA
BBG000B9WMF7  7613            NaN
              1911663         NaN
              1911664         NaN
              1911665         NaN
              1911666         NaN
              1911667         NaN
              1911668         NaN
              1911669         NaN
              1911670         NaN
              1911671    0.000002
              1911672    0.000002
              etc.       etc.

How can I translate this data into a time series graph, with each company_id being its own line?

I feel like I need to create a data dictionary off of the grouped data, but I'm not sure if that's the right approach.

You can make the plots by looping over the groups from groupby...
Or this should also work..
import matplotlib
import matplotlib.pyplot as plt
df['company_score'].groupby(df['company_id']).rolling(10).mean().unstack().plot()

See this for more information on plotting with pandas dataframes and this for looping over a groupby-object. Finally this link is super helpful.


Determining helpful Amazon feedback I am in an ML course and one of our tasks is to predict the helpfulness of Amazon reviews.  Currently, I am doing what most people seem to .  That is, a hashvectorizer(2-gram) on the text, tfidf, several scalar features, and (in my case) a Ridge classifier (parameters chosen by grid search).
  Honestly, it has been frustrating since I have tried to read and apply best practice from the course and any published works I can find.  That said, I am still only achieving marginal results:

{'Pos': 6588, 'Neg': 84412, 'TP': 2755, 'TN': 78990, 'FP': 5422, 'FN': 3833, 'Accuracy': 0.8982967032967033, 'Precision': 0.33692063103827807, 'Recall': 0.41818457802064357}

...which isn't great. I don't need help coding*, but if anyone can suggest useful case studies, or other sklearn classifiers/techniques I could research, I would be grateful.  I'm rapidly running out of directions to follow up on.

Thanks!

if relevant, you can find my latest iteration here

Naive Bayes is almost always my first go to for a problem with text data. This is primarily due to the independent distributions that integrate so well with the document by feature  independent style matrix by nature. I would extend your idea of Ridge with Lasso or ElasticNet. I would not recommend normalize your data if it is sparse, this has been known to cause problems, but StandardScaler is a rather conventional way to get a centered dataset without producing disharmony among the variables in the data. I would consider your error metric of precision or fmeasure under each of the above models as my personal next step, then do some visualizations to argue your best fit.


Changing behaviour of an ML model I am trying to create a ranking system for recommending books to an user. Let's suppose we have some subjects of books like 'A', 'B', 'C', 'D' and from the past behaviour, it is observed that the user is more inclined towards subjects 'A' and 'B'. My dataset has a format of binary classification dataset i.e. it contains specifications of books and a labels specifying whether the user read it or not (1/0). Based on this, I assign probabilities of the user liking a book, using a model like RandomForestClassifier. The problem is, based on the past behaviour, the books with subject 'A' and 'B' always tend to get higher probability scores. Is there any way to add some 'discoverability' to the model so that it assigns high scores to even books of other subjects?

This is a kind of confirmation bias, it's often found in recommender systems and it's hard to beat: since the system is designed to find similar books, it's normal that it returns the most similar books but it can be disappointing for the user. This bias is especially strong if there are few past books to refer to, i.e. the reference sample used by the model is too small to represent the user's preferences.
As Nikos suggested in a comment, one way to diversify the recommendations is to broaden the set of similarity features. A more advanced way is to use some second-degree approach: the system doesn't only recommend books similar to a specific user's past, but finds other users who liked the same books and recommends books that they liked (similar to "other users who like X also like Y").


Why there is only one type of artificial neuron? I find it strange that so many deep learning tricks and improvements have been invented in the past decade but I never heard about someone trying out different models of the artificial neuron other than the one based on perceptron:
y = w*x + b

I haven't been able to find much info on this online which is surprising. I am not a machine learning expert but from the little I know, it would make sense to me to at least experiment with other options. A trivial example, what would happen if there was a layer in a network consisting of neurons where
y = w*x^2 + b

Maybe there is an obvious answer to why my suggestion above isn't a good idea but I would prefer an answer that explains why generally this is not being looked into (at least as far as i know).

The example you cited (using x^2 instead of x) is the idea more popular outside deep learning community, called feature engineering. The trend in neural network modeling is instead to,

Play with weights (w) and fine tune them.
Not change the input vector (x) but feed it to the network directly.
If a single layer neural network is not good enough, add more layers.
Introduce non-linearities using activation functions.
And in general, not hand-roll features (like x^2) but let neural networks discover such features.


Predict task duration I'm trying to create a regression model that predicts the duration of a task. The training data I have consists of roughly 40 thousand completed tasks with these variables:

Who performed the task (~250 different people)
What part (subproject) of the project the task was performed on (~20 different parts)
The type of task
The start date of the task (10 years worth of data)
How long the person who has to do the task estimates it will take
The actual duration this task took to finish

The duration can vary between half an hour to a couple of hundreds of hours, but is heavily right skewed (most tasks are completed within 10 hour). On log scale the distribution is still slightly right skewed.

The prediction doesn't have to be perfect, but I'm trying to improve the people's estimations. One question to ask is "What measure can we use the define beter?" I think the best measure would be the Mean Squared Error (MSE) since it weighs large errors much worse than small errors.

Before I turned to machine learning I tried some simple approaches such as adjusting the estimation by the average or median error, adjusting it by the average/median error grouped by person, grouped by subproject but each of these happened to perform worse.

With machine learning, one of the first problem I encountered was the number of categorical variables since for most models these have to be encoded someway (e.g. one-hot). Anyway, I tried to apply some linear models, for example with Stochastic Gradient Descent my approach would be:

One-hot encode the categorical features
The converted the date to unix timestamps
Normalize all the features that are not already between 0 and 1
Split the data in 80/20 learn and test sets.
With Grid Search cross validation and the learn set try to find the best hyper parameters and fit the model.
Predict with the test set
Calculate the error/score

Now one thing I noticed was that the results varied quite a bit: On one run the MSE was close to double of another run (150 and 280). Another thing is that the MSE of the people's estimate is about 80, so my model performs a bit worse.

During my efforts to improve the performance I stumbled across this question where someone suggests to use survival models. Now I'm unfamilliar with these kinds of models but it sounded promising but during my initial tests with this it turns out to be way too slow for my purposes (too large of a dataset).

In the same Datascience answer that suggested to use the survival models (and the Wikipedia page) they also mentioned Poisson regression, but I'm not sure how I would apply this to my case.

So a long story short: I have just two questions: 1. Was my approach of using SGD 'correct' and do you think I can improve the results with that? 2. Are other models better suited for this kind of prediction and if so, can you explain a bit how I would use them?

I think the analysis which you have done was good.

Regarding the Survival Analysis procedure, I think using it in your scenario is good enough. Even it might take time but the results from that are good and very insightful.

Since you have applied survival analysis on the data, you need to make sure that these assumptions are met:

There are several different ways to estimate a survival function or
a survival curve. There are a number of popular parametric methods
that are used to model survival data, and they differ in terms of
the assumptions that are made about the distribution of survival
times in the population. Some popular distributions include the
exponential, Weibull, Gompertz and log-normal distributions.
Perhaps the most popular is the exponential distribution, which
    assumes that a participant's likelihood of suffering the event of
    interest is independent of how long that person has been event-free.
    Other distributions make different assumptions about the probability
    of an individual developing an event (i.e., it may increase,
    decrease or change over time). More details on parametric methods
    for survival analysis can be found in Hosmer and Lemeshow and Lee
    and Wang1.
Here on two nonparametric methods, which make no assumptions about
how the probability that a person develops the event changes over
time. Using nonparametric methods, we estimate and plot the survival
distribution or the survival curve. Survival curves are often
plotted as step functions, as shown in the figure below. Time is
shown on the X-axis and survival (proportion of people at risk) is
shown on the Y-axis. Note that the percentage of participants
surviving does not always represent the percentage who are alive
(which assumes that the outcome of interest is death). "Survival"
can also refer to the proportion who are free of another outcome
event (e.g., percentage free of MI or cardiovascular disease), or it
can also represent the percentage who do not experience a healthy
outcome (e.g., cancer remission).

You can go through this link for better understanding.

Regarding Poisson Distribution, Did you plot and check whether the data is following Poisson Distribution like:

The Poisson distribution is an appropriate model if the following assumptions are true.

k is the number of times an event occurs in an interval and k can
take values 0, 1, 2, ….
The occurrence of one event does not affect the probability that a
second event will occur. That is, events occur independently.
The rate at which events occur is constant. The rate cannot be higher
in some intervals and lower in other intervals.
Two events cannot occur at exactly the same instant; instead, at each
very small sub-interval exactly one event either occurs or does not
occur.
The probability of an event in a small sub-interval is proportional
to the length of the sub-interval.

Or

The actual probability distribution is given by a binomial
distribution and the number of trials is sufficiently bigger than the
number of successes one is asking about

If these conditions are met then you can use Poisson Model, go through this link

Implementation of this in R, Python.

Finally, to address your 2 questions:

Your approach is Correct there is no problem with using that method, to improve your results you need to work on feature engg(deriving new variables).Since it you are considering the duration as a continuous variable(did you perform that log transformation, which you have stated in the beginning?)
In your scenario I think Survival and Poisson gives you better result, if you think these take more time then try getting sample of data and get your work done. If you are considering the out-come as a continuous variable then you can use Random Forest,XGBoost , all methods which are used for predicting a continuous variable(but if were you I would spend more time in fitting Survival and Poisson and then shift to other prediction techniques)

Let me know if you have any issues!


Calculate volume from 3d_plot I read this interesting kernel on kaggle. I learned that you can make a 3D plot starting from CT sections. Is there a way to calculate the volume of the lungs or other structures you plot using 3d_plot? I couldn't find any reference so I need help.

Interesting question! I'm assuming you have a bunch of points $(X, Y, Z)$ plotting the lung you want to measure.
So your question is actually the measure of a convex hull.

Checkout scipy.spatial.ConvexHull. The ConvexHull object has a volumn attribute which indicates the column of the convex hull you want. It supports N dimensions, which your question is about three dimensions.


ngram and RNN prediction rate wrt word index I tried to plot the rate of correct predictions (for the top 1 shortlist) with relation to the word's position in sentence :

I was expecting to see a plateau sooner on the ngram setup since it needless context. However, one thing I wasn't expecting was that the prediction rate drops.
In my understanding since we already have a context of 3 words, the plateau should converge asymptotically to its highest value.
But both the recurrent network and the Ngram models are experiencing this drop.
I have no idea why it would be.

(Note RNNLM is the name of the framework used to build the recurrent neural net, it uses 500 neurons and 100M direct connections, RNN25 is the same setup but with a training base divided by for)

Here is the sentence size distribution :

Thanks in advance.

Recurrent Neural Network (RNN) create a single state vector over time. Thus that curve is to be expected. Initially, the state vector does have enough information to make a quality prediction. Then quickly reaches asymptotic performances. Overall, the predictions are between 15% and 22% correct.
The shape of the graph might be a function of sentence length in the training corpus. Possibly sentences could be between 3 and 7 words long. The drop could be because there is less training data for longer sentences.


Calculate a ranking function from classification features I am using 3 features (x1, x2, x3) for binary classification. All my feature values are in 0 to 1 range (unit range).

I obtained how important each feature was in classification as follows (i.e. feature importance)

x1 --> 0.1
x2 --> 0.5
x3 --> 0.7

It is clear that feature 3 (x3) contributes the most, x2 the second and x1 the least in classification.

I also performed correlation analysis to check if my features are positively or negatively correlated with the target (y) as follows.

x1 --> positively correlated
x2 --> positively correlated
x3 --> negatively correlated

I am wondering if it is possible to convert my classification features into a ranking function using feature importance and correlation.

For instance, my suggestion looks as follows.

ranking_score = 0.1*x1 + 0.5*x2 + 0.7*(1/x3)

The reason for using (1/x3) in the above equation is because it is negatively correlated with the target (y). Please let me know if my ranking_score equation is statistically correct? If not, please let me know your suggestions.

EDIT: Why ranking is important to me?

My features are related to employee details (x1, x2, x3). At first I used these 3 features to classify efficient and 'inefficient' employees. Now, I want to rank the efficient employees based on these 3 features. The above equation I proposed is to facilitate this task.

I am happy to provide more details if needed.

Generally speaking moving from a classification space to a ranking space is not straight forward. In classification problems there is no meaning to order between labels. This means that your suggested equation might not represent the order between labels at all.

This is somewhat dependent on the feature space and classification algorithm. Some classification algorithms(tree-based for example) don't use the concept of distance in their search for best fit. The separation created from the fit(which is used for classification) does not include how far the boundaries are from one another.

Some classification algorithms (logistic regressions, SVM, others) do have this distance feature, and this can be translated to the probability of an instance belonging to each class. Using this probability as a ranking mechanism (instead of how asking how efficient, asking what is the probability of being efficient) might make sense.


Normalization before or after resizing I'm training deep learning network using images (to be exact - I'm solving semantic segmentation problem).

What's the proper order of resizing (I need to resize images to fixed width X height) and normalization (dividing by 255 value) of images in preprocessing?

Does it make more sense to do first resizing and then normalization? Or first normalization and then resizing?

Resize and then normalize, that's the only pipeline that makes sense.

If you resize after normalization, depending on the resize algorithm, you may end up with values that are outside of the normalized range.


I find these sorts of operations much easier to do if the input data is in a long instead of a wide format, this has the additional advantage of being faster on larger datasets. I first transform the dataset such that each row is a sample for a specific element and the columns hold the concentration and standard deviation for that combination. I then use the same logic with dplyr::mutate() to create the new rsd column.
data %>%
  gather(type, value, -Sample) %>%
  separate(type, c("element", "type"), sep="_") %>%
  spread(type, value) %>%
  mutate(
    rsd = ifelse(
      is.na(conc) | is.na(std) | conc == 0,
      NA,
      ifelse(
        std >= conc,
        100,
        abs(100 * std / conc)
      )
    )
  )

# A tibble: 50 x 5
   Sample element    conc     std    rsd
   <chr>  <chr>     <dbl>   <dbl>  <dbl>
 1 A      Cu      104.    0.784   0.753
 2 A      Fe        0.729 0.00637 0.874
 3 A      K       284.    1.54    0.544
 4 A      Mg       21.3   0.100   0.469
 5 A      Mn      441.    4.59    1.04
 6 B      Cu       32.4   0.774   2.39
 7 B      Fe        0.783 0.0148  1.89
 8 B      K       452.    0.474   0.105
 9 B      Mg       43.2   0.00714 0.0165
10 B      Mn      727.    0.232   0.0319
# … with 40 more rows


Comparing text corpora sentence-wise in Python I have two corpora (in the form of list of sentences); one is the "original" one, the other is a version of it with some minor changes (with contractions expaned, for instance):

s1 = ['He's a real stupid', 'There's no need to worry', 'All right, my friends']
s2 = ['He is a real stupid', 'There is no need to worry', 'All right, my friends']

I'd like to get some rendering of the corresponding sentence pairs where IS some difference between the sentences. Is there any elegant solution to this?

There are several ways to do this, but, assuming the lists are the same length and the sentences you wish to compare are in the same indices int their respective lists, you can handle this with a simple list comprehension:

diffs = [(s1[i], s2[i]) for i in range(len(s1)) if s1[i] != s2[i]]


how to get prediction from trained random forest model? i have a dataset with two columns user posts (posts) and personality type (type) , i need personality type according to posts using this dataset so i used random forest regression for prediction
here is my code:-

df = pd.read_csv('personality_types.csv')

count_vectorizer = CountVectorizer(decode_error='ignore')
X = count_vectorizer.fit_transform(df['posts'])
y = df['type'].values

Xtrain, Xtest, Ytrain, Ytest = train_test_split(X, y, test_size=0.33)

random_forest = RandomForestClassifier(n_estimators=100)
random_forest.fit(Xtrain, Ytrain)
Y_prediction = random_forest.predict(Xtest)

accuracy:

random_forest.score(Xtrain, Ytrain)
acc_random_forest = round(random_forest.score(Xtrain, Ytrain) * 100, 2)
print(round(acc_random_forest,2,), "%")

100%

now i want to get prediction from a custom text how can i achive that ?
how can i get personality type of a post separately using this model.

Use the CountVectorizer you have fitted to preprocess your custom input then feed it to your model for prediction.

custom_input = ['insert text here']
custom_input = count_vectorizer.transform(custom_input)
custom_prediction = random_forest.predict(custom_input)


Gini Index in Regression Decision Tree I want to implement my own version of the CART Decision Tree from scrach (to learn how it works) but I have some trouble with the Gini Index, used to express the purity of a dataset.

More precisely, I don't understand how Gini Index is supposed to work in the case of a regression tree.

The few descriptions I could find describe it as :

gini_index = 1 - sum_for_each_class(probability_of_the_class²)

Where probability_of_the_class is just the number of element from a class divided by the total number of elements.

But I can't use this definition in the case of regression where I have continuous variables.

Is there something I misunderstood here ?

In regression trees, sum of squared error (SSE) is the criterion for tree split. The first split is based on the feature/predictor and its values in your training set that yields the lowest SSE value. And then so on for the further splits.


Am I using GridSearch correctly or do I need to use all data for cross validation? I'm working with a dataset that has 400 observations, 34 features and quite a few outliers, some of them extreme. Given the nature of my data, these need to be in the model.

I started by doing a 75-25 split on my data and leaving those 25% aside.

With the train set, I used GridSearchCV with a RepeatedKFold of 10 folds and 7 repeats and this returned my best_estimator results, which when we go in .cv_results_ we see it's the mean_test_score metric. I then called this my "Cross Validation score". Then, with this model fit, I ran it on the test set as grid.score(X_test, y_test) and called this my Test score.

def rf(df, score):

    X_train, X_test, y_train, y_test = train_test(df)

    params = {'n_estimators': [400, 700, 1000],
              'max_features': ['sqrt', 'auto'],
              'min_samples_split': [2, 3],
              'min_samples_leaf': [1, 2, 3],
              'max_depth': [50, 100, None],
              'bootstrap': [True, False]
}

    scorers = {'RMSE': make_scorer(rmse, greater_is_better=False),
               'MAE': make_scorer(mean_absolute_error, greater_is_better=False),
               'R2': make_scorer(r2_score)}

    cv = RepeatedKFold(n_splits=10, n_repeats=7)

    grid = GridSearchCV(estimator=RandomForestRegressor(random_state=random.seed(42)),
                              param_grid=params,
                              verbose=1,
                              cv=cv,
                              n_jobs =-1,
                              scoring=scorers,
                              refit = score)

    grid = grid.fit(X_train, y_train)

    print('Parameters used:', grid.best_params_)

    if score  == 'RMSE':
        print('RMSE score on train:', round(-1*grid.best_score_,4))
        print('RMSE score on test: ', round(-1*grid.score(X_test, y_test),4))

    elif score == 'R2':
        print('R Squared score on train:', round(grid.best_score_,4))
        print('R Squared score on test: ', round(grid.score(X_test, y_test),4))

    elif score == 'MAE':
        print('MAE score on train:', round(-1*grid.best_score_,4))
        print('MAE score on test: ', round(-1*grid.score(X_test, y_test),4))

When I set my metric to RMSE (the most important one), this is what it outputs:

RMSE score on train: 8.489
RMSE score on test:  5.7952

Have I done this correctly? Can I consider this discrepancy acceptable? With Random Forest for example, if I  deliberately ignore the gridsearch parameters and set my min_leaf_node to something like 10, my RMSE goes all the way up to 12 but it becomes very similar between the CV score and my test data. I'm experiencing similar results with SVR and MLP algorithms.

This is part of my thesis and now I have my supervisor telling me I should be using all my data for cross-validation which I don't think is correct.

My conclusion is that given the outliers in the model, without more observations, a discrepancy in results is to be expected, however I don't know if this conclusion is right or if I'm doing something wrong here.

Running my model in a somewhat similar dataset with fewer outliers gives results closer to one another.

RMSE score on train: 5.9731
RMSE score on test:  6.9164

Your procedure is, from what I can tell, correct. You are correctly splitting your data into train/test, and then using your training data only to find optimal hyper-parameters. Using all of the training data and the hyper parameters found in cross validation, you are then evaluating your final model on the test set.

Indeed, the outliers and the size of your dataset are the most probable causes for the large differences between validation and test. Basically, if the majority of these outlier observations fall in the test set after your initial splitting of your data it is highly likely that your test set scores will be larger than your validation scores. On the other hand, if these outliers are in your training set, you would expect the opposite to be true. This is because, regardless of what data your have in your training set for any random partitioning, these outlier observations are unlikely to be predicted well regardless of what you do, and because the size of your dataset is so small, these outliers will heavily impact your error estimate (especially with RMSE that is incredibly sensitive to outliers due to the squaring).

Either way, report the test set scores only as your final estimates if you want to be honest.

In scenarios in which the variance is large in my test set scores (usually due to a small data set and a large number of outliers), I would highly recommend repeating your entire model building procedure as you described but with a different seed each time (i.e. split your entire dataset into train/test with a different seed each time). Do this until you run out of patience or the variance in your estimate of model performance is small enough. Maybe form bootstrapped confidence intervals from these repetitions to give you information on how much your model's performance varies.


28x28 mnist image unrolling to concatenate using numpy.vstack. Why is numpy.vstack so slow? I'm trying to do a simple reshape of a 60000,28,28 list of mnist digits into a 60000,784 numpy array where the digits have been unrolled.
To do this the code is this:
(xdata,xlabel),(ydata,ylabel)=tf.keras.datasets.mnist.load_data()
newxdata=np.array([])
cnt=0
for i in xdata:
 tmpx=i.ravel()
 if cnt == 0:
  newxdata=np.concatenate((newxdata,tmpx))
 else:
  newxdata=np.vstack((newxdata,tmpx))
 cnt=cnt+1

Why does this take so long to run? Is there a way to speed it up? Ultimately the data will be fed into a keras model in smaller batches. Would writing a generator that does the loop unrolling when a batchsize is asked for be more performant or would it not make a difference?

You can reshape a numpy array simply by:
newxdata =  xdata.reshape((60000,28*28))

for example. Or simply:
newxdata =  xdata.reshape((len(xdata),-1))

Note that reshape is a numpy function which can used also as:
import numpy as np
newxdata =  np.reshape(xdata, (60000,-1))

To speed up your loop you could alternatively use libraries like multiprocessing.Pool or CuPy or Numba.


Compare the MAE Vs R^2 output for Machine learning 10-fold CV I am comparing two 10-fold cv machine learning outputs. I got:
For model 1:
MAE= 0.252
R^2= 0.447
RMSE= 0.37

For Model 2:
MAE= 0.434
R^2= 0.768
RMSE= 1.164

My question is: Is it possible to get higher $R^2$ for higher MAE? What does this mean?

I am not sure about the scenario when the data is following the assumption of linearity.
But let's say the data has non-linearity, then this situation is possible considering the fact that squares of bigger values can easily cover-up the absolute sum of too many smaller values.
Let's see this plot below -
If we move from purple to the red line, MAE will shoot-up since we have too many points with increased MAE but SSE might come down because larger errors have been corrected. SSE coming down will improve the R2Score.
$\hspace{3cm}$
 This was a rough calculation to get the intuition. You may extrapolate it.


How to pass input to deep learning models for Multiple choice question answering problem? I'm currently working on a multiple-choice question answering system. The training set consists of a question, answer and 4 options and I need to predict the correct answer among 4 options. Sometimes there is one paragraph too,  For example :
1.Which among the following is measured using a Vernier Caliper?

[A] Dimensions
[B] Time
[C] Sound
[D] Temperature

Answer : A [Dimensions]

Chapter text: [Book chapter related to Dimension, time, sound and temperature ]

How to feed this input to any of deep learning models?
I thought two approaches :

Using  tokens

and correct and as one hot encoding => [1, 0, 0, 0 ]

Using concatenation

Generating fix sized word embedding for each text :
 - Chapter text = [1,1024]
 - Text         = [1,1024]
 - option_a     = [1,1024]
 - option_b     = [1,1024]
 - option_c     = [1,1024]
 - option_d     = [1,1024]

final_input = concat( [ Chapter text, Text, option_a, option_b, option_c, option_d] ) ==> [1,6144]
and correct and as one hot encoding => [1, 0, 0, 0 ]

Is it good representation for understanding and reasoning over text for mcqa task?

The basic idea of most of the current question answering architectures is:

get a common representation of the question and the input text (e.g., using BERT)
get a representation of the answers
do sort of attention over the answers: compute a scalar score for each of the answers (using a dot-product, linear layer, multilayer-perceptron) and normalize the scores using softmax

The architectures are typically based on the Bidirectional Attention Flow Model, although it was designed for a slightly different task and although the pre-trained word embeddings and RNNs in the model are today usually replaced with BERT-like models. In 2018, there was a competition in question answering at SemEval where many interesting ideas on this problem were presented.


watch is linux command and not related to nvidia-smi command,which using that the nvidia-smi command will be updated every 1 second and it give you a view to see how gpu parameters changing like: RAM usage, temperature, P0(means performance,which p0 is best performance and performance decrease when you get p1,p2,...)

about "can it be used for improving performance of the gpu?" the performance of gpu is showing by p0 so your gpu currently is in max performance.

note that minimum interval for watch command is 0.1 second so if you use watch -n 0.1 nvidia-smi then every 0.1 second the output will be update.

for further information type man watch or man nvidia-smi in terminal.


Intuition behind the RNN/LSTM hidden state? What's the intuition behind the hidden states of RNN/LSTM? Are they similar to the hidden states of HMM (Hidden Markov Model)?

I personally don't think they are comparable to the hidden state of a Markov model. One key difference is that, in a HMM you can explain what a given state means to someone, where in a RNN/LSTM you cannot interpret a given state.

The closest thing that you can compare the hidden state of an RNN/LSTM is to think of it as the output of an intermediate layer of a fully-connected neural network but for time-series data.

And the larger the hidden state the more memory it can retain of the past.


Tagging Unix/Non-Unix logs using NLP I have a set of unstructured data consisting command output logs for different operating systems like Unix, Windows, etc.

For example:

  Releasing version 0.0.1 for Stackoverflow, on 01/01/2019. The coverage is 99% and build is passed.

(This is just an example, and not related to actual use case). This output is different for different operating systems. I want to perform tagging on this data

For the case above, the output should be:

  Releasing version VERSION_NUMBER for PRODUCT_NAME, on releaseDate. The TEST_TYPE is TEST_VALUE and TEST_TYPE is TEST_VALUE

Here, some words are replaced with their corresponding sample tags.

I have studied techniques like POS tagging, NER, LSTM, but I don't know which one is suitable for this particular problem. How can I gather data from raw output and how to apply those techniques here.

Thanks to everyone who is willing to help me with this.

PoS tagging works for natural language only and identifies grammatical parts of the sentence, nothing more. LSTM is an algorithm that can be used to predict series. Named Entity Recognition (NER) and Terminology Extraction could work if you have already data to engage in Information Extraction (IE). However, In order to use a these techniques you need to have a trained model and in order to train one you need data. In your case that would involve identifying and tagging parts of the sentence by hand and later training a model with that data.
The best approach, in my opinion, is just use regex to identify parts of the sentence as one of the approaches of Information Extraction and use hard coded rules to best identify what you are trying to replace later.
For instance, if you want to search the version in the example:

import re
s = 'Releasing version. 0.0.1 for Stackoverflow, on 01/01/2019. The coverage is 99% and build is passed.'
re.search(r'Releasing version(.*?)for', s).group(1)
' 0.0.1 '

Check out these resources that will help you parse a log in Python using Regex.
https://pythonicways.wordpress.com/2016/12/20/log-file-parsing-in-python/
https://medium.com/devops-challenge/apache-log-parser-using-python-8080fbc41dda
But if you prefer to use Named Entity Recognition or Terminology Extraction techniques you could hack a NER model and train it yourself with your data. Keep in mind though, that according to Poibeau, Thierry; Kosseim, Leila (2001). "Proper Name Extraction from Non-Journalistic Texts". Language and Computers. 37 (1): 144–157.:

Research indicates that even state-of-the-art NER systems are brittle, meaning that NER systems developed for one domain do not typically perform well on other domains.

And according to Wikipedia: Considerable effort is involved in tuning NER systems to perform well in a new domain; this is true for both rule-based and trainable statistical systems.
So even if you pull that off, accuracy will be less than if you just extracted data using regex. An approximate of a reduction of 97% to 93% only for named entities (companies, names, etc.) Accuracy reduction will be much less in you case.
Check this link for more information about Information Extraction: https://web.stanford.edu/~jurafsky/slp3/17.pdf


"Stationarity of statistics" and "locality of pixel dependencies" I'm reading the ImageNet Classification with Deep Convolutional Neural Networks paper by Krizhevsky et al, and came across these lines in the Intro paragraph:

  Their (convolutional neural networks') capacity can be controlled by varying their depth and breadth, and they also make strong and mostly correct assumptions about the nature of images (namely, stationarity of statistics and locality of pixel dependencies). Thus, compared to standard feedforward neural networks with similarly-sized layers, CNNs have much fewer connections and parameters and so they are easier to train, while their theoretically-best performance is likely to be only slightly worse.

What's meant by "stationarity of statistics" and "locality of pixel dependencies"? Also, what's the basis of saying that CNN's theoretically best performance is only slightly worse than that of feedforward NN?

At the time of writing the article, CNNs were not yet a particularly popular architecture for neural networks (and neural networks in themselves weren't as popular and common as today). In fact this article can be seen as the one who started the current age of deep learning as a default machine learning approach.

All this intro was for the soul purpose of saying that by "standard feed forward neural networks", the author meant neural networks that consist only out of dense layers (fully connected).

So to answer your last question first, in theory, anything a convolutional layer can do, a fully connected layer with the same number of input parameters and output parameters can also accomplish. i.e. you can produce a convolutional layer from a fully connected one (it will just have the same weights repeat themselves in a pattern of the same size as the convolution kernel), and you can also produce a lot more operations that a simple convolutional layer cannot perform. The simplest example can be a location dependent convolutional layer (where different areas of the image are convolved with different kernels). In reality, the search space for optimal parameters of such a layer would be so huge (and so non-convex), that it won't be able to converge.

The limitations on the convolutional layer are actually what helps her shine, which brings us to your first question. The "locality of pixel dependencies" is exactly what allows such a limited operation, to achieve such fine results. The meaning of the sentence is very simple, close pixels are very likely to be dependent on each other, so we can and should leverage this dependency to process them together (as is done with convolutional kernels).

Regarding the "stationary of statistics" phrase, I'm not 100% sure, but I think it refers to the fact that at small local patches, there is a higher probability of finding recurring patterns (the smaller the patch size, the smaller the possible variance of the patch). However as I said, I'm not sure about this one.


binary decision tree hyperplane parallel to axis? I'm reading the random forest paper.

  binary decision trees use a single feature at each nonterminal node. A test point is assigned to the left or right branch by its value of that feature. Geometrically this corresponds to assigning the point to one side of the hyperplane that is parallel to one axis of the feature space.

Can someone explain why the hyperplane is parallel to the axis?

Say the feature is represented by the $x$ axis, and that the decision is $x < 5$ vs $x >5$. To me it seems like the hyperplane should be vertical to the $x$ axis so that $x = 4$ is on one side and $x=6$ is on the other. What's wrong with my understanding? Thanks.

Your understanding is fine.

The hyperplane is $x=5$.

In $3$ dimension, it is a plane that is parallel to the the $y$-$z$ plane, which can be written as $x=0$. It is parallel to the $y$-axis and $z$-axis.

You are right that it is perpendicular to the $x$-axis. The $x$-axis is the normal direction for the plane $x=5$.


Predicting probability of order arriving on or before customer Requested Delivery Date(RDD) I have a historical data of orders from different customers of my company and delivery details. I wanted to develop a model which predicts the probability of customers order getting delivered before or on the "Requested Delivery Date".

In my data-set, i've details about
customer
delivery location
product
delivery type
product classification (Can be exported or not) etc.

I was not able to start this model as I am not sure how to tackle this problem. It would be of great help, if you could suggest some reading material on the same or few link directing the same.

The most important thing here is to model your problem as time series, as you know, each example consists of one customer that has many orders over time. see Predicting Clinical Events via Recurrent Neural Networks


help with Keras sequential model output I have trained a sequential model with keras for MNIST dataset and this is the code I've used.

# Create the model
model = Sequential()

# Add the first hidden layer
model.add(Dense(50, activation='relu', input_shape = (X.shape[1],)))

# Add the second hidden layer
model.add(Dense(50, activation='relu'))

# Add the output layer
model.add(Dense(10, activation = 'softmax'))

# Compile the model
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics = ['accuracy'])

# Fit the model
model.fit(X, y, validation_split=.3)

Output:

Train on 1750 samples, validate on 750 samples
1750/1750 [==============================] - 0s - loss: 0.1002 - acc: 0.9811 - val_loss: 0.3777 - val_acc: 0.8800

Can you explain what is loss, acc , val_loss, val_acc? How can I know my model performance from these metrics in output. Please explain if possible.

First will explain the terms loss, acc, val_loss, val_acc and then get into evaluating model performance.

Loss - This metric is used to understand the prediction by the network on training data vs the actual value. Given that you are using categorical crossentropy as your loss mechanism, your loss is actually telling you how near or far your predicitions are from the true classification values.

Acc - Is the accuracy of the network on the training data. After training on 1750 samples, your network can accurately predict with 0.98 accuracy.

Val_loss, Val_acc - The metrics are the same but are evaluated on validation data, that is the data that is not part of the training dataset, this will help you evaluate how your network performs on data that it is not trained on, sort of mimicking how your network would work when deployed in the real world on the chosen dataset.

The above metrics help you understand how your network is performing every epoch, giving you an idea on how to improve the performance.

How to perform model performance:

Model performance requires you to create/manage training dataset that would depict what you can expect in the real world when deploying your model to actually predict.

You would look at test accuracy, precision, recall, f1 scores, confusion matrix and other metrics based on the problem you are solving to truly evaluate your model performance.

Hope this answers your question. Have fun with deep learning networks :)


Why the first prediction of neural network in PyTorch is slower than following predictions? So I have ResNet50 trained to classify images.

For each prediction I track the time needed for it (input and model are moved to GPU):

    start = time.time()
    result = self.model.forward(transformed_image)
    end = time.time()
    print(end - start)

And always I get the following output:

1.0592937469482422
0.05996203422546387
0.06096029281616211
0.04996800422668457

So the first prediction is ~20 times longer than the following ones.

Why? And what happens behind the scenes when we launch prediction for the first time, using Torch?

I have seen a similar question several times before. See https://stackoverflow.com/a/55577921/9212382 for a possible explanation.


Theoretical background for model arhitecture choosing Suppose I have a dataset $X$ and two different binary labels $y_{1}$ and $y_{2}$. The classes are very imbalanced - 3% of true in $y_{1}$  and 2% in $y_{2}$. Moreover, there are no pairs of (0,1), so, if the  $y_{1} = 0$, then  $y_{2}=0$.

I have to choose between:

Associating the labels with each other, so it's a 3-class classification problem
Build a model for  $y_{1}$ scoring, then joining to $X$ and making scoring of  $y_{2}$, using $X$ and  $y_{1}$.

So the first idea is formalised as :

$$ y_{f} = y_{1} \bullet y_{2}; \,\,\,\, y_{f} = \{(1,1),(0,0),(1,0) \}; \,\,\, y_{f} = \Phi_{f}(X)$$

And the second road my be formalised like this:
$$ y_{1} = \Phi_{1}(X); \,\,\, \, \, X=X\cup y_{1};  \,\,\,\, y_{2}=\Phi_{2}(X) $$
Taking into account all condition, what is the possible theoretical background of projecting the best algorithm?  I'am trying to give estimates to an error, which is :bias + variance + noise. E.g.:
For both: $\Phi_{f}$ and $\Phi_{2}$, which are RandomForests, bias and noise will be the same, variance will be higher for  $\Phi_{f}$(just intuition till now). Can somebody point me the right way?

The theoretical estimation of the error depends on the data, and the fact that the labels can be predicted by the variable X. There's no way to a priori know which will be the best option regarding the two models, so just try.

However, I would bet that the second option will work better if y2 is correlated in some way to y1, and I would bet for the first option if they are independent.

However, when you train your model, I highly recommend sampling the data in a more balanced way, so that the model not learns to predict (0, 0) always regardless of the values of X (as a model that does these dummy predictions would have high accuracy). Another way to work around the imbalanced classes is to use AUC in ROC as a metric to choose among different models.


error length > 1 and only the first element will be used? I'm new i data mining and R, and i'm working with a dataset and trying to remove the "white space" elements. i tried to create a function to do that.

elim.fal=function(v){
contador=0
for (i in length(v)){
if (as.factor(v[i])== "" && contador<=length(v)){
  a=v[-i]
}
else {
  a=v

}
contador= contador+1
return(a)
}

but appear that error

I can't reproduce that error message, so I'm not sure what's going wrong.  That said, the typical way to filter a vector in R is to subset it with a conditional like so:

tst<-c("","foo", "bar", "", "baz")
tst[tst!=""]
[1] "foo" "bar" "baz"


Linearly increasing data with manual reset I have a linearly increasing time series dataset of a sensor, with value ranges between 50 and 150. I've implemented a Simple Linear Regression algorithm to fit a regression line on such data, and I'm predicting the date when the series would reach 120.

All works fine when the series move upwards. But, there are cases in which the sensor reaches around 110 or 115, and it is reset; in such cases the values would start over again at, say, 50 or 60.

This is where I start facing issues with the regression line, as it starts moving downwards, and it starts predicting old date. I think I should be considering only the subset of data from where it was previously reset. However, I'm trying to understand if there are any algorithms available that consider this case.

I'm new to data science, would appreciate any pointers to move further.

Edit: nfmcclure's suggestions applied

Before applying the suggestions

Below is the snapshot of what I've got after splitting the dataset where the reset occurs, and the slope of two set.

finding the mean of the two slopes and drawing the line from the mean.

Is this OK?

I thought this was an interesting problem, so I wrote a sample data set and a linear slope estimator in R.  I hope it helps you with your problem.  I'm going to make some assumptions, the biggest is that you want to estimate a constant slope, given by some segments in your data.  Another assumption to separate the blocks of linear data is that the natural 'reset' will be found by comparing consecutive differences and finding ones that are X-standard deviations below the mean. (I chose 4 sd's, but this can be changed)

Here is a plot of the data, and the code to generating it is at the bottom.

For starters, we find the breaks and fit each set of y-values and record the slopes.

# Find the differences between adjacent points
diffs = y_data[-1] - y_data[-length(y_data)]
# Find the break points (here I use 4 s.d.'s)
break_points = c(0,which(diffs < (mean(diffs) - 4*sd(diffs))),length(y_data))
# Create the lists of y-values
y_lists = sapply(1:(length(break_points)-1),function(x){
  y_data[(break_points[x]+1):(break_points[x+1])]
})
# Create the lists of x-values
x_lists = lapply(y_lists,function(x) 1:length(x))
#Find all the slopes for the lists of points
slopes = unlist(lapply(1:length(y_lists), function(x) lm(y_lists[[x]] ~ x_lists[[x]])$coefficients[2]))

Here are the slopes:
(3.309110, 4.419178, 3.292029, 4.531126, 3.675178, 4.294389)

And we can just take the mean to find the expected slope (3.920168).

Edit: Predicting when series reaches 120

I realized I didn't finish predicted when series reaches 120.  If we estimate the slope to be m and we see a reset at time t to a value x (x<120), we can predict how much longer it would take to reach 120 by some simple algebra.

Here, t is the time it would take to reach 120 after a reset, x is what it resets to, and m is the estimated slope.  I'm not going to even touch the subject of units here, but it's good practice to work them out and make sure everything makes sense.

Edit: Creating The Sample Data

The sample data will consist of 100 points, random noise with a slope of 4 (Hopefully we will estimate this).  When the y-values reach a cutoff, they reset to 50.  The cutoff is randomly chosen between 115 and 120 for each reset.  Here is the R code to create the data set.

# Create Sample Data
set.seed(1001)
x_data = 1:100 # x-data
y_data = rep(0,length(x_data)) # Initialize y-data
y_data[1] = 50
reset_level = sample(115:120,1) # Select initial cutoff
for (i in x_data[-1]){ # Loop through rest of x-data
  if(y_data[i-1]>reset_level){ # check if y-value is above cutoff
    y_data[i] = 50             # Reset if it is and
    reset_level = sample(115:120,1) # rechoose cutoff
  }else {
    y_data[i] = y_data[i-1] + 4 + (10*runif(1)-5) # Or just increment y with random noise
  }
}
plot(x_data,y_data) # Plot data


Negative examples and false positives in object detection using SSD model I am training a model to detect some objects. I have 8 classes and there are other similar objects that I don't want the model to detect. I have tried to make an unknown class for them but they are so many and there are no pattern for the model to recognize.
How can I provide a negative examples for the model? Since that tfrecord can't accept images without annotation and making a new class doesn't work very well.

I have found a way to create TFRecord that accepts images with no annotations using this code.


suggest a reinforcement learning agent that will learn to efficiently switch on a bit vector The following env describes a simple task: there's a bit-vector of size 10, with all bits initially set to 0.

The agent must learn to switch it all to 1s. It has 10 possible actions - each one switches the corresponding bit to 1. Switching a bit which was 0 to 1 gives a reward of 1, but trying to switch a bit which is already 1 gives a reward of -1 and ends the game. The agent sees the entire bit-vector after each step.

Why isn't it very very simple? I wrote a policy-gradient approach agent but it doesn't arrive to sufficient results (stay at around ~3 points average) after 10K episodes.

What am I missing? Is it indeed such a hard task???

Why isn't it very very simple?

The problem description is simple, but not in a way that is accessible to the standard RL algorithms. First of all, the analogy of switches and "corresponding" has no meaning to the RL. Secondly, the low complexity is not detectable and is outside of the assumptions of RL agents in general.

Reinforcement Learning is based on an internal model of the problem as a generic Markov Decision Process (MDP). The RL agent has to discover, by trial and error, what the MDP does, typically starting with no knowledge or assumptions other than the set of states and allowed actions. The agent needs to discover enough about the MDP to make optimal actions, whilst the only assumption is that the environment is a MDP.

The simple switch problem can definitely be modelled as a MDP. The agent then has to discover the simple configuration in the toy problem. What makes it harder than the problem description:

State/action space. There are 1024 states and 10 actions. There is no a priori reason to link any action with any outcome, so to be thorough the agent has to try them all. A minimal search would be across 10,240 state/action pairs.
State transitions. There are 1024 states where an action can be taken, and a terminal state. So there are 1024 * 1025 = 1,049,600 possible state transitions for every action. In your case, each state/action pair has one outcome, and there are roughly 5,000 different transitions (because half of them are to the terminal state).
The agent does not know that the state transitions are deterministic and sparse. It can only discover that by trial and error. For instance, perhaps in a particular configuration, taking one particular action would cause some bits to re-set. Maybe that only happens one time in four attempts. Once you add the possibility of arbitrary probability distributions here, then the number of potential MDPs is infinite. RL algorithms are designed to cope with this, still learning a good approximation of an optimal policy eventually. There is no simple way to make them dumber and faster against a simpler environment without encoding some of your knowledge about that environment.
The agent does not know that the rewards are deterministic and simple. Again it can only discover this by trial and error. To the agent the different state vectors and actions are neither similar to each other (all switches and bits) nor have any association (action 0 is not associated in any way with bit 0). Even after the associations are observed, they are not encoded in a way that the agent can reason by analogy. Again, once you add the possibility of arbitrary probability distributions for reward, then the number of potential MDPs is infinite.

  Suggest a reinforcement learning agent that will learn to efficiently switch on a bit vector

You have made things a lot harder for yourself by trying a policy gradient method with a neural network function approximator. Although this probably can be made to solve your problem, it is an advanced technique, and also known to be unstable and hard to train. It might be a good choice when the relationship between the state, transitions and rewards was very complex, and there were more states possible with meaningful similarity between them (in a numerical/statistical sense, not in terms of analogy, the neural network still won't find the analogy with switches and numbered bits). Even then, it will learn slowly and cautiously, needing to repeat state/action pairs multiple times in order to learn that the transitions and reward are reliable.

Unless there is some good reason for you to stick with a policy gradient method, I suggest using a tabular algorithm (i.e. no function approximation, just a table of action value estimates) and something like single step Q-Learning. That has the advantage that because you know the algorithm is deterministic, you can set a high learning rate and it will remain stable. In fact Q-learning will probably learn an optimal policy in much less than 10,000 episodes. However, initially it will learn just one of the many possible optimal policies and tend to stick with it  - i.e. some permutation of taking each action 0 to 9, once each, but always in the same order because it has learned that is safe.


Data Cleaning Techniques - What to drop? I have been struggling with this problem for a few weeks but it seems to be out of my league. I have a dataset of product sales over the course of 55 weeks. It contains information on the store ID, and the product ID. However, some of the data is missing. Some store IDs do not have continuous data for the 55 weeks, and some item IDs do not have continuous data. Also, combinations of store IDs and item IDs have missing data points. What is the most efficient way to decide what to drop and what to keep for the best results in my model? If the product I am forecasting is continuous that is a good start, but I am not sure what the best combination of stores and items to drop is.

Check the percantege of missing values in each column. If it is high then you should remove this column, otherwise you are able to try to impute the missing values. For the latter you may replace them with the mean (median or mode etc) or try to predict them using the other known values. You should check if these are values are mising at random or not.


Customized function for Agglomerative Clustering I need to use a customized linkage function for my project. For this function, on every step I should calculate the objective containing

similarity (distance) between subclusters to be merged
distances between leaves and roots in terms of edges in the forest of subclusters.

Currently I'm using scikit-learn AgglomerativeClustering. I looked through a question related to mine.

I found that the merging operation for max link and average link happens in the script.

However, I cannot find where the the distances between all the subclusters are calculated and how to derive a number of edges between leaves and roots in subclusters.

How can this be done?

UPD: I am trying to implement Chakrabarti algorithms for smooth clustering over time. For agglomerative clustering, a choice for next clusters to be merged depends on two components: similarity between subclusters at current time stamp and a historical cost related to the previous time stamp. The latter has several variations. For example, one includes merge distance (not only that): if we would merge two subclusters S1 and S2, what would be the average distance (in terms of edges) between all leaves in S1 and S2.

There are a couple of questions about sklearn AgglomerativeClustering:

What lines in the code implement a choice of clusters to be merged at the current step?
Is it possible to extract a distance between a leaf and a root of a subcluster at each merging step? I noticed there is an attribute 'children_' in AgglomerativeClustering which I could use to create the resulting tree and find distances between any nodes. How to extract such an information for every merging step?

Most implementations will begin with a distance matrix (and not a data matrix) as input, as the algorithms are O(n²) anyway.

For the second question, I don't understand how that is defined. If your linkage is not a Lance Williams update, you may need to reimplement hierarchical clustering completely, to make sure you have all the information you need. Lance Williams are much more elegant to compute.


How to best summarize and visualize the outcome and performance of a multi-class problems? I estimate a multi-class origin-destination model with 45 classes. In particular the classes are geographical regions between which people can move. Currently I summarize the overall performance using Cohen's kappa to take the unequal class sizes into account. But I would love to provide deeper insights into the performance. Do you have any recommendations of how to proceed with so many classes?

I would do a big confusion matrix and, by inspecting it, decide what classes to add to a smaller confusion matrix in order to clearly illustrate the main sources of confusion for the model.


Columns with no (or nearly no) differences between rows worth keeping? I have a dataset which contains multiple columns. On analysis, I found out that there were few columns which contain just a single value. There was 0 in all the rows of these columns. Does it hold any information or should I remove such columns from the dataset?

Few columns were also like that it contains 100k rows of type1 and 3-4 rows of type2. Should i also remove this columns or let them remain as such?

Note: Target column is different than the columns described.

The features you described are known as low variance features and in general you should remove those features. The rationale behind this, is that low variance feature contain less information. See this for a succinct explanation and some code in python. Be sure to normalize your features. One of the advantages is to speed the training phase.


How to determine the inputshape of a ANN in Keras I started to use Keras for ANN and something that I do not really understand is which values to choose for the input_shape parameter of the first layer in an ANN? I know that the number should be equal to the inputs but how can I determine the order and the other value in the vector. For example, here is the code of a Multilayer Perceptron that takes 3 inputs and calculates 1 output
model.add(keras.layers.Flatten(input_shape=[3,])),
model.add(Dense(20,  activation='relu'))
model.add(Dense(40,  activation='relu'))
model.add(Dense(20,  activation='relu'))
model.add(Dense(3, activation='linear'))

In this case I use 'input_shape=[3,]'. The 3 comes from the 3 inputs but why do I have a ",]" as the second argument. Here you see on the other hand the code for a recurrent neural network that is used for time series forecasting and that has 6 input features for every timeslot of the time series and calculates 24 outputs:
model6 = keras.models.Sequential([
    keras.layers.SimpleRNN(20, return_sequences=True, input_shape=[None, 6]),
    keras.layers.SimpleRNN(20, return_sequences=True),
    keras.layers.TimeDistributed(keras.layers.Dense(24))
])

Why do I have here another order with "input_shape=[None, 6]" and not "input_shape=[6, None]" and why do I need here "None" as the first argument and not like in the multilayer perceptron just ",]". Has this something to do with the recurrent neural network such that I always have to use the number of input features as the second argument and 'None' as the first argument.
I read about this in the keras documentation but it is still confusing for me. Can you tell me how to choose those input_shape argument`I'd appreciate every comment.

There are basically two parts.

Why there is a comma in the [3,]? In this case you can skip it and just use [3]. You can encounter it in the tutorials, because you can pass the tuple as shape as well - (3,) and if you skip the comma in the tuple, then it will be just number, not tuple. So, it's just more of a python, not a keras itself. Try this in terminal

>>> 3 == (3)
True
>>> 3 == (3,)
False
>>> [3] == [3,]
True

The bottom line is that keras in this case expects 1-dimensional objects of size 3.

About the recurrent networks. In contrast with fully-connected neural nets, the input to the RNN is two-dimensional. I.e. you have a time series with 10 steps, each defined by 3 numbers, then the shape would be (10, 3). The problem is that you don't know beforehand how many steps you will have - it could be 20, it could be 5. So, you use None as a placeholder to say the keras to expect some dimension there.

You can ask, why we use None here and not something else. The reason is numpy has been using similar notations for years. See the example below
>>> import numpy as np
>>> x = np.ones(5)
>>> x.shape
(5,)
>>> x_2 = x[None, :5]
>>> x_2.shape
(1, 5)

In this case None in index tells  numpy to add a dimension. It's not very transparent thing and could be confusing at first, especially with all the other notation, but you will get used to it.
EDIT

Worth mentioning, the number of dimensions of input tensors always will be bigger on one, i.e. 2 dimensions for MLP and 3 for the RNN, because we process data in batches. Thus, the first dimension always will be the number of samples.
Let's look on the examples.

For MLP input tensor shape could be [128, 3], where the 128 is the number of samples and 3 is number of features. input_shape=(3,)
For RNN input tensor shape could be [128, 30, 3], where the 128 is the number of sample, 30 is sequence length and 3 is number of features. input_shape=(None, 3)
For convolutional NN the inputs will be images and shape like [128, 220, 220, 3], where the 128 is the number of images, 220x220 - size of the image and 3 is number of channels (colors). input_shape=(220, 220, 3)

The interesting fact - we asked to specify the input shape not because keras authors are pedants, but because the specific size of the network is depend on it and we need this info during initialization. As batch size doesn't influence the model size, it is omitted by convention. We would probably omitted the sequence length as well, if the RNN was the only architecture, but for consistency with other type of models it is as it is.


Loss for CNN decreases and settles but training accuracy does not improve I am training a CNN with 2 conv layers 2 Relu and max pooling and 2 FC layers the last of which has only 2 units since it's a binary classification problem. The images are spatio-temporal continuous, basically those of stream functions of the atmosphere, obtained as contour plots in MATLAB from data generated out of a PDE solver. So, it is quite difficult to understand the difference between the two labels by just looking at the respective images. The labels are generated through some statistical algorithm that operates on the intensity of the pixels or simply the value of the matrices that make the contour plot. While training the CNN, I see that with a learning rate of .001, the loss decreases gradually and monotonically at all time where it goes down to 0.6 in the first 200 epochs (not suddenly, quite gradually, the slope decreasing as the value goes down) and settles there for the next 500 epochs. However, the training accuracy fluctuates around 50% and my testing accuracy is 50% too. Moreover, I see while looking at the false +ve and -ves that the CNN has got one label right all the time and the other label wrong all the time. I have used dropout of 0.5 and L2 regularization. I am using TensorFlow for all the programming. There is no imbalance in the training set between the TRUE and FALSE labels. I have 4500 TRUE example and 4500 False examples. I have randomly shuffled the training set and normalized them with 255 since I'm using RGB images. Testing is done on 200 TRUE examples and 200 False examples. Kernel size 5x5 in both layers , first one having 32 and second one having 64 filters and FC layer has 1024 neurons. I have played around with these parameters with no real improvement. Any suggestions would be really appreciated.

I guess I was able to figure out the answer to this when I looked carefully at the feature space and printed out the outputs from different convolutional layers. It appears that this issue appears when you have a lot of redundant features in your feature space. Since, this was a binary classification problem with FALSE and TRUE, each image by default had a lot of features that would correspond to a FALSE and the TRUEs only had a small percentage of the feature space being different. The trainer easily learnt the large percentage of the FALSE feature space and hence the loss easily went down while the small TRUE feature space was missed. This is the reason why the loss was going down but accuracy not improving. The trick to avoiding this issue was to downsample the initial image in the beginning before going to the convolutional layer. One could do it with some pyramidal pooling if on Tensorflow, or simply with imresize in matlab.