Optimizing CNN network I am currently trying to recreate the result of this paper, in which they do feature extraction from a "spectogram" of log-melfilter energies..

Since the paper doesn't state what kind of feature I am seeking, i am currently trying to extract features, and match them to MFCC features. The paper states a technique called LWS (Limited weight sharing) in which the spectogram frequency axis will be divided into section, and each section don't share their weight with others.

So i've divided the my input image into 13 section to receive 1 output features  from a (6,3,3) input image. 6 for the number of rows, 3 as each column represent [static delta delta_delta] data of the given log melfilter energi, and the last 3 is the color channels.

If i'd used 13 filterbanks, and made the plot, will the result of this be that each (1,3,3) matrix would result in one feature, but that seemed a bit too good to be true, so i decided to use 78 filterbanks and divide it into 13 section which should result in one feature can be extracted from a matrix of size (6,3,3)

I am training the network with this model structure:

def create_model(init_mode='normal',activation_mode='softsign',optimizer_mode="Adamax", activation_mode_conv = 'softsign'):
    model = Sequential()

    model.add(ZeroPadding2D((6,4),input_shape=(6,3,3)))
    model.add(Convolution2D(32,3,3 , activation=activation_mode_conv))
    print model.output_shape
    model.add(Convolution2D(32, 3,3, activation=activation_mode_conv))
    print model.output_shape
    model.add(MaxPooling2D(pool_size=(2,2),strides=(2,1)))
    print model.output_shape
    model.add(Convolution2D(64, 3,3 , activation=activation_mode_conv))
    print model.output_shape
    model.add(Convolution2D(64, 3,3 , activation=activation_mode_conv))
    print model.output_shape
    model.add(MaxPooling2D(pool_size=(2,2),strides=(2,1)))
    model.add(Flatten())
    print model.output_shape
    model.add(Dense(output_dim=32, input_dim=64, init=init_mode,activation=activation_mode))
    model.add(Dense(output_dim=13, input_dim=50, init=init_mode,activation=activation_mode))
    model.add(Dense(output_dim=1, input_dim=13, init=init_mode,activation=activation_mode))
    model.add(Dense(output_dim=1,  init=init_mode, activation=activation_mode))
    #print model.summary()
    model.compile(loss='mean_squared_error',optimizer=optimizer_mode)

    return model

This model keeps for some reason providing me very bad results..
I seem to keep getting an loss of 216, which is nearly 3 times the data range...

I did a grid seach to find out which parameter (activation function, init_mode, epochs and batch_size would be best, which are those chosen in the function above (eventhough there wasn't much change in the outcome..)

What can i do to get better results?
Is the CNN network poorly designed?

There are some suggestions which I think can improve CNN performance:

Input size is (6, 3) so it is not preferable to use MaxPooling layers.
Use padding = 'same' in all the convolution layers which output convolved image of the same dimension as input.
Use Relu or LeakyRelu as activation function.
Use Adam optimizer and tune the learning rate.
Convert some of the Dense layers to 1x1 Convolution with no. of filters = no. of units in dense layer as suggested by sh37211

e.g.

no_of_filters = [32, 64, 64, 32]
kernel_size = [3, 3, 3, 1]

input = Input(shape = (6,3,3), name = "input")
layer_output = [input]
for i in range(4):
    convolution = Convolution2D(no_of_filters[i], kernel_size[i], kernel_size[i], padding = 'same')(layer_output[-1])
    activation = LeakyReLU()(convolution)
    layer_output.append(activation)

flatten = Flatten()(layer_output[-1])
flatten_dropout = Dropout(0.5)(flatten)
fc = Dense(output_dim = 13)
activation = LeakyReLU()(fc)
layer_output.append(activation)
fc = Dense(output_dim = 1, activation = 'tanh')

model.compile(loss = 'mean_squared_error',optimizer = 'adam')


Mathematics: Can the result of a derivative for the Gradient Descent consist of only one value? I have a problem of a task using the formula of the Gradient Descent:

Perform two steps of the gradient descent towards a local minimum for the function given below, using a step size of 0.1 and an initial value of [1, 1]

I only get as result of the derivative 0.4x1, because x2 does not exist.
Is this correct or should the result for the derivative be (0.4, 0)?
Note: Sorry, if my equation of the derivative is bad. I'm not a mathematician. Please, correct me, if my equation is absolutely wrong.

Gradient, g(x) = 0.4*x

At [x1 = 1], Gradient,
g(1) = 0.4
x2 = x1- step*gradient
=>x2 = 1 - 0.1*0.4
=>x2 = 0.96
At [x2 = 0.96], Gradient,
g(0.96) = 0.4*0.96 = 0.384
=> x3 = 0.96 - 0.1*0.384 = 0.9216

Continue following the same steps and will reach near the minima.


What are deconvolutional layers? I recently read Fully Convolutional Networks for Semantic Segmentation by Jonathan Long, Evan Shelhamer, Trevor Darrell. I don't understand what "deconvolutional layers" do / how they work.

The relevant part is

  3.3. Upsampling is backwards strided convolution

  Another way to connect coarse outputs to dense pixels
  is interpolation. For instance, simple bilinear interpolation
  computes each output $y_{ij}$ from the nearest four inputs by a
  linear map that depends only on the relative positions of the
  input and output cells.
  In a sense, upsampling with factor $f$ is convolution with
  a fractional input stride of 1/f. So long as $f$ is integral, a
  natural way to upsample is therefore backwards convolution
  (sometimes called deconvolution) with an output stride of
  $f$. Such an operation is trivial to implement, since it simply
  reverses the forward and backward passes of convolution.
  Thus upsampling is performed in-network for end-to-end
  learning by backpropagation from the pixelwise loss.
  Note that the deconvolution filter in such a layer need not
  be fixed (e.g., to bilinear upsampling), but can be learned.
  A stack of deconvolution layers and activation functions can
  even learn a nonlinear upsampling.
  In our experiments, we find that in-network upsampling
  is fast and effective for learning dense prediction. Our best
  segmentation architecture uses these layers to learn to upsample
  for refined prediction in Section 4.2.

I don't think I really understood how convolutional layers are trained.

What I think I've understood is that convolutional layers with a kernel size $k$ learn filters of size $k \times k$. The output of a convolutional layer with kernel size $k$, stride $s \in \mathbb{N}$ and $n$ filters is of dimension $\frac{\text{Input dim}}{s^2} \cdot n$. However, I don't know how the learning of convolutional layers works. (I understand how simple MLPs learn with gradient descent, if that helps).

So if my understanding of convolutional layers is correct, I have no clue how this can be reversed.

Could anybody please help me to understand deconvolutional layers?

Deconvolution layer is a very unfortunate name and should rather be called a transposed convolutional layer.

Visually, for a transposed convolution with stride one and no padding, we just pad the original input (blue entries) with zeroes (white entries) (Figure 1).

In case of stride two and padding, the transposed convolution would look like this (Figure 2):

All credits for the great visualisations go to

Vincent Dumoulin, Francesco Visin - A guide to convolution arithmetic for deep learning

You can find more visualisations of convolutional arithmetics here.


How can I vectorize this code in R? Maybe with the apply() function? I am really struggling to replicate the output of the dist() function in R code without using 1 or 2 for loops. (If you're wondering why I'm doing this, it's so that I can play around with the distance calculation, and also to improve my R skills - so please only solutions that involve R!)

Overview: matrix is passed to dist(), which calculates the Euclidean distance row-wise and outputs a full distance matrix of the distance between each row (e.g. distance between rows 1 and 50 will be in distancematrix[1, 50] and distancematrix[50, 1]). The fast code looks like this:

distancematrix <- as.matrix(dist(myMatrix, method="euclidean", diag = T, upper = T))

I have successfully produced the same output in R using the following code:

for (i in 1:nrow(myMatrix)) {
  for (j in 1:nrow(myMatrix)) {
    distancematrix[i, j] <- sum(abs(myMatrix[i,] - myMatrix[j,]))
  }
}

However, using two nested for loops is much slower than using dist(). I have read a lot about using apply() to optimise slower for loops, but I haven't been able to get my head around it so far. I believe that at least one of the for loops is definitely avoidable by just outputting a vector, and dealing with it at the end. However, I cannot for the life of me work out how to remove both for loops.

Does anyone have any thoughts?

First of all it should be noted that the code you posted does not actually replicate the output of the dist function, because the line:

distancematrix[i, j] <- sum(abs(myMatrix[i,] - myMatrix[j,]))

does not calculate the Euclidean distance; it should be:

distancematrix[i, j] <- sqrt(sum((myMatrix[i,] - myMatrix[j,]) ^ 2))

Here are two solutions that rely on apply. They are simplified, and in particular do not take advantage of the symmetry of the distance matrix (which, if considered, would lead to a 2-fold speedup). First, generate some test data:

# Number of data points
N <- 2000
# Dimensionality
d <- 10
# Generate data
myMatrix = matrix(rnorm(N * d), nrow = N)

For convenience, define:

# Wrapper for the distance function
d_fun <- function(x_1, x_2) sqrt(sum((x_1 - x_2) ^ 2))

The first approach is a combination of apply and sapply:

system.time(
    D_1 <-
        apply(myMatrix, 1, function(x_i)
            sapply(1:nrow(myMatrix), function(j) d_fun(x_i, myMatrix[j, ]))
    )
)

   user  system elapsed
 14.041   0.100  14.001

while the second uses only apply (but going over the indices, which are paired using expand.grid):

system.time(
    D_2 <-
        matrix(apply(expand.grid(i = 1:nrow(myMatrix), j = 1:nrow(myMatrix)), 1, function(I)
            d_fun(myMatrix[I[["i"]], ], myMatrix[I[["j"]], ])
        )
    )
)

   user  system elapsed
 39.313   0.498  39.561

However, as expected both are much slower than dist:

system.time(
    distancematrix <- as.matrix(
        dist(myMatrix, method = "euclidean", diag = T, upper = T)
    )
)

   user  system elapsed
  0.337   0.054   0.388


What visualization technique to best describe a recommendation dataset? I've written a simple recommender which generates recommendations for users based on what they have clicked. The recommender generates a data file with the following format:

userid,userid,simmilarity (between 0 and 1 - closer to 0 the more similar the users)
a,b,.2
a,c,.3
a,d,.4
a,e,.1
e,b,.3
e,c,.5
e,d,.8

I've looked at some graphs, but I'm not sure which one to use, or if are there other ones that will better display the user similarities from the dataset above. Any suggestions?

I'm aiming this visualization at business users who are not at all technical. I would just like to show them an easy to understand visual that details how similar some users are and so convince the business that for these users the recommendation system is useful.

@Steve Kallestad do you mean something like this :

I think you're looking for a similarity matrix (see bottom of the page). If you don't have data on similarity between certain pairs, you can always leave them as grey or white. Also, this will only work for data sets small enough to actually make out what's going on. I'd say 25 rows / columns maximum.

In a similarity matrix, x, and y coordinates correspond to the two things you're comparing, while a colormap magnitude represents similarity

EDIT:
One thing you could do to replace the colormap is the insert, say, circles of different sizes according to the similarity metric. Or you could insert the numbers themselves, again, varying the size of the number as the magnitude of that number varies. Size usually works best is business visualizations.


Is there any existing algorithm or formula that can calculate the complexity of a dataset? Suppose that we have a dataset of n elements each element is composed of m sub elements, is there any existing algorithm that we can use to calculate the complexity of this dataset?

I mean by the complexity anything that includes the following elements:

Diversity of dataset (number of distinct elements)
Distance between each element
Correlation between the elements of this dataset
Size of the dataset

well ... "if n elements each element is composed of m sub elements" sound like you have a set of sets from discrete objects. If this is the case you can go through Combinatorial Analysis like Graph Theory. You can model it in two ways:

1) Using Hypergraphs: You have your elements as hyper-edges of a Hypergraph. This model lets you compute distances, diversities, sizes, and many more graph theoretical measures e.g. look at this, this and this.

2) Simple Graphs: You can create simple complete graphs from each sub-element and connect them to other complete graph (forming cliques) of other sub-elements. See the example bellow:

element1 = [[a,b,c],[b,c,d],[e,f,g]]
Graph = [(a,b),(b,c),(a,c),(b,c),(c,d),(b,d),(e,f),(e,g),(f,g)]

using this you will have a huge graph (network) of your object on which you can calculate tones of structural and statistical measures.

I assume my thought were naive as your question is not clear and you need to provide more detailed example to get a more accurate answer.

Good Luck!


Interpretation of an SVD for recommender systems The idea is to motivate the SVD for use in a recommender system.

Consider a matrix $A\in \mathbb{R}^{f\times u}$ where $A_{ij}$ caputures how user $j$ rates film $i$ (on a scale from 1-10, some entries may be missing).
Considering matrices $K=AA^T$, $L = A^TA$ what do $K_{ij}$ and $L_{ij}$ tell us?
What interpretations can you give for matrices $U$ and $V$ in the SVD $A=U\Sigma V^T$?

Riiight ... so
K = $AA^T$ = $U\Sigma^2U^T$, L = $A^TA = V\Sigma^2V^T$
With $K_{ij} = \langle A[i,:], A[j,:] \rangle$ being the dot product of all ratings for film $i$ with all ratings for film $j$.
Similarly, $L_{ij} = \langle A[:, i], A[:,j] \rangle$ being the dot product of all ratings from user $i$ with all ratings from user $j$.
That tells us ... what exactly? I expect it to amount to some kind of similarity measure, but beyond that, I have no idea.
As for $U$ and $V$ ...  I know the first $r$ of them to be Eigenvectors of $K$ and $L$ for some $r$. I also know
$u_1,..,u_r$ is an orthonormal basis for the column space
$v_1,..v_r$ is an orthonormal basis for the row space
but that doesn't really tell me anything.
Yet have no idea if the notion of "row space" and "column space" even makes sense. What would the "span" of $A$'s columns / rows even mean?
I also have no idea what the nullspace of $A$ or $A^T$ would represent - if it represents anything at all - nor what
$u_{r+1},..,u_{m}$ being an orthonormal basis for $ker(A)$
or $v_{r+1},..,v_{n}$ being an orthonormal basis for $ker(A^T)$
implies - again, if anything at all.

First of all, note that the dot product between two movies/users is by definition the correlation between them. So your intuition for treating it as a similarity measure is not wrong.

Now, applying SVD to $A$ is simply a re-representation of it in some other basis. Let's call it feature space. If we'll take it one step further and keep only $k$ leading vectors of $U$ and $V$, we'll get two matrices: $U^{(k)} \in \mathbb{R}^{f,k}$ and $V^{(k)} \in \mathbb{R}^{k,u}$ (where $f$ and $u$ are the dimensions you supplied in your question). where $U^{(k)}\cdot V^{(k)} \approx A^{(k)}$ is a $k$-rank approximation for $A$ ** (note it has the same dimensions).

So you can of $U^{(k)}$ as a matrix which rows movies, and each movie has some $k$ latent features. Similarly, $V^{(k)}$ will be a "list" (columns) of users and each user is described by some $k$ values (latent features).

Note that you can "split" the scaling matrix $\Sigma^{(k)}$ between $U^{(k)}$ and $V^{(k)}$: $U'^{(k)} = U^{(k)}\Sigma^{(k)0.5}$ and $V'^{(k)} = V^{(k)}\Sigma^{(k)0.5}$, to make up for the lack of scale of $U$ and $V$.

This factorization can be elaborated into more interpretable methods such as LDA.

** The variational approach to singular values tell us that this approximation minimizes the difference between $A$ and any other $k$ rank matrix in terms of Frobenius-norm and operator-norm (and possibly other norms I'm not aware of).


How to download BAIR action free robot pushing dataset? I'm trying to download BAIR action free robot pushing dataset. I tried downloading from here. In browser, it shows its size is 30GB, but downloads some data and then fails. I tried multiple attempts with no success. Then I tried to download using wget

wget http://rail.eecs.berkeley.edu/datasets/bair_robot_pushing_dataset_v0.tar

Even with this, it shows total size is 30GB, but after downloading some 199MB, it ended saying download is complete

wget http://rail.eecs.berkeley.edu/datasets/bair_robot_pushing_dataset_v0.tar
--2019-05-16 12:30:50--  http://rail.eecs.berkeley.edu/datasets/bair_robot_pushing_dataset_v0.tar
Resolving rail.eecs.berkeley.edu (rail.eecs.berkeley.edu)... 128.32.189.73
Connecting to rail.eecs.berkeley.edu (rail.eecs.berkeley.edu)|128.32.189.73|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 32274964480 (30G) [application/x-tar]
Saving to: ‘bair_robot_pushing_dataset_v0.tar’

bair_robot_pushing_dataset_v0.tar                    0%[                                                                                ] 189.95M   456KB/s    in 10m 59s

2019-05-16 12:41:50 (295 KB/s) - Connection closed at byte 199172826. Retrying.

--2019-05-16 12:41:51--  (try: 2)  http://rail.eecs.berkeley.edu/datasets/bair_robot_pushing_dataset_v0.tar
Connecting to rail.eecs.berkeley.edu (rail.eecs.berkeley.edu)|128.32.189.73|:80... connected.
HTTP request sent, awaiting response... 416 Requested range not satisfiable

    The file is already fully retrieved; nothing to do.

Also, I found a script that downloads BAIR dataset here. But I encountered the same problem here as well.

I'm confused now. Is the dataset so small or am I doing something wrong?

BAIR dataset can be downloaded here
https://sites.google.com/berkeley.edu/robotic-interaction-datasets

Additionally, here is the code to extract data from the dataset

import datetime
import os
import time

import cv2
import numpy as np
import skvideo.io
import tensorflow as tf
from PIL import Image
from tensorflow.python.platform import gfile

def get_next_video_data(data_dir):
    filenames = gfile.Glob(os.path.join(data_dir, '*'))
    if not filenames:
        raise RuntimeError('No data files found.')

    for f in filenames:
        k = 0
        for serialized_example in tf.python_io.tf_record_iterator(f):
            example = tf.train.Example()
            example.ParseFromString(serialized_example)
            # print(example)        # To know what all features are present

            actions = np.empty((0, 4), dtype='float')
            endeffector_positions = np.empty((0, 3), dtype='float')
            frames_aux1 = []
            frames_main = []
            i = 0
            while True:
                action_name = str(i) + '/action'
                action_value = np.array(example.features.feature[action_name].float_list.value)
                if action_value.shape == (0,):      # End of frames/data
                    break
                actions = np.vstack((actions, action_value))

                endeffector_pos_name = str(i) + '/endeffector_pos'
                endeffector_pos_value = list(example.features.feature[endeffector_pos_name].float_list.value)
                endeffector_positions = np.vstack((endeffector_positions, endeffector_pos_value))

                aux1_image_name = str(i) + '/image_aux1/encoded'
                aux1_byte_str = example.features.feature[aux1_image_name].bytes_list.value[0]
                aux1_img = Image.frombytes('RGB', (64, 64), aux1_byte_str)
                aux1_arr = np.array(aux1_img.getdata()).reshape((aux1_img.size[1], aux1_img.size[0], 3))
                frames_aux1.append(aux1_arr.reshape(1, 64, 64, 3))

                main_image_name = str(i) + '/image_main/encoded'
                main_byte_str = example.features.feature[main_image_name].bytes_list.value[0]
                main_img = Image.frombytes('RGB', (64, 64), main_byte_str)
                main_arr = np.array(main_img.getdata()).reshape((main_img.size[1], main_img.size[0], 3))
                frames_main.append(main_arr.reshape(1, 64, 64, 3))
                i += 1

            np_frames_aux1 = np.concatenate(frames_aux1, axis=0)
            np_frames_main = np.concatenate(frames_main, axis=0)
            yield f, k, actions, endeffector_positions, np_frames_aux1, np_frames_main
            k = k + 1

def extract_data(data_dir, output_dir, frame_rate):
    """
    Extracts data in tfrecord format to gifs, frames and text files
    :param data_dir:
    :param output_dir:
    :param frame_rate:
    :return:
    """
    if os.path.exists(output_dir):
        if os.listdir(output_dir):
            raise RuntimeError('Directory not empty: {0}'.format(output_dir))
    else:
        os.makedirs(output_dir)

    seq_generator = get_next_video_data(data_dir)
    while True:
        try:
            _, k, actions, endeff_pos, aux1_frames, main_frames = next(seq_generator)
        except StopIteration:
            break
        video_out_dir = os.path.join(output_dir, '{0:03}'.format(k))
        os.makedirs(video_out_dir)

        # noinspection PyTypeChecker
        np.savetxt(os.path.join(video_out_dir, 'actions.csv'), actions, delimiter=',')
        # noinspection PyTypeChecker
        np.savetxt(os.path.join(video_out_dir, 'endeffector_positions.csv'), endeff_pos, delimiter=',')
        skvideo.io.vwrite(os.path.join(video_out_dir, 'aux1.gif'), aux1_frames, inputdict={'-r': str(frame_rate)})
        skvideo.io.vwrite(os.path.join(video_out_dir, 'main.gif'), main_frames, inputdict={'-r': str(frame_rate)})
        skvideo.io.vwrite(os.path.join(video_out_dir, 'aux1.mp4'), aux1_frames, inputdict={'-r': str(frame_rate)})
        skvideo.io.vwrite(os.path.join(video_out_dir, 'main.mp4'), main_frames, inputdict={'-r': str(frame_rate)})

        # Save frames
        aux1_folder_path = os.path.join(video_out_dir, 'aux1_frames')
        os.makedirs(aux1_folder_path)
        for i, frame in enumerate(aux1_frames):
            filepath = os.path.join(aux1_folder_path, 'frame_{0:03}.bmp'.format(i))
            cv2.imwrite(filepath, cv2.cvtColor(frame.astype('uint8'), cv2.COLOR_RGB2BGR))
        main_folder_path = os.path.join(video_out_dir, 'main_frames')
        os.makedirs(main_folder_path)
        for i, frame in enumerate(main_frames):
            filepath = os.path.join(main_folder_path, 'frame_{0:03}.bmp'.format(i))
            cv2.imwrite(filepath, cv2.cvtColor(frame.astype('uint8'), cv2.COLOR_RGB2BGR))
        print('Saved video: {0:03}'.format(k))

def main():
    data_dir = '../softmotion30_44k/test'
    output_dir = '../ExtractedData/test'
    frame_rate = 4
    extract_data(data_dir, output_dir, frame_rate)
    return

if __name__ == '__main__':
    print('Program started at ' + datetime.datetime.now().strftime('%d/%m/%Y %I:%M:%S %p'))
    start_time = time.time()
    main()
    end_time = time.time()
    print('Program ended at ' + datetime.datetime.now().strftime('%d/%m/%Y %I:%M:%S %p'))
    print('Execution time: ' + str(datetime.timedelta(seconds=end_time - start_time)))

References:
https://github.com/edenton/svg/blob/master/data/convert_bair.py


Connect Orange 3.20 to postgresql database I installed orange 3.20 on windows 7. It works so far, the problem is connecting it to a server based Postgres database. While the connection can be made in the moment, when you try to load a table the message "missing extension quantile" comes up. A few problems are coming up with this message. It seems like it is not possible to install this extension on a windows server without a lot of stress. The extension seems not to be actual (version 2015) and is not compatible with the actual postgresql versions (which will cause trouble for future updates). Does anyone know if Orange will support this in the future and a convenient way to use Orange for windows 7?

To summarize my linked answer. While you may be able to connect Orange to Postgresql, Orange does not handle large datasets very well (it will cause it to crash). Generally, it does not support datasets the size that you would be storing in SQL sized databases. Personally, I think Orange works, and is most effective, with smaller sized datasets for illustrative tasks and exploratory data analysis.


PCA on matrix with large M and N Based on this answer, we know that we can perform build covariance matrix incrementally when there are too many observations, whereas we can perform randomised SVD when there are too many variables.

The answer provide are clear and helpful. However, what if we have a large amount of observations AND variables? e.g. 500,000 samples with 600,000 observations. In this case, the covariance matrix will be huge (e.g. 2,000 GB, assuming 8byte float, and if my calculation is correct) and will be impossible for us to fit it into memory.

In such scenario, is there anything that we can do to calculate the PCA, assuming we only want the top PCs (e.g. 15 PCs)?

There are a couple of things you can do.

Sample a representative, but small set of your data, which will allow you to compute PCA in memory. But seeing as you have 600,00 observations this will most likely not result in any meaningful results.
Use incremental PCA, here is a link: http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.IncrementalPCA.html#sklearn.decomposition.IncrementalPCA

But the main problem you have is that a number of samples are less than the amount of observations you have. I would recommend a different approach to dimensionality reduction. Autoencoders would be my recommendation to you. Autoencoders can be trained in an iterative fashion, circumventing your memory issue, and can learn more complicated projections than PCA (which is a linear transform). In case you want a linear projection you can have an autoencoder with one hidden layer, and the solution found by the neural network will be equal to the solution found by PCA.

Here are a couple of links you will find helpful:

http://ai.stanford.edu/~quocle/tutorial2.pdf
https://www.quora.com/How-is-autoencoder-compared-with-other-dimensionality-reduction-algorithms
https://www.cs.toronto.edu/~hinton/science.pdf


Voting combined results from different classifiers gave bad accuracy I used following classifiers along with their accuracies:

Random forest - 85 %
SVM - 78 %
Adaboost - 82%
Logistic regression - 80%

When I used voting from above classifiers for final classification, I got lesser accuracy than the case when I used Random forest alone.

How is this possible? All classifiers are giving more or less same accuracies when used individually, then how does Random Forest outperform their combined result ?

The approach you are considering is similar to a multi-class SVM or a one-vs-the-rest approach.

And here is how I describe the problem. The support vector machine, per example, is fundamentally a two-class classifier.

In practice, however, we often have to tackle problems involving K > 2 classes. Various methods have therefore been proposed for combining multiple two-class SVMs in order to build a multi-class classifier.

One commonly used approach (Vapnik, 1998) is to construct K separate SVMs, in which the kth model y_k(x) is trained using the data from class C_k as the positive examples and the data from the remaining K − 1 classes as the negative examples. This is known as the one-versus-the-rest approach where :

y(x) = max_k y_k(x)

Unfortunately, this heuristic approach suffers from the problem that the different classifiers were trained on different tasks, and there is no guarantee that the real-valued quantities y_k(x) for different classifiers will have appropriate scales.

Another problem with the one-versus-the-rest approach is that the training sets are imbalanced. For instance, if we have ten classes each with equal numbers of training data points, then the individual classifiers are trained on data sets comprising 90% negative examples and only 10% positive examples, and the symmetry of the original problem is lost.

Therefor, you got your bad accuracy.

PS: Accuracy, in most cases, is not a good measure for evaluating a classifier model.

References :

Vapnik, V. - Statistical Learning Theory. Wiley-Interscience, New York.
Christopher M. Bishop - Pattern Recognition and Machine Learning.


kde plot for interpreting the correlation i have created some new features for my model. I found people use kde plot to find out the correlation between the created feature and the target variable, but I am not really sure how to find the correlation from kde.

Any help on how to interpret the correlation from kde plot will very helpful

CoreyLevinson on kaggle answered my question. I am simply adding the explaination.

The kde shows the density of the feature for each value of the target. There are usually 2 colored humps representing the 2 values of TARGET. If the humps are well-separated and non-overlapping, then there is a correlation with the TARGET. If the humps are overlapping a lot, then that means the feature is not well-correlated with the TARGET, because the TARGET is equally as common on those values of the feature.


Targeted information extraction / focused extractive summarization I have a large collection of project manuals, each with a large number of pages. Each manual contains some form of summary paragraphs, although these are not necessarily similar in structure or format from one to the next. The rest of the manual generally contains a large amount of various information in relation to the project, and is not always relevant to the desired content to be extracted and summarized.

In theory-

paragraph 1 - Project Summary (Extract this)
paragraph 2 - Background info (ignore)
paragraph 3 - Background info (ignore)
paragraph 4 - Background info (ignore)
paragraph 5 - Project Requirements (extract this)

Is it possible to somehow selectively target paragraphs/ sentences within a document for extractive summarization, and if so, is possible to train a model with datasets containing full texts and their desired summarizations?

So far, I have tried general extractive summarization methods utilizing TF-IDF. However, due to the substantial amount of extraneous information within the document, critical text within the summary paragraphs is usually ignored. I've considered manually increasing word counts across various keywords, but I think this will still ignore relevant sentences and phrase within close proximity (same worry with binary classification of desired paragraphs/ sentences).

Can this be done, and is there a better approach than what I've been trying so far?

What about splitting the text into different paragraphs and select only the important paragraphs based on the keywords or patterns and then apply the summarization techniques only on the extracted paragraphs.


When can AUC and accuracy rate be equal? The terms AUC and accuracy has been confusing for me.
In which circumstance AUC rate and accuracy rate become exactly equal?

AUC (or most often AUROC = "area under receiver operating characteristic") and accuracy are different measures, but used for same purpose - to objectively measure performance of a simple binary classifier.

The two measures can be equal at extreme values of 0 and 1 for perfect classifiers - or inverse perfect classifiers (you can just invert the output to get a perfect classifier). It might be possible for the values to be numerically equal at other times, but if so it would be a coincidence with no specific meaning.

Both these metrics can be used with a simple classifier that only outputs true or false values for class membership. However, the AUROC metric requires some kind of parameter you can vary in order to plot the ROC curve. Usually this is a threshold for classification, used against the class probability output of a classifier.

There are other possible metrics. For example, F1 score and cross entropy. The F1 score again has 1.0 for a perfect classifier and 0.0 for a bad classifier, but cross entropy scores lower for better classifiers - 0.0 for perfect and no upper limit for bad output. Again, the values of these might be equal to other metrics at some points, but if so it is not meaningful.

If you are comparing two classifiers for a particular task, then it is important to compare them using the same metric, on the same test data. The metric and test data you choose should relate to your original problem.

If you are reading other people's published results, and want to compare them, it is not really possible if one person has used AUC and the other accuracy.


Using random forest when not all the observations have data I have this given database:

I would like to predict column "y" using the columns "index_1","index_2","index_3" using the random forest classifier.

as you can see, column "size: does not have values for each observation.

My question is : Can I still use random forest classifier when I don't have data for all the observations, and if yes, is it ok?should I give value (e.g "noData") to the empty cells? will it harm the prediction? or maybe no need?

In theory decision trees (and random forests) are able to deal with missing values in the data. But whether a particular implementation of the algorithm allows this (and how to use it with this implementation) depends on the specific package.


Confusion matrix and ROC AUC curves are not in sync I created a classification model with three target classes and created a confusion matrix to measure the accuracy, here is the matrix code

from sklearn.datasets import load_wine
data = load_wine()
x = pd.DataFrame(data=data.data, columns=data.feature_names )
y = pd.DataFrame(data=data.target, columns=['target'])
x_train, x_test, y_train, y_test = train_test_split(x,y,test_size=0.2,shuffle=True,random_state = 42)
model = DecisionTreeClassifier()

model.fit(x_train, y_train)

y_pred = model.predict(x_test)

mat=confusion_matrix(y_test, y_pred, labels=[0,1,2])

The output of the above matrix is

array([[13,  1,  0],
       [ 0, 14,  0],
       [ 1,  0,  7]])

So, this is a balanced dataset and giving me an accuracy of 94% almost.

The problem is, when I tried to draw ROC AUC curve for class 0 using the below code, the AUC curve is the opposite and I am getting only 0.05 area under the curve.

fpr,tpr, thres =  roc_curve(y_test, y_pred, pos_label=0)
roc_auc = auc(fpr,tpr)

plt.title('Receiver Operating Characteristic')
plt.plot(fpr, tpr, 'b', label = 'AUC = %0.2f' % roc_auc)
plt.legend(loc = 'lower right')
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0, 1])
plt.ylim([0, 1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()

I tried to calculate and draw ROC AUC manually and I got TPR as 0.9285 and FPR as 0.04545 and the graph is perfect on the sheet, could you please help me why the above code is giving the graph the other way. I verified the above code for the other two classes and the graph is good. Thanks in advance.

The ROC curve is built by taking different decision thresholds, and should be built using the predict_proba of your estimator.  In particular, in your multiclass example, the ROC is using the values 0,1,2 as a rank-ordering!  So there are four thresholds, the one between 0 and 1 being the most important here: there, you declare all of the samples the model predicts as being class 0 to be negative, and all others as positive, giving you the complementary point in the ROC-space.

Instead, you should have

fpr,tpr, thres =  roc_curve(y_test, model.predict_proba(x_test)[:,0], pos_label=0)
# or
fpr,tpr, thres =  roc_curve(y_test==0, model.predict_proba(x_test)[:,0])


Convert .cbf to lossless image format with defined "dynamic range" We collected electron diffraction data and converted the data into .cbf (Crystallographic Binary File). Those are about 10'000 files. Now I would like to convert the .cbf into an image format like .tif but with a specified dynamic range.

I tried it with fabIO which is a python module. But I could not figure out how to successfully convert .cbf into .tif. .edf worked but it's not the  desired format. Also I don't know how to pick the dynamic range.

Maybe it is possible to use Adxv in the command line? I've been using it to export .tif but was only able to do so one by one.

What follows is my python script to convert .cbf into .edf:

import os
from glob import glob
import fabio

path = '/Users/myuser/Documents/Crystals'

result = [y for x in os.walk(path) for y in glob(os.path.join(x[0], '*.cbf'))]

i = 0
text = ''
while i < len(result):

    if i == 4000:
        image = fabio.open(result[i])
        image.convert("edf").save("4000.edf")

    i += 1

Do you know how to use fabIO in order to export .tif with specific dynamic range? Or do you maybe know how to use Adxv in the command line in order to process many images? Or maybe you have another solution?

import os
from glob import glob
import fabio

path = '/Users/myuser/Documents/Crystals'

result = [y for x in os.walk(path) for y in glob(os.path.join(x[0], '*.cbf'))]

i = 0
command = ''
while i < len(result):

    path = os.path.dirname(result[i])
    filename = os.path.splitext(result[i])[0]+'.tif'

    command = '/Applications/Adxv.app/Contents/MacOS/Adxv -smv32bits -sa -overload 500 '+result[i]+' '+filename
    os.system(command)

    i += 1

In the end I figured it out using Adxv in stand alone -sa and -overload N as dynamic range.


How do I obtain the weight and variance of a k-means cluster? I am trying to reproduce the results of this paper, but using python and the HMMlearn library instead of matlab. The paper describes a procedure for using HMM (Hidden Markov Model) in order to predict stock prices.

The paper details the use of a 4-state, 5-mixture Gaussian distribution as the model. The transition probabilities and the initial state probabilities are uniform, however the emission probabilities are determined based on the results of a k-means algorithm using the data-set of existing stock prices.

This latter part is where I have gotten stuck, the paper advises to use the means, variance and weight of each cluster returned from the k-means algorithm as the mean, variance and weight of each component of the mixture. As I understand it the mean of the cluster is simply the center of each centroid, however I'm not sure how you would obtain the variance or the weight.

TL;DR
Given a 3 dimensional dataset X (in the form [[a, b, c], [d, e, f]...]) and using the k-means algorithm where k = 5 (k = number of mixture components), how would I determine the weight and variance of each cluster?

It is valid to use the k-means to initialize the EM for Mixture of Gaussian modeling. As you said, the mean of each component will be the average of all samples belong to the same cluster (it depends on the used clustering algorithm, some times the centroid is not the average of the cluster but is one of the samples). for the weight you can use the following: the weight of cluster x = the number of samples belong to cluster x divided by the total number of samples. thus, the cluster with the highest number of samples is the cluster with the highest weight. for the variance: just find the variance of all samples belong to the same cluster.


Linear function gradient descent I am trying to implement a gradient descent algorithm for a simple linear function:
y(x) = x

Where initial hypothesis function is:
h(x) = 0.5 * x

and learning rate:
alpha = 0.1

Target function graph is blue and hypothesis is green.
Cost function:
J = 1/2m * sum[(h(x) - y(x)) * (h(x) - y(x))]

Gradient descent:
q = q - alpha/m * sum[(h(x) - y(x)) * x]

My implementation does not converge:
import numpy as np
import matplotlib.pyplot as plt

def y(x):
    return x

def get_h(q):
    """ Create hypothesis function

        Args:
            q - coefficient to multiply x with

        Returns:
            h(x) - hypothesis function
    """
    return lambda x: q*x

def j(x, y, h):
    """Calculte a single value of a cost function

        Args:
            x - target function argument values
            y - target function
            h - hypothesis function

        Returns:
            Value of a cost function for the given hypothesis function
    """
    m = len(x)
    return (1/(2*m)) * np.sum( np.power( (y(x) -h(x)),2 ) )

def df(h, y, xs):
    """Calculate gradient of a cost function

        Args:
            h - hypothesis function
            y - target function
            xs - x values

        Returns:
            differential of a cost function for a hypothesis with given q

    """
    df = np.sum((h(xs)-y(xs))*xs) / len(xs)
    return df

xs = np.array(range(100))
ys = y(xs)
hs = h(xs)

costs = []
qs = []
q = 0.5
alpha = 0.1
h = get_h(0.5) # initial hypothesis function
iters = 10
for i in range(iters):
    cost = j(xs,y,h)
    costs.append(cost)
    qs.append(q)
    print("q: {} --- cost: {}".format(q,cost))
    df_cost = df(h, y, xs)
    q = q - alpha * df_cost  # update coefficient
    h = get_h(q) # new hypothesis

What am I doing wrong? Should I account for q0 even if my target function intercept is zero?
Update
Answer: https://stats.stackexchange.com/questions/484750/linear-function-gradient-descent

As you and @gunes pointed out in this post, your formula are correct, but hyperparameters $\alpha$ and $iterations$ were not well adjusted.


Sparse IR with user feedback I'm considering a problem framing within an information retrieval context.

I have a sequence of documents that feature different attributes.  In the web context, these would be webpages.  One attribute could be "is this a top-10 content creator," etc.  When we convert multi-labeled values to binary indicators, we end up with a matrix like:

   a b c d e f
A: 1 0 0 0 1 0
B: 0 1 0 0 1 0
C: 0 0 1 0 0 0
D: 0 1 0 0 0 1
E: 0 0 0 0 1 0
F: 0 0 1 0 0 1

We can ask a user a series of progressive questions about their preferences within this dataset.  Eg. "Do you care about it being from a top 10 content creator?"

However, these are just preferences.  Just because we know the "answer" to a question doesn't immediately invalidate documents with that attribute.  It should just bump it down in the eventual ranking.

The task at hand becomes knowing A) which questions to ask to maximize information gain and B) ranking the resulting list.  This seems like it would be an existing research area but thus far I haven't been able to find anything on it.  Is there a name for this area of algorithm design?

This is a learning to rank problem, in particular feature selection for learning to rank.

There are many ways to solve the problem. One common method is maximizing feature importance while minimizing similarity. In other words, rank documents based on the most valuable and unique features. This is a variation of Minimum-redundancy-maximum-relevance (mRMR) feature selection.

To maximize feature importance, first rank all documents using all features, evaluate performance with evaluation measures, and show documents with the highest importance scores.

To minimize similarity, find features that have the least redundancy. Redundancy could be measured as the average value of all mutual information values between each feature pair.

"Feature Selection for Ranking" goes into greater detail.

Once the features are ordered with mRMR, the user can select specific features. Then re-rank documents based on the user-selected features.


ML/NN as Function Evaluator for further Optimization (maximization) - Practical Example I am working on a production optimization problem; a very similar idea to what is described by Vegard Flovik How to use machine learning for production optimization. The following image, taken from the referred post, summarizes it very well:

First step is obvious, and I do have a model in the form of machine learning or neural networks model. How would I go about the second step? How can I use the trained model as the function evaluator for further multi-dimensional nonlinear optimization (e.g. maximization) either via Scipy, Bayesian Optimization etc.?
I cannot seem to find a practical example. Having a closed-form analytical function as the objective of an optimization problem is well-established. The article Optimization with SciPy and application ideas to machine learning by Tirthajyoti Sarkar gives a few examples using Scipy, & introduces packages that do  optimizations with bound constrains and more. Yet examples are quite simple (a closed-form mathematical function) and he only glosses over the extension of such idea to use NN as the objective function, I'm quoting:

You are free to choose an analytical function, a deep learning network
(perhaps as a regression model), or even a complicated simulation
model, and throw them all together into the pit of optimization.

Any leads/hints/links are appreciated!

[Appendix]
In order to have a concrete example, let's imagine we have a dummy data set with a set of feature and a imaginary ProductionYield that is a nonlinear combination of input variables:
import numpy as np
import pandas as pd

df = pd.DataFrame(columns=['Pressure','Temprerature','Speed','ProductionYield'])

df['Pressure'] = np.random.randint(low= 2, high=10, size=2000)
df['Temprerature'] = np.random.randint(10, 30, size=2000)
df['Speed'] = np.random.weibull(2, size=2000)
df['ProductionYield'] = (df['Pressure'])**2 + df['Temprerature'] * df['Speed'] + 10
df['ProductionYield']= df['ProductionYield'].clip(0, 100)

   Pressure  Temprerature     Speed  ProductionYield
0         7            20  1.810557        95.211139
1         2            29  0.674221        33.552409
2         8            17  0.537533        83.138065
3         3            24  1.945914        65.701938
4         6            23  0.514679        57.837610

1.Predictive Algorithm (a simple Neural Network):
## Train/Test Split
from sklearn.model_selection import train_test_split

x_train, x_test, y_train, y_test = train_test_split(df[['Pressure','Temprerature','Speed']].values, df['ProductionYield'].values, test_size=0.33, random_state=42)

## Build NN Model
import tensorflow as tf
from tensorflow.keras import layers

def build_model():

    # create model
    model = tf.keras.Sequential()
    model.add(layers.Dense(64, input_dim=3, kernel_initializer='normal', activation='relu'))
    model.add(layers.Dense(128, kernel_initializer='normal', activation='relu'))
    model.add(layers.Dense(1, kernel_initializer='normal'))

    # Compile model
    model.compile(loss='mean_squared_error', optimizer='adam')

    return model

model = build_model()
model.fit(x_train, y_train,
          validation_split=0.2,
          verbose=0, epochs=1000)

2.Otimization [Core of the Problem]:
Problem lies herein, when a ML/NN is trained, I do not get to see (export as I wish) the mathematical form of the function (here in this example NN) and its variables (which should be my feature variables) to do the optimization as we do with closed-form explicit mathematical functions.
[UPDATE 15.01.2021]:
Following Valentin's great answer, I've put pieces together in a practical example showcasing how one can use a ML/NN model as an input function for further optimization (herein via scipy.optimize) using the dummy data set shown in the Appendix. Please see this notebook for more details.

This post seems similar to yours and may help. It seems that what you are looking for is a derivative-free optimization method. The Wikipedia page for the concept lists such methods.
Intuitively, these techniques will sample the function (the network in your case) with various inputs (pressure, temperature, speed) and will figure out which inputs optimize it. Where they differ is in their sampling strategy, as it may be impractical or expensive to sample.
You can use scipy.optimize.minimize to do that. Pass your network as func and use an initial guess, which can be the last values of your variables. Scipy expects a function with the following signature: fun(x, *args) -> float where x is a one-dimensional numpy array. This might mean you will need to wrap your network in something like this:
def wrapper(x, *args) -> float:

    network_input = _numpy_to_valid_network_input(x)

    network_output = network.predict(network_input, *args)

    scipy_output = _network_output_to_float(network_output)

    return scipy_output

And then, you can pass wrapper as your func. Negating the output of scipy_output will turn the minimization problem into a maximization problem. If your input variables are bounded, i.e. [0, 100], you can do two things:

Use algorithms that allow you to define these bounds explicitly (i.e. L-BFGS-B using the bounds parameter)
Implicitly bound your inputs by using a sigmoid function for instance. By doing something like I did below, you can create a function bounded_value that always returns values within the range of your choice, even if the optimization algorithm might try any float.

import numpy as np

def sigmoid(x):
  return 1 / (1 + np.exp(-x))

def bounded_value(x, min_value=0, max_value=100):
   return min_value + (sigmoid(x) * (max_value - min_value))

If you want to bound your output, that's even easier. If your goal is to minimize, return a very large value if your network outputs any value outside of your range. Obviously, ensuring that your network always returns sensible values could be achieved by tweaking the loss used during training (which is difficult with AutoKeras).
If you want to implement your own method, I would recommend coordinate descent as you don't have many input dimensions (3) and it is quite simple to implement from scratch. Obviously, a brute-force approach where you randomly sample the space and choose the inputs that yielded the best function value is even easier to implement.


Is there a way to visualise what/how my image classifier has identified an image? Using tensorflow.
If I have trained my image classifier many images of a close up of melanomas and other skin cancers.

Then ask it to classify using the trained model a photo that has another element for example melanoma on a hand (with other elements that look like melanoma in the background example a hypothetical tablecloth that has melanoma patterns).

Is there a way to visualize what the image classifier has identified as melanoma ? Outputting a circled/highlighted the found melanomas in the photo?

What you are looking for is a model with an "Attention mechanism". Unfortunately, this means that you have to change your model and to retrain it.
It is more complicated that a simple image classifier but I can assure you that it is worth it to learn how it works. Or you can simply find an existing repository that implement it.

Alternatively, you can train your model on smaller images that contain exactly the melanoma. The preprocessing will be much heavier.
Then you will scan your bigger images with a bounding box this is known as sliding window. You will have probabilities for each small images.

Sliding Window (Formally):

The propose is:
Given a classifying model $F(J)$ where $J$ are images of size $w \times h$ you will apply this model to a image $I[m,n]$ by slicing it into many blocks $B[m,n]$ where
$$ B[m,n] = I\Biggl[\Biggl(m-\frac{w}{2}:m+\frac{w}{2}\Biggr),\Biggl(n-\frac{h}{2}:m+\frac{h}{2}\Biggl)\Biggr]$$

And you will apply the model as $F(B[m,n]), \forall m \in [w,W-w] and n \in [h,H-h]$ where $W$ and $H$ are the width and height of $I$.

Important to notice that for OpenCV images the height is actually the first coordinate of the variable (numpy array).

If you have a mean size of melanoma images you can run through  $m$ and $n$ with steps of half of the mean size.

So you might search for which parts of the image where classified as melanoma by looking into $$H[m,n] = F(B[m,n])$$ and using it as a binary mask to identify melanoma classified image parts.


Searching for a dataset that targets difficult words I am trying to find a dataset in which dataset targets words that are difficult. I understand there would be different levels of difficulty for each individual , but if we considered an average individual, I want to detect the difficult words that would be present in a sentence.
Example:
Yes, may be today's Britains are not responsible for some of these reparations but the same speakers have pointed with pride to their foreign aid - you are not responsible for the people starving in Somalia but you give them aid surely the principle of reparation for what is the wrongs that have done cannot be denied.
IN the above sentence, the model should detect the word - reparations successfully.

On the basis that difficult words are difficult because they are not commonly used, I think something as simple as TF-IDF would work well.


How do I approach a classification problem where one of the classes is defined by 'not any of the others' Suppose that I am interested in three classes $c_1$, $c_2$, $c_3$. But my dataset actually contains several more real classes $(c_j)_{j=4}^n$.

The obvious answer is to define a new class $\hat c_4$ that refers to all classes $c_j$, $j>3$ but I suspect this is not a good idea since the samples in $\hat c_4$ will be rare and not very similar to each other.

To visualize what I'm trying to say, suppose I have the following two variable space and the classes $c_1$, $c_2$, $c_3$, $\hat c_4= \bigcup_{j=4}^n c_j$ are depicted in red, til, green and black respectively. This is how I suspect my data would look like.

Is there any standard way to approach this problem? What would be the most efficient classifier and why?

I would use a two-step approach, using the idea of the $\hat{c_4}$ class you mentioned.

In the first step, use a binary classifier(trained on the whole dataset) to decide if a sample belongs to the class $\hat{c_4}$ (i.e. in any non-interesting class). For this, step you could also take a look in outlier detection methods, if the samples belonging in the "interesting" classes are much different than the rest.

If the result is negative, move on to the next step, a new classifier trained only on samples belonging in the classes $c_1,c_2,c_3$ and use that prediction as your final one.

I think that even using a simple clustering approach as a first step (e.g. 4-clustering k-means using as initial centroid values the average centroid $cent_j = \frac{\sum\limits_{x_i\in D: y_i=j}x_i}{\sum\limits_{x_i\in D: y_i=j}1}$  for each $c_1,c_2,c_3, \hat{c_4}$), would still be useful.


How to use 5 by 5 or 7 by 7 kernel size for a deep learning network with 3 by 3 kernels? I am using a U-Net architecture. The visual area of the segmentation mask is very small and after learning it is giving a lot of false positives. I am thinking of changing the kernel size of 3 by 3 to 5 by 5. What precautions/care shall I take while creating the model? I am getting issue of dimensionality mismatch if just change 3 by 3 to 5 by 5 kernel size.

The main think you need to be mindful about is the size of the output of your convolutional layer, remember that it's going to be
$$
\frac{W-K+2P}{S} + 1
$$
with your K the kernel size, so if you increase your kernel size the size of your output will decrease faster. You might have to reduce the number of layers. If you're using a framework for which you need to specify input size for each layer then in that case you would need to change that as well.


Calculating cosine similarity between 3D arrays using Python I have two matrices with multiple columns and three rows each. I calculated the cosine similarity (sklearn) but it gives the result as a matrix. How can I obtain one single value?
The matrices are the embeddings of two words each, obtained from BERT.

Your input matrices (with 3 rows and multiple columns) are saying that there are 3 samples, with multiple attributes. So the output you will get will be a 3x3 matrix, where each value is the similarity to one other sample (there are 3 x 3 = 9 such combinations)

If you were to print out the pairwise similarities in sparse format, then it might look closer to what you are after.

I have created two example matrices of random numbers that fits your description:

from sklearn.metrics.pairwise import cosine_similarity
from scipy import sparse

a = np.random.random((3, 10))
b = np.random.random((3, 10))

# Create sparse matrices, which compute faster and give more understandable output
a_sparse, b_sparse = sparse.csr_matrix(a), sparse.csr_matrix(b)

sim_sparse = cosine_similarity(a_sparse, b_sparse, dense_output=False)
print(sim_sparse)

Output:

  (0, 2)    0.7938732813430508
  (0, 1)    0.7575978172453429
  (0, 0)    0.7897664361147338
  (1, 2)    0.740418315571796
  (1, 1)    0.833981672896221
  (1, 0)    0.7184526671218405
  (2, 2)    0.8746293481677073
  (2, 1)    0.6456666045233884
  (2, 0)    0.7925289217609924

Hopefully this output makes it clearer, what you are actually getting as output.

Have a look here for a few more details on performance aspects, and the documentation on sparse matrices is here.


Error using corpus widget in Orange v 3.25.0 text mining and no import document option? I am experimenting with Orange data mining tool. When I use the 'Corpus' widget from the text mining section it gives me the error: [![Corpus widget error][1]][1]
I have tried many things, but am still unable to resolve this issue.
[1]: https://i.stack.imgur.com/X4tKu.png
Besides that, in text mining, it does not show import document option in orange v 3.25. I just want to know the options to import text files into the Orange.

I believe your Text version is outdated. Please update Text add-on. For issues with Orange, please always head over to the issue tracker and see if your issue already exists. If not, open a new issue.


What's the correct objective function for cosine similarity of two vectors to be 1 or 0? The representation learning model produces vectors for objects. I want the cosine similarity of some vector pairs to be (close to) 1, some to be 0. What objective function should I use? MSE as training a regression model?

There are a wide variety of loss metrics to compare two vectors (e.g., Triplet loss, Lifted Structure Loss, N-Pair loss, and Angular Loss).

Cosine distance (1-cosine similarity) can be used. Typically, a loss metrics needs to be minimized so distances metrics should be used.


How does a zero centered activation functions like tanh helps in gradient decent? I know that, if X are all positive, or negative then the sign of the downstream gradient will be same as that of the upstream gradient, but what I don't understand is how the zero centered activation function can overcome this problem?

Since, even in the case of tanH function, if all X are positive then too the sign remains same.

Forgive my english, not a native speaker.

$\tanh$ function is scaled standard sigmoid function $y=\frac{1}{1+e^{-ax}}$. Due to that scaling it has more steep gradient that standard sigmoid function. Steep gradient is important, because it makes backprop training faster and less likely to get stuck in zero-gradient area.

Furthermore $y(0) \neq 0$ where $\tanh(0) = 0$. Sometimes it is very important, especially when dealing with normalized $[0,1]$ signals as inputs.


How to find 4th point equidistant from 3 other points For example, if a=100 , b =200, c=300, then what should be the 4th point equidistant to all the 3 points ie a,b,c. Similarly, is there is any solution for nth point equidistant from n-1 points. The values are 1 dimensional points.

As Ankit said in comments, this is not possible in your concrete case, as you cannot just expand 1d point coordinates to 2d without inventing some information.

In 1d, where each point it represented by one single number, it would only be possible to find the 3rd point, equidistant to two original points. If you imagine the two points on a line, the third point would then just be the centre of those two points, which also happens to be the mean of those two points.

For three points, you would require a 2d space, i.e. each point must be a vector of length two: so basically an (x, y) coordinate pair.

If you did have (x,y) coordinates for three unique points, they would form a triangle, and the equidistant position (i.e. your fourth point) is called the circumcenter, and it found by finding the centre of each of the sides of the triangle, then drawing a line through each, which is perpendicular to its corresponding side of the triangle. These three lines will cross at the circumcenter - and it is not necessarily inside the triangle!

Example:


TF-IDF for Topic Modeling Can TF-IDF be used a sole method for Topic Modeling ? (I know there are better methods like LDA , LSA etc)
I just want to understand if TF-IDF alone can help us in Topic modeling . If yes , can someone explain how that simple framework works ?
I want to understand the application and capabilities of TF-IDF as a sole method for Topic Modeling. I could not find this anywhere else in the internet .

Formally the problem of topic modelling is a clustering problem: given a collection of text documents, group together the documents which are topically similar.
So technically it can indeed be done with a TF-IDF representation of documents as follows:

Collect the global vocabulary across all the documents and calculate the IDF for every word.
Represent every document as a TF-IDF vector the usual way: for every word, obtain the term frequency in the document (TF) then multiply by the global IDF for this word (IDF). Note that every vector must represent the document over the global vocabulary.
Use any clustering method over the vector representations of the documents: K-means, hierarchical clustering, etc.

Note that this method is unlikely to be as good as state of the art methods for topic modelling.


Decision Trees and Feature Selection I'm trying to experiment with the performance of different machine learning algorithms before and after applying feature selection.

I tested SVM, Random Forest, KNN, Linear Regression, and, Decision Tree and I used Random Forest as a feature selection method by utilizing the function (SelectFromModel) provided by sklearn.

All the algorithms improved significantly after applying feature selection, except the Decision Tree. I'm trying to find an explanation of why that happened? Does it mean that Decision Tree needs a large number of features to produce a good model?

I understand CART as a Classification and Regression Tree, as just a Decision tree. Decision Trees are greedy and deterministic.

Random Forest is an ensemble of decision trees, so you will be using random trees ensembling to select features.

I am guessing, your question is not really descriptive about the situation, that since the decision tree is not pruned and uses all data it gives a good solution there is no need to take care of overfitting or pruning They normally achieve better in the train than Random Forest because they overfit easily.

Taking out features will just give the tree less information, so less power.

  Does it mean that CART needs a large number of features to produce a good model?

Decision trees do good with a small number of features, it is not related.

Just to check, are you evaluating in the test set right?


How to detect misclasification data after multiclass classification? I have trained a neural network multi-class classification model with around 150 classes having around 85% accuracy. Once the model is trained and deployed, it's predicting on new data and I am saving the logs. Now I have to detect those data-points which is wrongly predicted by the model.
For example, The model predicted on 10 data-points, out of which there might be 3 data-points which are wrongly classified by the model. Is there any way to get those data-points? I have the following data.

The deployed model.
The data-points.
Corresponding predicted classes by the model.
Confidence scores on the prediction.

I know that using confidence score I might get some idea on wrong prediction, but I am thinking is there any other way to get that?

Sadly, this isn't possible. Simply put, if you could know for sure that your model made a mistake, then, your model will never make mistakes.
This is the painful reality of deploying models in practice. You will need manual validations to confirm whether your model performed correctly. In some systems, you may be able to outsource this to the user itself (i.e. give them an option to say the model was wrong), but you'll likely want to run independent validation.
Now, there are indeed ways to make this part less painful. First, as you said, you should take a look at the confidence scores. It should indeed be less necessary to verify predictions where the model is highly confident. You should also use your confusion matrix to figure out which predictions are more crucial to verify. Indeed, some mistakes may be less harmful than others.
What you could also do is experiment with building auxiliary binary models that take in the input, the model output, and predicts whether the model made a mistake. This isn't guaranteed to work, but it may give you better confidence scores for your model.


It sounds like you want to use a regression model (which produces a real value as the output, e.g. a value in the range [-1, 1] but possibly outside as well depending on the learning algorithm) instead of a classification model (which produces one of a set of discrete output values based on the training data, e.g. {-1, 0, 1} in your case).
If the code is changed to use MLPRegressor instead of MLPClassifier, it will provide a real value as output.
Instead of:
from sklearn.neural_network import MLPClassifier
model = MLPClassifier(...)

Use:
from sklearn.neural_network import MLPRegressor
model = MLPRegressor(...)


Time series normalization using min max technique I have a time series dataset and I would like to normalize the data (diff which is of type list) as below using Min Max technique. But, I get the following error:

Code:

# split data into train and test-sets
train, test = diff[0:1486], diff[1486:2123]
from sklearn.preprocessing import MinMaxScaler
# scale train and test data to [-1, 1]
def scale(train, test):
    # fit scaler
    scaler = MinMaxScaler(feature_range=(-1, 1))
    scaler = scaler.fit(train)
    # transform train
    train = train.reshape(train.shape[0], train.shape[1])
    train_scaled = scaler.transform(train)
    # transform test
    test = test.reshape(test.shape[0], test.shape[1])
    test_scaled = scaler.transform(test)
    return scaler, train_scaled, test_scaled
# transform the scale of the data
scaler, train_scaled, test_scaled = scale(train, test)

Error:

ValueError: Input contains NaN, infinity or a value too large for dtype('float64').
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

To resolve the issue, I used diff() method to remove trends in diff.diff()


How to get probabilities values with keras? tensorflow version = '1.12.0'

keras version = '2.1.6-tf'

I'm using keras with tensorflow backend.

I want to get the probabilities values of the prediction.
I want the probabilities to sum up to 1.
I tried using 'softmax' and 'categorical_crossentropy' but nothing works.

This is my model:

X = pickle.load(open("X.pickle", "rb"))
y = pickle.load(open("y.pickle", "rb"))

number_of_gestures = 5
y = to_categorical(y, num_classes=number_of_gestures) #to_categorical is a function from keras - np_utils.

model = Sequential()
model.add(Conv2D(16, (2,2), input_shape=(IMG_SIZE, IMG_SIZE, 1), activation='relu'))

model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='same'))

model.add(Conv2D(32, (5,5), activation='relu'))

model.add(MaxPooling2D(pool_size=(5, 5), strides=(5, 5), padding='same'))

model.add(Conv2D(64, (5,5), activation='relu'))

model.add(Flatten())

model.add(Dense(128, activation='relu'))

model.add(Dropout(0.2))

model.add(Dense(number_of_gestures, activation='softmax'))

sgd = optimizers.SGD(lr=1e-2)
model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])
model.fit(X, y, batch_size=500, epochs=40, validation_split=0.1)

The probabilities looks like that:

  [1. 0. 0. 0. 0.]

And I want it to look like this:

  [0.897. 0.023. 0.158. 0.780. 0.1021]

I know it does not sum up to 1 but this is just an example.

I faced such a problem using CNN in Keras. Even, I thought that the output is being processed in the such a way like the argmax().

But, after investigating the model, I found that the model are learned and generalised so well that the labels got binarized ( 0s and 1s ). Remember, that we give the label of an image as a one hot vector ( like [ 1 0 0 0 ] ). The NN learned it so well that the output, usually being a probability vector , was now a one hot vector on which it was trained.
Another way to prove this is that you are using a softmax activation function at the output layer. A softmax function produces probabilities which sum up to 1. The sum of all class probabilities is 1. In the vector [ 1 0 0 0 0 ]. The sum of all numbers is 1. Hence the output is valid.

  This is just a different phase in training your model. You can try to increase the size of your dataset or lower the learning rate to 0.0001 or even smaller.


Interpreting coefficients after transforming Y (dependent) variable I have been working on linear regression for forecasting purposes.

I have a model where I have used a BoxCox transformation on the $Y$ variable (Sales) with $\lambda = 0.3$. The model is as follows:

$$Y = 13 - 0.2 X_1 + 3 X_2$$

How can I interpret the effect of sales from each variable, given the transformation?

For instance, assume $X_1=5$ and $X_2=0.3$ then we get:

$$Y = 13 - 0.2 \times 5 + 3 \times 0.3 = 12.9$$

Reverse BoxCox of 12.9 with $\lambda = 0.3$ is 196. So it forecasts 196 sales. But if we increase $X_2$ from 0.3 to 1 and increase $X_1$ from 5 to 10 then we get:

$$Y = 13 - 0.2 \times 10 + 3 \times 1 = 14$$

Reverse BoxCox then gives 244 sales forecasted sales. An increase of 48 sales. But how much of that is down to $X_1$ and $X_2$?

If we calculate them separately the answer doesn't add up!

EDIT: to make the above point clear:

If $X_1$ went from 5 to 10 but $X_2$ doesn't change:

$Y = 13 - 0.2 \times 10 + 3 \times 0.3 = 11.9$, transform to 159 sales. A -37 decrease from the starting position (196-159)

However, if $X_2$ went from 0.3 to 1 but $X_1$ doesn't change:
$Y = 13 - 0.2 \times 5 + 3 \times 1 = 15$, transforms to 294 sales. A +98 increase in sales (294-196).

But if $X_2$ increases sales by 98 and $X_1$ decreases sales by 37 you would expect the net increase to be +61 when you include both changes. But from earlier we found that by applying both we actually get a +48 increase in sales!

How can we work out the actual sales change resulting in changes in $X_1$ and $X_2$?

By using BoxCox transformation on the Y variable, you are turning your model from linear regression into non-linear regression. Note that although it seems you are training a linear regression, it’s only linear with respect to the transformed version of Y but non-linear regarding the original Y. Therefore you can not expect a linear behaviour from a non-linear model which means you can not work out the actual sales change resulting in changes in X1 and X2. That’s because unlike linear models, in non-linear models f(x+b) is not equal to f(x) + f(b)


Different hidden layer architectures deliver the same classification results, is that normal? I have a data set with 600 data points with about 10 attributes (binary).
The dataset has been normalized:

Xnormalized = StandardScaler().fit_transform(X)

The outputs are binary (2 classes)
I tried different MLP architectures using 1 or 2 hidden layers with different number of nodes and performed a repeated stratified cross validation as shown below:

names=[]
classifiersmlp = []
for i in range(5, 26,5):
    names.append("mlp-"+str(i))
    classifiersmlp.append(MLPClassifier(solver='sgd',
                                        random_state=1,
                                        activation='tanh',
                                        hidden_layer_sizes=[i]))
    for j in range(5,26,5):
        names.append("mlp-"+str(i)+"_"+str(j))
        classifiersmlp.append(MLPClassifier(solver='sgd',
                                            random_state=1,
                                            activation='tanh',
                                            hidden_layer_sizes=[i,j]))

cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=1)
scoring = {'accuracy': 'accuracy',
           'recall': 'recall',
           'precision': 'precision',
           'f1_score':'f1'}
mlpResults = []
for name, clf in zip(names, classifiersmlp):
    print(name)
    cvresultMLP = cross_validate(clf, Xnormalized, y, cv=cv, scoring=scoring)
    mlpResults.append(cvresultMLP)
    print(np.mean(cvresultMLP['test_recall']))
    print(np.mean(cvresultMLP['test_precision']))
    print(np.mean(cvresultMLP['test_f1_score']))

The results of all architectures are very similar (only 1-2% difference in all 3 evaluation measures (recall (approx 78%) and precision (approx 74%)). Is it normal that the architectures are equally good or should there be greater differences? What does it mean that the results are so similar?

After a comment from @yohanesalfredo that the scaling is outside the cv, I updated the code:

names=[]
classifiersmlp = []
for i in [5,25,50]: #range(5, 5,3)
    names.append("mlp-"+str(i))
    classifiersmlp.append(MLPClassifier(solver='sgd',
                                        learning_rate_init=0.01,
                                        random_state=1,
                                        activation='tanh',
                                        hidden_layer_sizes=(i)))
    for j in [5,20]: #range(1, 5,3):
        names.append("mlp-"+str(i)+"_"+str(j))
        classifiersmlp.append(MLPClassifier(solver='sgd',
                                            random_state=1,
                                            learning_rate_init=0.01,
                                            activation='tanh',
                                            hidden_layer_sizes=(i,j)))

scoring = {'accuracy': 'accuracy',
           'recall': 'recall',
           'precision': 'precision',
           'f1_score':'f1', # according to docu only score for the 1 label
          'roc_auc':'roc_auc'}
mlpResults = []
rand_state=1
for name, clf_temp in zip(names, classifiersmlp):
    cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=rand_state)
    rand_state+=1
    classifier_pipeline = make_pipeline(preprocessing.StandardScaler(), clf_temp)
    cvresultMLP = cross_validate(classifier_pipeline, X, y, cv=cv, scoring=scoring)
    mlpResults.append(cvresultMLP)
    print(np.mean(cvresultMLP['test_recall']))
    print(np.mean(cvresultMLP['test_precision']))
    print(np.mean(cvresultMLP['test_f1_score']))
    print(np.mean(cvresultMLP['test_roc_auc']))

There are several things I would like to mention :

I do not think that much change on your architecture will impact a lot. Try comparing 10, 20, 50, 100 or more depth. Difference will be most likely to be slightly more noticeable.
You are comparing accuracy which is based on labels instead of the probabilities. Try comparing logloss or auc score which is more sensitive to small changes.

Hope this helps.


Understanding these probabilities, and reading the pdf and cdf UPDATE: this is a math question but as i would like to codify it, I thought the DS SE might be appropriate.

I am trying to sketch out this problem statement. Where am I going wrong since I don't see how they get

"6% chance of losing any money at all",
"41% chance of making more    than \$100M" and
"75% of earning at least \$40M".

My attempt at a solution is in the second image: Row 48 (CASES) enumerates the number of ways the sum can take a given value, while row 49 (PROB) is that number in row 48 divided by the total number of outcomes. This gives me the pdf and row 50 is the cdf. Yet I don't get the answers in the problem statement. Are they wrong or am I reading the pdf and cdf the wrong way?

The first image is the text below.

The payoff aren't +30 and -10. They're +20 and -10. Successful projects return \$30M after investing \$10M.


Type of images used to train a neural network I am a newbie in neural network.

Saw this article Object detection with deep learning and OpenCV.
These three type of neural network are shortlisted in the article

Faster R-CNNs
You Only Look Once (YOLO)
Single Shot Detectors (SSDs)

Found a lot of online resources that help in understanding how the neural network actual works.

As building a neural network from scratch is time consuming and not entirely foolproof to get the desired efficiency.

I came across these articles Transfer learning & The art of using Pre-trained Models in Deep Learning,Transfer Learning and Transfer Learning, where we transfer the learning from an existing pre trained network to train the object which we would desire the network to detect.

All of these pre trained network have been trained on a data from either COCO or ImageNet or PASCAL VOC which contain different categories images.

Example case:

I want to train one of these to count the number of bananas here in this image

How should my training set of banana images be?

I need a fixed resolution of the image I feed into the network, so can I even feed a half banana like this for the training? I don't have to remove the background for training. Please correct me if I'm wrong.

You should read this post from the same author. It is about how to build a learning dataset with Google's search, some javascript and some python.

The idea there is to collect as many pictures of the object as possible (object = banana) and then let the deep learning network to compare and learn to differentiate them from bulk random images from some free database.

I suppose background or partial image variations are all ok. You cannot know which part of the object is visible when the whole bunch is evaluated.

Data needs to be fit to same size, at least Adobe has some tool for that.


Why are we taking the square root of the gradient in Adagrad? This is how we update weights with Adagrad:
$$w_i = w_i - \frac{lr}{\sqrt{g_i+E}}$$
where, $w_i$ is the $i^{th}$  weight, $lr$ is the learning rate, $g_i$ is the gradient of the $i^{th}$ weight at all the timesteps and $E$ is epsilon to prevent division by zero.
Now why are we taking the square root of $g_i+E$, can't we just remove the root and just divide.
I know removing the root gives bad performance, but why does it gives bad performance on removing the root?

We are taking the square root of $G_i$  because $G_i$ is the sum of squares of the gradient of weight $i$ from iterations $1$ to the current iteration. Since we squared all the gradients of all timesteps, $G_i$ was very large, and so we take its square root. Now you may think that, then why don't we just add the gradients instead of squaring the gradients, then adding. The answer to this question is that we square it to prevent negative gradients, so that negative gradients and positive gradients don't cancel each other out. Now you may think that why don't we take absolute value of the gradient then, the answer to this question is
here
That post was about root mean squared error or mean absolute error.
If you think carefully, these both issues are the same.
So, we take the square and not the absolute value. Now you may think that why don't we  power the gradient to some even power, why only square? Now, I leave this question to the consideration of the reader


How do I learn experimental methodology? When is it relevant? I just graduated in Computer Science, with a very theoretical background but without any kind of Data Science or Artificial Intelligence experience, and I working on my own to discover those two fields. More precisely, I try to work on a toy subspace clustering k-means-like algorithm, and I think I successfully learned basic optimization techniques.

So now I have some ad-hoc subspace clustering algorithm, and... I'm stuck. How do I validate it? During my studies, I studied a lot of formal proving or model checking techniques, but I feel they are totally irrelevant here. I read several AI papers, and it seems that there is a strong tradition of experimental validation in Clustering, with a lot of validity measures. My problem is that they mean nothing to me. I don't understand what they prove, or even why they are relevant. I would be very interested in a general course about the experimental validation methodology (if it actually is a thing!), if possible with theoretical justification.

Moreover, I don't really know what are the good properties people are looking for in clustering algorithms. What general theorems should I aim for? I understand this may be a question too related to what I am working on - a generic answer for k-means-like algorithms would be enough.

On the metrics. A common technique is to make simulated data with known labels. Then see if your algorithm can reproduce the clustering; if it can't there's a problem, if it can there may still be problems but you're off the starting blocks. Typical measures would include ROC and AUC (essentially how many times does the algo correctly place object i and j in the same class, balanced against false positives).

Another tactic is to use well known real datasets to benchmark against. E.g. the classic iris dataset. See how your algo performs in terms of speed and accuracy versus other algos.

As to what people are looking for this could mean a multitude of things. You might do well to consider where your algo is suited, find an application in that area, and find out whether it performs for that purpose.


How to analyse move kindness in python (Logistic regression, neural network, etc.)? With a team of researchers we were given the assignment to make a scale for the move kindness (how inviting a room or place is for exercise--e.g., a gym). In order to get objective results we were asked to make a measurement device. This device receives input through various sensors and maybe some true/false questions, and then analyze it with training information in mind. After the analysis it returns a number from, say: 1 to 10, indicating the move kindness. So a gym gets, for example an eight and a classroom a three. The problem with this is, that move kindness is very subjective, so we have conducted some surveys. For example one of the criteria is the temperature of the room/place. While the survey was being conducted we measured the temperature:

  From a scale from 1 to 10, what is your opinion about the temperature?

And then we measured the temperature. We have put all these information into some  spreadsheets:

Rating (Move Kindness): 8
Temperature: 18 degrees Celsius

At the end of this survey we asked them to give the move kindness a rating.
So we have this, for example:
Temperature: 8, 18
Light: 7, 300
Humidity: 8, 50
....
Rating (Move Kindness): 8

So my question is, what's the best way to analyse these data for a reliable measurement device using python?
We were thinking of using neural networks, because they can be trained, but logistic regression or some other machine learning algorithm is also an option. Can anyone give me some direction on this?

Okay, so from what I understand, you have a regression problem taking into account a variety of physical features. The reason I say that this is a regression problem, verses a classification problem is because the scale you are trying to predict is an ordinal scale.
There are a couple approaches to this. If your features are discriminative and linear enough, a simple least squares linear regression might work. If you believe the problem you have is to complicated for linear regressions, a simple vanilla neural network with one single output. I would recommend using the scikit-learn library in python for all models that are not neural networks. Here is a link to the generalized linear regression page.
That link has code samples and mathematical explanations. If you decide to use neural networks, and you don't have a great amount of samples or a need to use the GPU, the pyBrain library is great.
I wouldn't recommend using a logistic regression (since you mentioned it in your question), simply because a logistic regression is a classification problem, and I believe you would be better off approaching this from a regression standpoint.


Is this a valid way of training a model ? training loss in 1.1921e-07? While training a keras model, i got this logs.

Epoch 49/50
2/2 [==============================] - 2s 943ms/step - loss: 1.1921e-07 - acc: 1.0000

As per my knowledge, i know that the ideal loss should be btw 1 to 10 (not sure) while training a model, but i faced this loss value while training in e-07.

Is this a valid training? Or I'm doing wrong somewhere ??

Loss is just a sum of errors that your model makes, so lower values are desirable. In your case, the training accuracy is 1.00, meaning that your model is making perfect predictions during training. No mistakes - no loss, so this is to be expected.

However, having 100% accuracy in training is usually an indication that the model has become too familiar with the training data, effectively memorizing it instead of learning to generalize. Such a model will most likely fail to make adequate predictions on data it hasn't seen before. Unless your validation accuracy is also around 100%, then you're severely overfitting. Great explanations that will get you started on interpreting loss and accuracy can be found here.


What is the shape of the vector after it passes through the TfidfVecorizer fit_transform() method? I am trying to understand what happens inside the IDF part of the TFIDF vectorizer.

The official sci-kit Learn page says that the shape is (4,9) for a corpus of 4 documents having 9 unique features.
I get the Term Frequency (TF) part, it makes sense to me that ( for every unique feature(9), for each document(4) we calculate each term's frequency, so we get a matrix of shape (4,9)
But what doesn't make sense to me is the IDF part the formula for IDF is

So applying this formula, for every feature (9)  we calculate the log ( total no.of documents / no.of docs having the term or feature in it)
I think
This will result in a shape of (1,9) , please correct my understanding here.

The inverse document frequency transformation of TFIDF does not affect the shape of your vector. The shape is only altered during the tokenization phase which, in your case, results in a shape of (4, 9). The inverse document frequency portion only scales your existing features in accordance with how frequently these tokens arise elsewhere in your corpus. This can be thought as a one-to-one mapping of your original term-frequency vector (the tokenized corpus) to the tfidf vector (scaled using information on how frequent your vectors tokens exist elsewhere in your corpus).


Differences between pseudocolor plot and heatmap? I found some materials using the word `pseudocolor plot' and I think that it's just a heatmap. Is there any difference between them?

A heatmap is effectively a pseudocolor plot with labelled rows and columns. So, basically heatmap is a pseudocolor plot based on a pandas DataFrame rather than a matrix. Basic difference is in the plotting methods available but you can convert the matrix to dataframe and vice-versa.
Reference :

Pseudoplot
Heatmap


Regression with Neural Networks in Tensorflow problem I have recently started learning Neural networks and Python. I am trying out linear regression for a dataset with 14 features and 1 outcome. I have divided the data into training and test data. I have experimented with many parameters (learning rate, nodes per layer, number of layers, number of steps and optimization algorithm) but my test errors are as high as 150%.

I have posted my code below along with the cost curve (cost vs epochs).
Where am making a mistake and what should I change? Or can you suggest some other important checks?

import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf

# importing features and observations data for training and validation
training_filename_X = "training_set_X.csv"
training_filename_Y = "training_set_Y.csv"
test_filename_X = "test_set_X.csv"
test_filename_Y = "test_set_Y.csv"
training_features = np.loadtxt(training_filename_X, delimiter=',')
training_observations = np.loadtxt(training_filename_Y, delimiter=',')
test_features = np.loadtxt(test_filename_X, delimiter=',')
test_observations = np.loadtxt(test_filename_Y, delimiter=',')

# normalizing training data
training_features_stddev_arr = np.std(training_features, axis=0)
training_features_mean_arr = np.mean(training_features, axis=0)
normalized_training_features = (training_features-training_features_mean_arr)/training_features_stddev_arr

# normalizing validation data with training set mean and standard deviation
normalized_validation_features = (validation_features-training_features_mean_arr)/training_features_stddev_arr

# normalizing test data with training set mean and standard deviation
normalized_test_features = (test_features-training_features_mean_arr)/training_features_stddev_arr

# layer parameters
n_nodes_hl1 = 20
n_nodes_hl2 = 20
n_nodes_hl3 = 20
no_features = 14
learning_rate = 0.01
epochs = 200

cost_history = np.empty(shape=[1], dtype=float)

X = tf.placeholder(tf.float32)
Y = tf.placeholder(tf.float32)

# defining weights for each layer taken from a normal distribution with variance 2/n
hl1_weight = tf.Variable(tf.random_normal([no_features, n_nodes_hl1], stddev=np.sqrt(2/no_features)))
hl2_weight = tf.Variable(tf.random_normal([n_nodes_hl1, n_nodes_hl2], stddev=np.sqrt(2/n_nodes_hl1)))
hl3_weight = tf.Variable(tf.random_normal([n_nodes_hl2, n_nodes_hl3], stddev=np.sqrt(2/n_nodes_hl2)))
output_weight = tf.Variable(tf.random_normal([n_nodes_hl3, 1], stddev=np.sqrt(2/n_nodes_hl3)))

# defining biases for each layer
hl1_bias = tf.Variable(tf.random_uniform([n_nodes_hl1], -1.0, 1.0))
hl2_bias = tf.Variable(tf.random_uniform([n_nodes_hl2], -1.0, 1.0))
hl3_bias = tf.Variable(tf.random_uniform([n_nodes_hl3], -1.0, 1.0))
output_bias = tf.Variable(tf.random_uniform([1], -1.0, 1.0))

# defining activation functions for each layer
hl1 = tf.sigmoid(tf.matmul(X, hl1_weight) + hl1_bias)
hl2 = tf.sigmoid(tf.matmul(hl1, hl2_weight) + hl2_bias)
hl3 = tf.sigmoid(tf.matmul(hl2, hl3_weight) + hl3_bias)
output = tf.matmul(hl3, output_weight) + output_bias

# using mean squared error cost function
cost  = tf.reduce_mean(tf.square(output - Y))

# using Gradient Descent algorithm
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)

init = tf.global_variables_initializer()

# running the network
with tf.Session() as sess:
    sess.run(init)

    for step in np.arange(epochs):
        sess.run(optimizer, feed_dict={X:normalized_training_features, Y:training_observations})
        print (sess.run(cost, feed_dict={X:normalized_training_features, Y:training_observations}))
        cost_history = np.append(cost_history, sess.run(cost,feed_dict={X:normalized_training_features, Y:training_observations}))

    pred_y = sess.run(output, feed_dict={X:normalized_test_features})
    print (sess.run(output, feed_dict={X:normalized_test_features}))
    mse = tf.reduce_mean(tf.square(pred_y - test_observations))
    print("MSE: %4f" % sess.run(mse))

# plotting the cost history
plt.plot(range(len(cost_history)), cost_history)
plt.axis([0, epochs, 0, np.max(cost_history)])
plt.show()

Based on my observations

I don't see your training_observations (output) normalized. Refer
this example
I see you are using sigmoid in all your hidden layers.
       Generally tanh is used as mean of data is closer to zero, this actually makes the learning for next layer a bit easier.


How do I show plots when using Graphlab? Sorry, if this topic is not connected directly to Data Science.

I want to understand how the Graphlab tool works. Firstly I want to execute the toy examples from the Gallery site. When I try to execute the example code, everything is OK except one command: I can not see the graphlab plot after show(). The command show() returns to me some kind of object in IPython and nothing in the IPython Notebook.

If the example code has the plot, which depends directly on the matplotlib module, I can produce the real plots and save it on my machine. Consequently, I suppose the main error depends on the graphlab (or object from its class).
If somebody already used this tool and rendered the plot, can he/she tell me, how I can execute the plots command?

In [8]: import graphlab

In [9]: from IPython.display import display

        from IPython.display import Image

        graphlab.canvas.set_target('ipynb')

In [10]:import urllib

        url = 'https://s3.amazonaws.com/GraphLab-Datasets/americanMovies   /freebase_performances.csv'

        urllib.urlretrieve(url, filename='freebase_performances.csv')  # downloads an 8MB file to the working directory

Out[10]: ('freebase_performances.csv', <httplib.HTTPMessage instance at 0x7f44e153cf38>)

In [11]: data = graphlab.SFrame.read_csv('remote://freebase_performances.csv', column_type_hints={'year': int})

...
...
...

In [15]:data.show()

No plot after this line
...
...

In [19]:print data.show()

<IPython.core.display.Javascript object at 0x7f44e14c0850>

The object of graphlab (?) after print command

I have found some solution and will post it here, because somebody, who works with graphlab, can have the same question.

We can look at the example here: Six degrees of Kevin Bacon

At te beginning of the program execution you need to run next command:

graphlab.canvas.set_target('ipynb')

Exactly this is a key of the whole problem (at least by me:-)

At the beginning it is important to know, which parameter of set_target() command you want to execute. You can use two options for argument of this command:  'ipynb' (which is executed direct in iPython Notebook, like in example) or 'browser' (which open the new window with the plots)
On my machine 64-bit, Ubuntu, I can not use the command 'ipynb'. Only the 'browser'-command get me the plot back. I don't think, it is necessary to change https to http, but you can do it anyway. We have also the other machine by us (32-bit, Ubuntu) and it executes the other command 'ipynb', but not 'browser' (without to change https to http)


Unsupervised Learning and Training Data As far as I know, we need to use training data to find out the relation between the features, also known as input values, and labels, that are output values, in supervised learning. After that, by using this relation, our learning system tries to predict labels of data samples in next data sets.

However, there is no need to find out such a relation in unsupervised learning because the data samples do not have labels; they only consist of features. In this case, do we need a training set in unsupervised learning ?

In unsupervised learning,the learning procedure is finding similarity between training samples, and putting similar items into a same cluster,
training phase in unsupervised learning produce some sets with similar items.
Then,in test phase the similarity is calculated for all items of each set, and check if the test item is similar to each cluster's items or not, if it was similar to at least one item, the test item belongs to that cluster.

SO... YES, we need training set and test set, the training set helps to find thresholds to find similarity


Am I supposed to scale binary features alongside other numerical features? When I am cleaning my data, I have some features which contain large numbers and some features that are binary. Should I scale the large features and then add the binary columns or just scale them all together?

My fear is that scaling them all together makes the binary features seem less important than they really are.

Note: I am prepping a neural network for binary classification. I am using a sigmoid output and scaling my features from [0,1]

There are two important concepts that you need to understand.

Scaling your features won't affect their "importance" in your neural network

Intuitively, your neural network will itself learn which feature is important or not by learning weights.

Scaling your features will speed up your convergence and will limit the risks of overshooting or being stuck in a local optimum. It also makes numerical sense.

Intuitively, scaling features can make your training faster because your "path to convergence" will probably be shorter if your features are scaled (right picture).

It also makes numerical sense to scale your features because if you have very large and very small values, some of your weights might drop very low and this could cause some numerical issues and hinder the performance of your model.


Dataset : Handling rows where some column value is not applicable I am working on some LTE Data say of shape (100000, 10) .

Now suppose I have a column Jitter Avg. Now Jitter occurs only for any connected call and for call which could not be setup due to some reason Jitter is not applicable.

So for those rows representing failed called setup there is no value for Jitter i.e NaN. I cannot replace it with 0 since that means good quality. All in all any value would not be applicable here.

So how to deal with such situation.

You can use a model that can handle NaN values. Most commonly this would be something based on decision trees. I recommend LGBM and XGBoost.

Decision trees also work well when these kind of values are set outside of the range for this feature, i.e. setting it to something like -999. This alternative is better if you want to use models from scikit-learn since NaNs are not supported there.


How do Classification Algorithms such as Catboost and Random Forest parse test data? I would like to know how classification works with the algorithms listed above. My specific question is this, say I have a high signal continuous feature which has a certain distribution and I train a model according to some training data and it finds the best split for that feature. When I use the model on test data, would it split according to a specific number or by distribution? i.e if the number '10' provides the best split for the training data which happens to be 75th percentile. When it comes to test data will it split according to the percentile or the number 10? I hope that is clear.

Both the models i.e. RandomForest/CatBoost are based on a Decision Tree.
A Tree splits on feature values. Check this Tree for Iris dataset. Scikit-Learn