How does Performance function classify predictions as positive or negative? Package:ROCR I'm performing a logistic regression on my training data. I used the glm function to get the model m. Now using the below codes from this link, I calculated AUC

  $test\$score<-predict(m,type = 'response',test)$
  $pred <- prediction(test\$score,test\$good_bad)$
  $perf <- performance(pred,"tpr","fpr")$

where score is the dependent variable (0 or 1).
To score the tpr (True positive rate) and fpr (False positive rate), you have to classify the predicted probabilities into 1 or 0.
What is the cutoff used for that? how can we change it?

Could not find anything useful in this main documentation as well.

I cant access an R console at the moment to check, but I'm quite certain the cutoff is 0.5: if your glm model does prediction, it first produces real values and then applies the link function on top. To the best of my knowledge, you can't change it inside the glm function, so your best bet is probably to check ROC, find what the optimal threshold is and use that as cutoff.


How to get sentence from embedding vector with Universal Sentence Encoder? I'd like to ask, if there is possibility to get sentence (or word) from embedding vector using Universal Sentence Encoder?

First of all, I've clustered my embedded sentences and I've got a vector which is representing center of the cluster, and now I want to convert this to some sentence which will have the best meaning for this specific cluster.

Best regards.

You cannot run the universal sentence encoder in reverse. There is no practical way to take an arbitrary embedding vector and get a sentence.

My suggestion would instead be to find the sentence in your data with the embedding closest to your center. Euclidean distance works well, specially if you used K-means or another euclidean method to create your clusters.


SGD data should be randomly selected or sequentially feed? When training neural network, is it better to randomly choose data for every batch or feed the data sequentially? Why?

If you feed the data randomly, there are fewer chances of overfitting than if you feed the data sequentially.

If the data is fed sequentially, then the network will not learn to generalize as the weights will be updated for that batch according to that same image again and again.

It is always recommended to feed the randomly.


Is there an R build tool like Maven? Is there an R build tool (like Maven or Gradle for Java) to get the dependencies and package an R project?

The packrat package is what you're looking for.

It bypasses R's native packages and allows you to build and deploy a bundle of packages and dependencies.

However, it does not do Maven-style dynamic dependency resolution; you need to specify a list of packages that you believe will work together, then the bundle is installed/deployed as a single unit.


Does Gradient Boosting perform n-ary splits where n > 2? I wonder whether algorithms such as GBM, XGBoost, CatBoost, and LightGBM perform more than two splits at a node in the decision trees? Can a node be split into 3 or more branches instead of merely binary splits? Can more than one feature be used in deciding how to split a node? Can a feature be re-used in splitting a descendant node?

Gradient boosting can be applied to any base model, so doing it with a Quinlan-family decision tree (which allow for such higher-arity splits for categorical features) should make this possible.  However, all implementations of gradient boosted trees that I know of (and certainly XGBoost, CatBoost, LightGBM) all use CART as their tree model, so you won't get anything but binary trees.  (These GBMs do modify CART a little, e.g. in using histogram binning to reduce the split searches, but nothing as drastic as n-ary splits for categoricals.)


Why is the cost increasing in the linear regression method? I am implementing the linear regression model from scratch.

#!/usr/bin/env python2
# -*- coding: utf-8 -*-
"""
Created on Thu Nov 16 14:40:53 2017

@author: user
"""

import os
import random
os.chdir('/home/user/Desktop/andrewng/machine-learning-ex1/ex1')

import pandas as pd
data = pd.read_csv('/home/user/Desktop/andrewng/machine-learning-ex1/ex1/ex1data1.txt',header=None)

theta_0 = random.random()
theta_1 = random.random()
alpha = 0.001

print(len(data))

hist = -90
cost = 0
print('theta_0    + theta_1    ')
while(cost-hist>0.001):
    hist = cost
    cost = 0
    a = 0
    b = 0
    for i in range(len(data)):
        k = data.iloc[i]
        a = a +  theta_0 + theta_1*k[0] - k[1]
        b = b + (theta_0 + theta_1*k[0] - k[1])*k[0]
    theta_0 = theta_0 - alpha*a/len(data)
    theta_1 = theta_1 - alpha*b/len(data)
    #print(str(theta_0)+'    '+str(theta_1))
    for j in range(len(data)):
        k = data.iloc[i]
        cost  = cost + (theta_0 + theta_1*k[0] - k[1])**2
    cost = cost/(2*len(data))
    print(cost)
    #if(cost>hist):
    #    print(str(theta_0)+'    '+str(theta_1))
    #    break
print(str(theta_0)+'    '+str(theta_1))

According to the theory the cost should decrease at each iteration but for me the cost keeps increasing.

(DATA)

The reason is your learning rate alpha is too large for this optimization problem. Start with a really small value (< 0.000001) and you will observe a decrease in your cost function.

Keep in mind that when the learning rate is too large, the gradient descent algorithm will miss the global minimum (global because MSE cost function is convex) and will diverge. This is why you have observed still increasing cost function values with alpha = 0.001.


Training Examples used in Stochastic Gradient Descent Hi I was reading the difference between GD and SGD and found the below link.

[What is the difference between Gradient Descent and Stochastic Gradient Descent?

Based on this information I wanted to understand how would SGD train in the below scenario :

Say we have a dataset having 10000 rows and 45 predictors. Now, since SGD trains each predictor on one example(1 from 10000) , does that mean it uses only 45 examples in total to train the 45 predictors?

I'd really appreciate a clear explanation of this scenario.

Thanks!

In a given iteration of the stochastic gradient descent algorithm, all 45 predictors are updated using a randomly generated subset of your 10,000 observation sample.  This subset may consist of only 1 observation, but typically cross validation is used to determine the optimal subset size.  You could even try randomly generating different subset sizes each iteration.


Keras OOM for data validation using GPU I'm trying to run a deep model using GPU and seems Keras running the validation against the whole validation data set in one batch instead of validating in many batches and that's causing out of memory problem
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[160000,64,64,1] and type double on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:GatherV2]
I did not have this problem when I was running on CPU, it's just happening when I'm running on GPU, my fit code looks like this
history =model.fit(patches_imgs_train,patches_masks_train, batch_size=8, epochs=10,
  shuffle=True, verbose=1, validation_split=0.2)

When I delete the validation parameter from the fit method the code works, but I need the validation.

There may be two causes to your problem:

In validation the network needs more memory.
There is another problem not directly related to this.

In this Keras issue you can find a discussion of a very similar problem. Basically, you can try:

Reducing the batch size.
If you are using Tensorboad, try disabling it or setting its batch_size parameter.


What is Word2Vec approach I am new to Data Science and I am trying to understand the 'Word2Vec' approach from a long time. Can someone please explain it in simple terms? Moreover, what are the problems that can be solved through 'Word2Vec' approach?

What is Word2Vec?

Word2Vec is an implementation of word embedding techniques.

Word embedding tries to represent relationships that may exist between the individual words (those contained in your processing texts) by giving them each a vector with same predefined dimension. In this vector space words that share common contexts may be located closer. How to assign the words into the vector space? This work has often been done through neural network training.

What problems can it solve?

The main problem solved by Word2Vec (or word embedding) is that it creates a way to represent relationships between the words in your processing texts, as opposed to just treat the words as individual symbols. This makes any following data mining or machine learning more effective.

Besides Word2Vec, you can also use many other pre-trained embeddings like GloVe. If the dataset is large, it is better to train your own word embedding which will get a better performance than using pre-trained embeddings.


Deep learning performance on classifying simple geometric figures I'm trying to find papers or just performance data on classifying simple geometric shapes, e.g. the first six convex regular polygons. The input data is computer graphics generated, producing high contrast, clean images of convex regular polygons in various scales and orientations. One polygon per image. There is no limit, other than reasonable CPU time, on number of training images used. Testing is also done on randomly generated CG images.

Regular polygons are just an example. You can add circles, ellipses, non-regular polygons, etc. Basically any simple geometric shape, which a competent elementary school student can easily classify with 100% accuracy.

A CNN network should easily achieve close to 100% accuracy on this task. Few aspects make this task easier :

Clean High-contrast images generated from program
Only 1 shape per image
Given few images of a shape, libraries can generate thousands of training samples automatically with various transformations like rotations / scaling / flip
Within few layers network should identify edge-detection patterns for vertices and learn the 3 vertices = triangle and so on

For example, this code can generate training examples based on few input images :

https://keras.io/preprocessing/image/

datagen = ImageDataGenerator(
    featurewise_center=True,
    featurewise_std_normalization=True,
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True)

Few articles :

https://www.researchgate.net/publication/312328148_Binary_shape_classification_using_Convolutional_Neural_Networks

https://towardsdatascience.com/object-detection-with-neural-networks-a4e2c46b4491


Relating two data sets at the row level where table one may correlate to zero or more rows in another I'm just looking for some high level recommendations on libraries, design patterns, or algorithms here. I obviously don't expect you to build a model for me.

I would like to predict the monetary impact that an outage will have on the electric grid.

I have data of historical outages that I could train a model on. This basically consists of the equipment-ID that was on outage, with a start and end timestamp.

An outage may cause zero or more constraints. A constraint is where a line becomes loaded to the extent that power must be flowed an alternate way. I already have data on which constraint-IDs occurred at what start and end timestamp, as well as the economic impact of that particular occurrence of that constraint. Note that a constraint may exist without any outages, so there is a bit of a loose relationship. Also, outages may occur without any constraints occurring, or it may take a combination of outages to cause a particular constraint. Generally speaking though, there is a moderate to strong relationship between outages and constraints.

I presume that there should be an algorithm of sorts, that can deduce which outages generally correlate to what constraints after they have been seen to overlap throughout history.

To recap, an outage may cause a constraint. I would like to feed the model outage data and constraint data, so that if I have planned outage data, it can predict if zero or more constraints are X% likely to occur.

I prefer to use Python, but am open to R.

How would you go about modeling a problem like this? In simplified terms, how do you correlate a row in one table to zero or more rows in another where there is no direct mapping? Most solutions I have seen correlate at the column level as opposed to the row.

Edit:

After some thinking about this, I've come up with what I think is a solution. Make each row with a timestamp for that particular hour of the day. One hot encode all outages for that hour. Also, create a column for each potential constraint, with the monetary impact as the value (Scaled to a 0-1 scale).

The thing that seems crazy to me, is that this data set is going to end up being thousands of columns wide if I do it this way. Are super-wide data sets like this common in ML? - or, is there a better approach?

I would like to predict the monetary impact that an outage will have on the electric grid.

It may help to think of your problem as two problems:

Given an outage (or set of outages), predict the number of constraints that will result
Given a constraint, predict the monetary impact

You would need to build two regressive models - one for each of the problems.  It's difficult to give advice about which specific techniques or algorithms to use without a better understanding of the domain and of your dataset.

It sounds like you have a set of timestamped historical outages and a set of timestamped historical constraints.  To me, your description implied that there was no authoritative mapping between outages and constraints, so the first wrinkle may be assigning blame for each outage to one or more constraints.  This likely requires domain knowledge that most SO contributors won't have.

Another wrinkle is that you only have date and time information about outages and constraints.  If the temporal information explains a lot of the variance in number of outages and cost of those outages, then no problem.  Otherwise, it's going to be difficult to build accurate models.

If you go with the approach I suggested above, then you have a timeseries forecasting problem on your hands.  Traditional algorithms for this type of problem include auto-regressive models like ARMA, ARIMA, SARIMA, etc.  You can read up on those to see if they seem suitable for your problem.

Alternatively, Facebook recently released an open-source forecasting library Prophet designed to be easy-to-use and to work well out-of-the-box.  It's super easy to try out, so you could give it a shot.


How to import image data into python for keras? I'm new to CNNs, starting off with keras. I'm currently using ImageDataGenerator to import my train/validation folders (which each have 2 class subfolders for my binary classification task). Was wondering how can I import my train/validation files without using ImageDataGenerator? I'm aware that ImageDataGenerator is good for accuracy as it does some augmentation, but I want to compare the accuracy to a training set without any augmentations. Essentially I think I need to put all the images into an array, but not sure how to.

Basically I want to know what is the normal way to import training/validation data for images, so I can compare what is the accuracy difference with/without imagedatagen. I know with normal NN tasks it's easy as you can just do pd.read_csv().

I'm currently importing like so:

train_datagen = ImageDataGenerator(
        rescale=1./255,
        shear_range=0.2,
        zoom_range=0.2,
        horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
        'data/train',
        target_size=(150, 150),
        batch_size=32,
        class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
        'data/validation',
        target_size=(150, 150),
        batch_size=32,
        class_mode='binary')

The docs for ImageDataGenerator suggest that no augmentation is done by default. So you could instantiate it without any augmentation parameters and keep the rest of your code for handling your directory structure:

train_datagen = ImageDataGenerator(rescale=1./255)

You are also allowed to write your own custom data generator and pass it to model.fit_generator(). Here is a nice tutorial.

Or if your data fits in memory you could write some simpler code possibly using keras.preprocessing.image.load_img to load all the images into an array and pass them to model.fit instead.


How to use a dataset with only one category of data I am performing a classification task, to try to detect an object. A picture of the environment is taken, candidates are generated of this possible object using vision algorithms, and once isolated, these candidates will be passed through a CNN for the final decision on whether the object has been detected or not. I am attempting to use transfer learning on InceptionV3 but am having difficulty training it, as I only have one set/class of images.
The dilemma is that I only have one class of data and when I pass it through the network, I get a 100% accuracy (because there is nothing to compare it to). How should I overcome this? Should I find more categories online to add to my dataset? What should these categories be?
Just to clarify, as an example, I have class "cat".
Not "cat" and "dog".
Not "cat" and "no cat".
Just "cat". That is what my dataset consists of at the moment.

The Model learns to match the weights as per the image and feedback from label data.
If you will feed a few Image classes as "Not Cat", it will learn to classify similar features as "Not Cat". But might fail for a new Class.
 e.g. if it is trained on "Car/Furniture/Dog" as "Not Cat", then chances are high that a Wild Cat will be classified as Cat.
 Dumping all the Imagenet dataset will definitely provide quite a good variance to the "Not Cat" class and may work most of the time but that is not the appropriate solution for the problem.
Such type of problem will fall under One-Class-Classification.
Core idea is to use CNN to extract features then use some specialized models e.g. one-class SVM, Gaussian Mixtures, etc. to define a boundary for "Cat"

This problem, as defined by the one-class SVM approach, consists of identifying a sphere enclosing all (or the most) of the data. The classical strategy to solve the problem considers a simultaneous estimation of both the center and the radius of the sphere.

You may start with these links(In the specified order) -
Hackernoon blog
Arxiv
Researchgate
There are other approaches too i.e. based on Auto-encoder. Here, we try to put a Threshold on reconstruction error.
References-
Quora
SO
Keras blog
Also, may look here to check an idea which generated Random images for "No Cat" Class.Here


Spark DataFrame courses If I understand the Databricks philosophy correctly, Spark will soon be heavily moving toward dataframes, i.e. away from the usual map/reduce on RDDs. I was wondering if there are any good suggestions for online courses or books that introduce Spark from the dataframe point of view? I know Databricks has a good number of resources on dataframes but I would really like to see some more detailed courses.

edX offers some spark courses. Try the following:

https://www.edx.org/course/introduction-apache-spark-uc-berkeleyx-cs105x?gclid=CjwKEAjwm8-6BRDgnb-Dk96UmRASJADbMycYmMKC3xWs_3jGV9pUd0k3gxIbQRHE2H4_hlXeNMw1BhoCqHjw_wcB

I believe this is through a partnership with Databricks


Recommender System: how to treat different events I'm trying to build recommender based on user history from e-commerce. There are two(potentially more) types of events: purchase and view.
Is it okay to sum up number of purchases and views for a given item(with purchase and view having different weights)? Or I`ll just mix up user intent this way?

I assume you're using an implicit feedback recommender approach like ALS. Otherwise, summing data points generally won't make sense, such as if you're feeding it to a recommender that expects ratings.

The input to implicit ALS is, conceptually, weighted user-item pairs. Therefore it makes sense to perhaps use a sum of user-item clicks as the weight. Summing makes sense.

However, does a purchase and view seem to carry the same weight? obviously not. A purchase is a much stronger association and should be weighted accordingly.

As to how much, I'd suggest weighting purchases simply by price, to start. Then weight clicks by price times purchase-to-click ratio. A $10 item purchase is weight 10; if 1 in 200 clicks results in a purchase, then weight a click 0.2.

This is crude but probably about as close as anything for capturing this info in the context of ALS.


Why Root Finding is important in Logistic Regression? (i.e. Newton Raphson) I'd like to ask what is the main reason why we find the roots in logistic regression (i.e. why we use Newton Raphson method on logistic regression ). I understand the basics of Newton Raphson method, but I just can't understand what is the importance of finding the roots or using second derivatives.

P.S. I know the idea of Newton Raphson, I am wondering why do we need to use this method and find the zero's or roots of function? What does it want to tell us for logistic regression for instance?

This blogpost gives a broad answer to your question. In short, Newton's method is not used to find a root of the loss, but a root of the gradient. If you find a root of the gradient, then you are either in a maximum, a minimum, or a saddle point (the three of them are critical points). When using the cross-entropy loss function in logistic regression, it can be proved that this loss function has only one critical point, and this critical point corresponds to the global minimum. For this reason, finding a root of the gradient is equivalent to finding the only critical point, therefore the global minimum of the cross-entropy loss function.


Keras input dimension bug? Keras has a problem with the input dimension. My first layer looks like this:

model.add(Dense(128, batch_size=1, input_shape=(150,), kernel_initializer="he_uniform", kernel_regularizer=regularizers.l2(0.01), activation="elu"))

As you can see the input dimension should be (150,) and with the fixed batch_size it is (1, 150)

My data has dimension (150,) and could be for example a numpy array with 150 zeros.

old_qval = model.predict(old_state_m)

Here I call the model to make a prediction. Normally Keras should automatically add the batch size as an extra dimension so I should end up with (1, 150) which would work. But Keras adds the dimension for the batch size at the wrong place and I end up with (150, 1). I tried tensorflow and theano backend.

Do I have a bug in my code or is it a problem with Keras?

How can I fix the problem? I could reshape my input data but it already has the needed shape of (150,) and should be fine. What else could I do?

If I should provide more data or code feel free to ask.

When you do model.predict(X) the first axis in X is always an index in a batch. So if you want to predict on one sample, do something like X = np.expand_dims(X, axis=0)

In your case, this should work:

old_qval = model.predict( np.expand_dims(old_state_m, axis=0) )


Keras: Dimension Error with Sparse Categorical Crossentropy I'm trying to make an NN that, given the time on the clock, would try to predict which class (out of 32 in this example) is making a request to the system. As a first attempt, I've tried to use categorical_crossentropy, but this will obviously not work because the targets are very sparse, so the system will be heavily rewarded by just always predicting the non-requests.

Now I'm trying to use sparse_categorical_crossentropy, but I keep getting a dimension mismatch error (the train and test sets are the same in this case because I just wanted to evaluate performace in the training set at first):

Error when checking target: expected dense_90 to have shape (1,) but got array with shape (32,)

The DataFrame is here (a simple clock and another column for the requests) and the code is:

from keras.models import Sequential
from keras.layers import Dense
from keras.utils import to_categorical
import tensorflow as tf

requests = df['requests'].values
requests_cat = to_categorical(requests, 32)

length = len(df['clock'])
train = np.reshape(df['clock'].values, (length, 1))
train = train.astype(np.int)
target = requests_cat

model = Sequential()
model.add(Dense(25, activation = 'relu', input_shape = (train.shape[1],)))
model.add(Dense(25, activation = 'relu'))
model.add(Dense(32, activation = 'softmax'))

model.compile(optimizer = 'adam', loss = 'sparse_categorical_crossentropy')
model.fit(x = train, y = target, epochs = 100, validation_data = (train, target))

On a sidenote:

This architecture doesn't seem to be the best in this case. As a second prototype I was thinking about doing something with an LSTM, since past requests can affect later ones. Is there a standard architecture for scheduling?
What would be the proper way of splitting sparse sets into training and testing ones?

I thing you've misunderstood what the difference between categorical_crossentropy and sparse_categorical_crossentropy is. The sparse part doesn't refer to the sparsity of the data but the format of the labels.

If your labels are one-hot encoded: use categorical_crossentropy
If your labels are encoded as integers: use sparse_categorical_crossentropy


Python inserts newline by writing to csv I am trying to scrape http://www.the-numbers.com/movie/budgets/all

but when I write the table into a csv file, there is an additional line with the counter index written in between each movie row... how can I get rid of this? I dont understand how that counter line is being written to the csv...

import csv,os
from bs4 import BeautifulSoup
from urllib.request import Request, urlopen, URLError
from selenium import webdriver

counter = 0
currentDir=os.getcwd()
filename = currentDir + "\\theNumbersScraper.csv"

pagecount = 1
headers=['ID',
 'Release Date',
 'Movie',
 'Production Budget',
 'Domestic Gross',
 'Worldwide Gross']

with open(filename, 'w' ,newline='\n',encoding='utf-8') as csvfile:
        #writer = csv.DictWriter(csvfile, fieldnames=dictionary)#write headers
        #writer.writeheader()
        #csvfile = open(filename, 'w', newline='',encoding='utf-8')
    writer = csv.writer(csvfile,delimiter='|')
    writer.writerow(headers)

#with open(filename, 'a', newline='',encoding='utf-8') as csvfile:
    #writer = csv.DictWriter(csvfile, fieldnames=dictionary)#write headers
    #writer.writeheader()
    #csvfile = open(filename, 'w', newline='',encoding='utf-8')
#    writer = csv.writer(csvfile,delimiter='|')

    while pagecount <5401:

        """
        #movie-entries go from http://www.the-numbers.com/movie/budgets/all/1
        #to http://www.the-numbers.com/movie/budgets/all/5401
        #so there are 5400 entries
        """
        request = Request("http://www.the-numbers.com/movie/budgets/all/"+str(pagecount))
        request.add_header('User-agent', 'wswp')
        website = urlopen(request).read().strip()

        soup = BeautifulSoup(website,'lxml')

        """#obsolete
        headertags = soup.find("table").find_next("tr").find_all("th")

        headers= []
        for line in headertags:
            headers.append(line.string)

        headers[0] = 'ID'

        """

        #movie-entries go from http://www.the-numbers.com/movie/budgets/all/1
        #to http://www.the-numbers.com/movie/budgets/all/5401
        #so there are 5400 entries

        all_tr = soup.find_all("tr")
        for movie in range(1, len(all_tr)):
            row=[]
            counter+=1
            row.append(counter)
            td = all_tr[movie].find_all("td")
            for colIndex in range(1, len(td)):
                row.append(td[colIndex].string)
            writer.writerow(row)

        """
        for tr in all_tr:
            row = []
            td = tr.find_all("td")
            for i in range(1, 4):
                row.append(str(counter))
                row.append(td[i].string)
            writer.writerow(row)
            counter+=1
        """
        pagecount +=100

csvfile.close()

The part of interest is actually this block only:

all_tr = soup.find_all("tr")
for movie in range(1, len(all_tr)):
    row=[]
    counter+=1
    row.append(counter)
    td = all_tr[movie].find_all("td")
    for colIndex in range(1, len(td)):
        row.append(td[colIndex].string)
    writer.writerow(row)

The output is:

ID|Release Date|Movie|Production Budget|Domestic Gross|Worldwide Gross
1|12/18/2009|Avatar|$425,000,000|$760,507,625|$2,783,918,982
2
3|5/20/2011|Pirates of the Caribbean: On Stranger      Tides|$410,600,000|$241,063,875|$1,045,663,875
4
5|5/1/2015|Avengers: Age of Ultron|$330,600,000|$459,005,868|$1,408,218,722
6

and so on, with additional counter lines between the output that I don't want, how can I get rid of that? (and where does it come from?)

Check the source code of http://www.the-numbers.com/movie/budgets/all/5401

In that table, every second row is an empty <tr> tag.


What is difference between intersection over union (IoU) and intersection over bounding box (IoBB)? Can someone give a detailed explanation IoU and IoBB along with that the differences between them.

The Intersection over Bounding Box is the Intersection over Union (IoU) for object detection tasks, where you have a bounding box.

There are many tasks (e.g. image segmentation) where you have an IoU (the predicted segment vs the actual segment), but there are no bounding boxes.


default kernels in cnn in keras? what are the default kernels used in convolution done in cnn

for example in this code of keras there are 32 filters of size 5x5 each.

my question is 1) What will be the different default kernels in those 32 filters.
for example in an tutorial I read that the
\begin{bmatrix}
    1 & 1 \\
    -1       & -1\\
\end{bmatrix} could be taken as horizontal edge detector and
\begin{bmatrix}
1& -1\\
1 & -1
\end{bmatrix}
could be taken as vertical edge detector .

def baseline_model():
    # create model
    model = Sequential()
    model.add(Conv2D(32, (5, 5), input_shape=(1, 28, 28), activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Conv2D(32, (5, 5), activation='relu'))
    model.add(Dropout(0.2))
    model.add(Flatten())
    model.add(Dense(128, activation='relu'))
    model.add(Dense(num_classes, activation='softmax'))
    # Compile model
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model

The initial value of the CNN kernels can be seen from the documentation found here.

The default Conv2D layer looks like

keras.layers.Conv2D(32, (5, 5), activation='relu', kernel_initializer='glorot_uniform', bias_initializer='zeros')

So we see that the kernel weights are initialized by Glorot uniform method an dthe bias is initialized as all zeros.

Glorot Uniform

Uniform sampling from $[-limit, limit]$ where

$limit = \sqrt{\frac{6}{fan\_in + fan\_out}}$

where $fan\_in$ is the number of input neurons to the weight tensor and $fan\_out$ is the number of output neurons of the weight tensor.


Predict how many days late or early someone will finish their work So I have a set of deadlines and people, with a database of when those people finished their previous work and how much after the deadline it was, as well as when the work was given. The work itself were articles, so I also have the word count for each. How do you, based on the previous data, calculate the amount of days earlier or later somebody will most probably finish their work?

As a concrete example of the problem I'm trying to solve:

John finished his last 5 projects 5,4,3,6,2 days late. What is the most probable amount of days earlier or late he will finish his work?

Basically I'm looking for an appropriate machine learning algortihm to implement to calculate this probable end date.

If we assume that each task delivery is independent of eachother, and the process does not change a lot over time (stationary), we can treat this as a standard regression problem.

Since this is about deadlines, we expect that there might be variations over time, or patterns of delay across the seasons of the year or week. So time-based features might look something like:

|deadline_year|deadline_week_number|deadline_day_of_week|

We also expect that the size of a delay might depend on the size of the task. So if you have the start date, or an estimate on number of days, definitely include that. If people can have multiple tasks at the same time, include that also.

|workdays_between_start_and_deadline|workdays_estimated|concurrent_tasks|

And we expect that delays may depend on the person who performs the task, and who created the task.

|task_owner|task_creator|

Use Exploratory Data Analysis and your knowledge about the processes that created to find more of these possible relationships. Use scatterplots of each feature against the target days_delayed (negative=before time, 0=on time).

One can start with a strong non-linear model like RandomForest. This can give estimates which can be scored (by mean squared error for example), and indicate whether your features are predictive or not.
To get probability intervals, you can use a Bayesian model such as Bayesian Ridge Regression. This is a linear model, so may have to spend more time on feature engineering to make the relationships between feature and target (roughly) linear.


What does GlobalMaxPooling1D() do to output of LSTM unit in Keras? The keras model looks like this

features_input = Input(shape=(features.shape[1],))
inp = Input(shape=(maxlen, ))
x = Embedding(max_features, embed_size, weights=[embedding_matrix], trainable=False)(inp)
x = Bidirectional(LSTM(num_filters, return_sequences=True))(x)
max_pool = GlobalMaxPooling1D()(x)
x = concatenate([x_h, max_pool,features_input])
outp = Dense(6, activation="sigmoid")(x)

What does GlobalMaxPooling1D()(x) really do to the output of LSTM?
I know the input to LSTM layer is of dimension (batch_size, steps, features).

Does GlobalMaxPooling1D take max across num_filters/hidden units of each LSTM unit?

No, it takes max across steps.

Suppose the output x of your Bidirectional(LSTM()) has shape  (batch_size, steps, hidden_size), then after GlobalMaxPooling1D() your max_pool would have shape (batch_size, hidden_size).


Keras multi-gpu batch normalization 1) How does the batch normalization layer work with multi_gpu_model?

Is it calculated separately on each GPU, or is somehow synchronized between GPUs?

2) Which batch normalization parameters are saved when saving a model? (Since when using multiple-gpus in Keras, the original model must be saved, as suggested here)?

In the docs of multi_gpu_model is says:

  Specifically, this function implements single-machine multi-GPU data
  parallelism.

What does it mean for Batch Normalization?

1) How does batch normalization layer work with multi_gpu_model?

For N GPUs, there are N copies of model, one on each GPU. For each copy, forward and backward passes are executed for a sub-batch (each sub-batch is 1/Nth of a batch). This means, batch normalization is actually sub-batch normalization, there is no access to the rest of batch.

# This `fit` call will be distributed on 8 GPUs.
# Since the batch size is 256, each GPU will process 32 samples.
parallel_model.fit(x, y, epochs=20, batch_size=256)

  2) Which batch normalization parameters are saved when saving a model
  (since when using multiple-gpus in Keras original model must be saved,
  as suggested here)?

One unified (template) weight is saved instead of N different weights. Each GPU calculates a different gradient due to receiving a different sub-batch. Then, either (1) weights are updated separately on each GPU and become synchronized periodically, or (2) N outputs/gradients are aggregated on the template model (on CPU), and then new weights are broadcasted back to GPUs. In both cases, there is a unified (template) model, although in the first case, the template model may not always have the latest weights since it is kept out of synchronization and gets updated occasionally.

Extra remarks

Some researchers have proposed a specific synchronizing technique for batch normalization to utilize the whole batch instead of a sub-batch. They state:

  Standard Implementations of BN in public frameworks (suck as Caffe,
  MXNet, Torch, TF, PyTorch) are unsynchronized, which means that the
  data are normalized within each GPU.

Because of weights synchronization we cannot expect a linear speed-up w.r.t. number of GPUs.
Some technicalities of multi_gpu_model are discussed in this github issue. multi_gpu_model has a speed gain when weights are sparse (in comparison to Dense layers), otherwise  weights synchronization becomes a bottleneck.
Also, here is an example of GPU-GPU weight synchronization flow from Nvidia:


Loss and val loss too big I have a random vector with known pdf x and i'm trying to predict some behaviour y. I created a dataset containing (X,y), where X is a vector of D realizations of the VA X and y = (y1,y2). That is, for each instance I have a vector D of input and the vector y for output. I tried:

 from tensorflow import keras
 model = keras.models.Sequential([
 keras.layers.Dense(300, input_shape=(D,), activation="relu"),
 keras.layers.Dense(300, input_shape=(D,), activation="relu"),
 keras.layers.Dense(100, input_shape=(D,), activation="relu"),
 #keras.layers.Dense(10, activation="relu")
 keras.layers.Dense(2)
 ])

 model.summary()
 model.compile(loss='mean_squared_error', optimizer="adam")
 history = model.fit(X_train, y_train, epochs=30, validation_data=     (X_val, y_val))

And in training:

 Epoch 2/252 \\
 252/252 [==============================] - 0s 965us/sample - loss: 67222078.7937 - val_loss: 99721252.0000
 Epoch 2/252
 252/252 [==============================] - 0s 325us/sample - loss: 66736292.6984 - val_loss: 99545232.0000
 Epoch 3/252
 252/252 [==============================] - 0s 246us/sample - loss: 64592091.8095 - val_loss: 99156788.0000
 ...
 Epoch 252/252
 252/252 [==============================] - 0s 305us/sample - loss: 23.6345 - val_loss: 92329540.0000
 80/80 [==============================] - 0s 143us/sample - loss: 29423635.9000

Loss and val_loss is too bad. Can anyone give me an insight?

It could be normal to have such large loss value if your predicted values are far from the ground truth values (because of the square in the mean squared error loss).
Try to print the predicted values and ground truth with tf.Print to check that.
You can normalize your ground truth values so that their range is not too big. For example it should be easier for the network to predict values between 0 and 10 than values between 0 and 1000.


Switching Keras backend Tensorflow to GPU I use Keras-Tensorflow combo installed with CPU option (it was said to be more robust), but now I'd like to try it with GPU-version. Is there a convenient way to switch? Or shall I re-install fully Tensorflow? Is the GPU version reliable?

I suggest reinstalling the GPU version of Tensorflow, although you can install both version of Tensorflow via virtualenv. GPU version of Tensorflow supports CPU computation, you can switch to CPU easily:

with device('/cpu:0'):
    # your code here

I have been using GPU version of Tensorflow on my Tesla K80 for a few months, it works like a charm. Feel free to have a try!


Need help understanding Hard SVM quadratic program equation This is from the textbook "Understanding Machine Learning" by Shalev-Schwarz p. 169.

Can anyone help me understand why the solutions to this optimization problem need to be divided by the norm of w?
(Other resources online mention this translation to quadratic programming, but omit the division part.)
Thank you very much!!

You want to maximize the margin, so that all points of each class are as far as possible into their halfspace. Mathematically this implies your equation: $$y_i\left(\langle\mathbf{w},\mathbf{x_i}\rangle+b\right) > 1   \;\;\;\;\;  \forall_i$$
Let's think about what this actually means.
If you divide both sides of the inequality by $||\mathbf{w}||$ you get:
$$y_i\left(\langle\frac{\mathbf{w}}{||\mathbf{w}||},\mathbf{x_i}\rangle+\frac{\mathbf{b}}{||\mathbf{w}||}\right) > 1/||\mathbf{w}||   \;\;\;\;\;  \forall_i$$
The expression:
$$\langle\frac{\mathbf{w}}{||\mathbf{w}||},\mathbf{x_i}\rangle+\frac{\mathbf{b}}{||\mathbf{w}||}$$
basically expresses the distance of each point from your hyperplane, which is parameterized by the normal-vector $\frac{\mathbf{w}}{||\mathbf{w}||}$ (hyperplane orientation) and $\frac{\mathbf{b}}{||\mathbf{w}||}$ (hyperpplane bias), since you are projecting $\mathbf{X_i}$ on a unit-length vector.
Now if you maximize the righthand side of the equation (minimizing $\mathbf{w}$), it will necessarily maximize the lefthand side as well, due to the inequality direction, but, assuming that your representation is normalized as described above.
Hopes this helps shed some light here.


How many layers should I replace in transfer learning CNN I am designing a convolutional neural network that I believe requires transfer learning to function in practice. The network will be a character level CNN for text classification, more specifically, authorship identification of an author given unknown texts. The initial model will be trained on millions of texts from thousands of authors. In practice, if I want to be able to determine the authorship of a new given author/class not trained upon originally, I need to use transfer learning.
The structure of the network involves 6 convolutional layers and 3 fully connected layers. Given that the amount of data of the new author/class will be minimal in most cases, which layers should I replace and retrain for the new class for it to be the most effective? Or are there other methods I could consider to solve this problem?

To build on the previous answer:
In transfer learning, the goal is to use a pre-trained model and tweak the model to then specialise it to suit a certain task.
So, what we do is, as SrJ has eluded to, keep the main model's architecture in tact. So this would be the 6 CNN layers (and possibly the three linear layers, if they were also involved in pre-training).
After a model has been pre-trained, what we do is add additional layers to the model so that it is suited for our task. So in this case, the least you would do is have a final output softmax layer, which produces a probability distribution over the authors.
In between the final output layer and the original model's architecture, you can add more layers if it is appropriate.
When training this model with your task-specific data (this stage is called fine-tuning). We freeze our original model's architecture. This essentially means that the parameters within the models original layers will not change to prevent possible loss in generalisation performance. We only allow the additional layer's parameters to change during fine-tuning.
Overall message is to not replace layers, always add onto the existing model to tailor the model more to your classification task.


what are the main differences between parametric and non-parametric machine learning algorithms? I am interested in parametric and non-parametric machine learning algorithms, their advantages and disadvantages and also their main differences regarding computational complexities. In particular I am interested in the parametric Gaussian Mixture Model (GMM) and the non-parametric kernel density estimation (KDE). I found out that if a "small" number of data points is used then parametric (like GMM/EM) are the better choice but if the amount of data points increases to a much higher number then non-parametric algorithms are better. Could someone please explain both in bit more detail regarding comparison?

Non-parametric machine learning algorithms try to make assumptions about the data given the patterns observed from similar instances. By not making assumptions, they are free to learn any functional form from the training data and hence are flexible.
Unlike parametric approach, where the number of parameters are fixed, in non-parametric approaches the number of parameters grow with training data.
If your data set is too small or otherwise is a set that is not representative of the entire population, then your result will be biased in more ways than possible with parametric methods. To get better results in Non-parametric machine learning algorithms we need large amount of data where relationship between features are not known.
A Gaussian mixture model (GMM) is basically a k-means clustering with probabilistic cluster assignment so it depends on number of components/clusters as parameter whereas in KDE, the free parameters are the kernel which specifies the shape of the distribution placed at each point and the kernel bandwidth which controls the size of the kernel at each point.
Refer this blog to visualize KDE better.


How should I process music play data I have music play data organized by the day on which each track was played, from March 1st, 2015 to August 30th, 2015. The data set contains count data for every day a song was played. I'd like to predict play counts for each track for an out-of-time window, from 2015-09-01 to 2015-09-30.
Details of the data
The data is from Alibaba's TianChi Competition
Action Table (mars_tianchi_user_actions)

Any user’s action data for a song is indicated by a unique row
Songs Table（mars_tianchi_songs）

Result Set
The participants need to predict the artists’ plays data in the following two months (20150901-20151030).
Participant’s Result Table (mars_tianchi_artist_plays_predict)

For song id daa234f183aee2373d20987b247cd768(all the song ids are hash values). The play plot is:

As you can see it, the variance of each day's play for this track is quite large.
I also have data on who played each track, with each user ids represented as a hash value.
I'd like some guidance on which analysis method I could use here. I am considering time series analysis.

Interesting problem. You certainly could use time series methods for this, but you should consider whether you think there is face-validity to song plays following a temporal pattern. From your plot above, I don't see any structure in the variation, but perhaps there is something that an algorithm can pick up. A good starting point would be R's auto.arima function for autoregressive integrated moving average (ARIMA). The ARIMA method is more fully described elsewhere (e.g., here), but an interesting extension to this approach might be using exogenous predictors in your model. For example, does the presence or absence of rain affect the likelihood of some songs being played? My guess would be yes (I love listening to post-rock bands on rainy days), but perhaps that's not true of everyone. This extension to ARIMA is referred to as ARIMAX in the literature. A really nice breakdown of how to handle covariates in your type of analysis can be found on Rob Hyndman's blog.

A second thought is that, with sufficient computational resources, there's an interesting hidden markov model (HMM) problem here. You could potentially model the transition probabilities from one song to another. Presumably there's a high likelihood of transitioning from track 1 to track 2 of a given record, but sometimes people might switch to a different record after listening to a favorite track on an album. The extent to which this is true for different tracks could be a part of this analysis, and you could attempt to model the hidden underlying states that motivate a listener to stay within a record, or to move to a new one.


How do I determine if variables are correlated? Is it simply a mathematical calculation? I'm self learning data science so bear with me as I try and make my question as clear as possible. Lets assume I have a dataset of a dependent variable y and some independent variables x1, x2 and x3 etc.
Is the correlation between these variables simply a mathematical calculation? I mean if i ran a correlation test in R for example and the test came back with no correlation between the variables is that necessarily true? What if I knew or suspected that one of the variables is correlated to another how would I define that in my model or would relying on a correlation test be sufficient.

Yes, correlation is a mathematical concept, and it as well known as Pearson correlation. This is probably the one you are obtaining in your analysis.

However, there are other correlation analysis you can try in order to be totally sure about your result. The most famous ones out of Pearson are Kendall correlation and Spearman correlation.

In addition, sometimes relation between variables is less evident than a correlation analysis. For example, if the variables are time-related, one may have a lag over the other. For example, in finances it is known that the GDP is correlated with a lag of some months of the Employment Rate. It is part of a good Data Science knowledge to have a business knowledge, an intuition of why the variables may be related in some way and try to find how to prove that this non evident dependence exists.

Edit: I noticed you put R as a tag. You can study as well libraries polycor and ggm to learn about polychoric correlations and partial correlations, although the may not apply in your case.


Pandas: access fields within field in a DataFrame Suppose I have such a JSON file:

[
  {
    "id": "0",
    "name": "name0",
    "first_sent": "date0",
    "analytics": [
        {
            "a": 1,
            ...
        },
        {
            "a": 2,
            ...
        }
    ]
  }
]

and I want to parse it with Pandas. So I load it with

df = pd.read_json('file.son')

It's all good until I try to access and count the number of dictionaries in the "analytics" field for each item, for which task I haven't found any better way than

for i in range(df.shape[0]):
    num = len(df[i:i+1]['analytics'][i])

But this looks totally non-elegant and it's missing the point of using Pandas in the first place. I need to be able to access the fields within "analytics" for each item.
The question is how to use Pandas to access fields within a field (which maps to a Series object), without reverting to non-Pandas approaches.

A head of the DataFrame looks like this (only fields 'id' and 'analytics' reported):

0    [{u'a': 0.0, u'b...
1    [{u'a': 0.01, u'b...
2    [{u'a': 0.4, u'b...
3    [{u'a': 0.2, u'b...
Name: analytics, dtype: object
0      '0'
1      '1'
2      '2'
3      '3'

The first number is obviously the index, the string is the 'id', and it is clear that 'analytics' appears as a Series.

Multi-indexing might be helpful. See this.
But the below was the immediate solution that came to mind. I think it's a little more elegant than what you came up with (fewer obscure numbers, more interpretable natural language):
import pandas as pd
df = pd.read_json('test_file.json')
df = df.append(df) # just to give us an extra row to loop through below
df.reset_index(inplace=True) # not really important except to distinguish your rows
for _ , row in df.iterrows():
    currNumbDict = len(row['analytics'])
    print(currNumbDict)


Why a sign of gradient (plus or minus) is not enough for finding a steepest ascend? Consider a simple 1-D function $y = x^2$ to find a maximum with the gradient ascent method.
If we start in point 3 on x-axis:
$$ \frac{\partial f}{\partial x} \biggr\rvert_{x=3}   = 2x \biggr\rvert_{x=3} = 6 $$
This means that a direction in which we should move is a $6$.
Gradient ascent gives rule to update:
x = old_x + learning_rate * gradient
What I can't understand why we need to multiply a learing_rate with gradient. Why we can't just use x = old_x + learning_rate * sign(gradient).
Because if we made a  learning_rate step in a positive direction it is already a maximum switch of x we can make.
I know the reasoning behind finding maximum direction in this equation:
$$grad(())⋅⃗=|grad(())||⃗|cos() $$
But I can't undestand why just to accept a sign of gradient (plus or minus) is not enough for ascending.

Using only sign of gradient is a way to go, but might result in slow convergence. Nevertheless it is a valid variation of the method.

The Geometry of Sign Gradient Descent

Sign-based optimization methods have become popular in machine
learning due to their favorable communication cost in distributed
optimization and their surprisingly good performance in neural network
training. Furthermore, they are closely connected to so-called
adaptive gradient methods like Adam. Recent works on signSGD have used
a non-standard "separable smoothness" assumption, whereas some older
works study sign gradient descent as steepest descent with respect to
the ℓ∞-norm. In this work, we unify these existing results by showing
a close connection between separable smoothness and ℓ∞-smoothness and
argue that the latter is the weaker and more natural assumption. We
then proceed to study the smoothness constant with respect to the
ℓ∞-norm and thereby isolate geometric properties of the objective
function which affect the performance of sign-based methods. In short,
we find sign-based methods to be preferable over gradient descent if
(i) the Hessian is to some degree concentrated on its diagonal, and
(ii) its maximal eigenvalue is much larger than the average
eigenvalue. Both properties are common in deep networks.

Properties of the sign gradient descent algorithms

The aim of this article is to study the properties of the sign
gradient descent algorithms involving the sign of the gradient instead
of the gradient itself and first introduced in the RPROP algorithm.
This article provides two results of convergence for local
optimization, a first one for nominal systems without uncertainty and
a second one for systems with uncertainties. New sign gradient descent
algorithms including the dichotomy algorithm DICHO are applied on
several examples to show their effectiveness in terms of speed of
convergence. As a novelty, the sign gradient descent algorithms can
allow to converge in practice towards other minima than the closest
minimum of the initial condition making these algorithms suitable for
global optimization as a new metaheuristic method.


Is there a way to manually remove support vectors from a scikit-learn model? I'm in a situation where I need to limit the number of support vectors (SVs) in my support vector machine binary classifier. A simple way to do this would be to manually remove the least important SVs. Does anyone know of a way to do this?

I've tried manually modifying the following fields of the SVM to remove SVs, but without success:

clf.dual_coef_
clf.intercept_
clf.n_support_
clf.support_
clf.support_vectors_
clf.shape_fit_

where clf is my RBF SVM.

Note that increasing regularization (by decreasing C) does not necessarily reduce the number of SVs (https://dgroppe.com/2018/01/21/increased-regularization-does-not-necessarily-decrease-the-number-of-support-vectors/)

I was having the same problem. I managed to do it with Scikit's OneClassSVM; maybe it can also help in your case.
I adjusted some of the parameters you mentioned plus a few more (just in case) including:

_dual_coef_
dual_coef_
_intercept_
intercept_
support_
support_vectors_

Ex: For OneClassSVM, if you obtained 10 support vectors (after fitting dataset X) and wanted to prune the last one (index -1) the code would be as follows:

model = OneClassSVM()
model.fit(X)

model._intercept_ = model._intercept_ + temp_model.dual_coef_[0][-1]
model.intercept_ = model.intercept_ + temp_model.dual_coef_[0][-1]
model._dual_coef_ = np.delete(model._dual_coef_, -1, axis=1)
model.dual_coef_ = np.delete(model.dual_coef_, -1, axis=1)
model.support_ = np.delete(model.support_, -1, axis=0)
model.support_vectors_ = np.delete(model.support_vectors_, -1, axis=0)


Reshaping Pandas DataFrame I am trying to do the following as shown below.

    Input       Output
Letter Number    A B C
A      1         1 1 1
A      2         2 2 2
B      1           3 3
B      2             4
B      3
C      1
C      2
C      3
C      4

I have wrote the following code which works just fine.

import pandas as pd

df = pd.read_excel('Test.xlsx')
df = df.pivot(columns='Letter', values='Number')

list = []
for col in df.columns:
    col = df[col].sort_values()
    col.index = range(len(col))
    list.append(col)
ndf = pd.concat(list, axis=1, sort=False)
ndf = ndf.dropna(axis=0, how='all')
print(ndf)

Is there any other alternative way of doing this without using loops? Any help would be very much appreciated.

Here is the simplistic way.

>>> df = pd.DataFrame(data={'Letter': list('AABBBCCCC'),
>>>                        'Number': [1,2,1,2,3,1,2,3,4]})

>>> dfx = df.groupby('Letter').agg({'Number':list})
>>> dfx

              Number
Letter
A             [1, 2]
B          [1, 2, 3]
C       [1, 2, 3, 4]

>>> dfx = dfx['Number'].apply(pd.Series)
>>> dfx

          0    1    2    3
Letter
A       1.0  2.0  NaN  NaN
B       1.0  2.0  3.0  NaN
C       1.0  2.0  3.0  4.0

>>> dfx.T.fillna(0).astype(int)

Letter  A  B  C
0       1  1  1
1       2  2  2
2       0  3  3
3       0  0  4

So basically the sequence is:

aggregate by letter and put all numbers into a single cell by using df.groupby('Letter').agg({'Number':list}).
apply(pd.Series): turn "column with lists" into 2-dimensional array
T to transpose, and cleanup with fillna and type casting.


Multiple-input multiple-output CNN with custom loss function I have a set of 2D input arrays $(n\times m)$ namely $A,B,C$ and I have to predict two 2D output arrays namely $d,e$ for which I do have the expected values. You can think of the inputs/outputs as grey images if you like.

Because of the spatial information is relevant (there are actually 2D physical domains) I want to use a Convolutional Neural Network to predict $d$ and $e$. My design (not tested yet) looks as follows:

Because I have multiple inputs, I guess I should use multiple columns (or branches) to find different features for each of the inputs (they look fairly different). Each of these columns follows a encoding-decoding architecture used in segmentation (see SegNet): Conv2D block involves a convolution+batch normalisation+ReLU layer. Deconv2D involves a deconvolution+batch normalisation+ReLU.

Then, I can merge the output of each column by either concatenating, averaging or taking the maximum for example. To obtain the original $m\times n$ shape for each of the outputs I have seen I could do this with a $1\times1$ kernel convolution.

I want to predict the two outputs from that single layer. Is that okay from the network structure point of view? Finally my loss function depends on the outputs themselves compared to the target plus another relation I want to impose.

A would like to have some expert opinion on this since this is my first design of a CNN and I am not sure if I it makes sense as it is now and/or if there are better approaches (or network architectures) to this problem.

Looks correct. When u use same branches in NN architecture - this name siamese-type nn. U may see there how to implement it in PyPorch. In my opinion, PyTorch must easiest framework for this task. Also implementation of encoder-decoder network u can find there.

This is frequently using technique when u need to compute, as example, distance between few inputs. But i'm had doubts about concatenation part. Witch dimension u need for outputs? If same as input, then all looks fine. If different dim, then just add fully connected layers after last convolution.


when will the incareful features harm the model? I am working on financial prediction problem(time-series prediction problem).
I think feature engineering is importance in this problem. So i am careful to check the feature's effectiveness. And i perfer linear regression, because i think it's easy to explain, and I am not good  at machine learning models
But i think i have another method: build a lot of features without careful check, select good model to handle it.
So, my question is:
In which model, add what kind of features, will harm this model's ability?
And Is there any models, i can just add without worrying the harm of trash features? in this model, what should i pay attention to?

In very simple words,
Those models which use all the features of the instances will suffer from irrelevant features e.g. Neural Nets, KNN etc.
While the models which have an internal strategy to compares the features to decide the best while training will not suffer(at least for this reason) e.g. Tree
 Here is a snap from "The Elements of Statistical Learning, by Trevor Hastie,
Robert Tibshirani, Jerome Friedman

You can also read this section 10.3 Effect of Irrelevant Features from "Feature Engineering and Selection: A Practical Approach for Predictive Models by Max Kuhn and Kjell Johnson"
You can try these approaches to filters the features -

Using correlation with the target
Using Feature Importance
Using L1 penalty

Also, keep in mind that you might face other issues because of this and the models may suffer indirectly e.g.

High dimensionality
Issue of Trees "default feature importance approach" with high Cardinal features etc.


When being in a perfect "Long Valley" situation, does momentum help? I am thinking of the following image where we have two weights and an error (so we can make a 3D visualization). In this case a "long valley" looks like $x^2$ in the plane of the gradient, but in a perpendicular plane the function can still be minimized:

According to my understanding, all optimization algorithms which are based on the gradient should be "trapped" at the saddle point like SGD is. Why does having a momentum term help in this case?

(I'm sorry, I don't know who made those images. More of them are available.)

The answer is simple: in a perfect situation never. These optimizers are 1st order methods (unlike for example Hessian-Free optimization). The reason for they escaping from the stationary point above is that there is some small gradient (too small for ordinary SGD) in the "correct" direction. Momentum and Nesterov gradient can build up the velocity over time in that direction and Adagrad/Adadelta/RMS can immediately scale the gradient (increase it in the direction for which the past gradients were small).


Do CNN convolution and pooling layers get backpropogated? I can't find a simple answer to this by Googling which leads me to think the answer is no, but I want to be sure...

In a feed forward network, all of the layers of weights get backpropogated, but what happens in a convolutional neural network on the backprop step? Is it only the feedforward part of the network (after the convolution and pooling layers) that gets backpropped? Which would mean that the convolutional layers are a form of static feature extraction...

All layers of a neural network take part in the back-propagation process.
This includes the convolutional layers and the pooling layers. In general, every step of the network that the input has to go thru, the back propagation goes thru as well (in reverse order).

However, not all layers contain trainable parameters. For example standard pooling layers (max-pooling, average-pooling) and standard activation layers (sigmoid, ReLU, softmax) don't have any parameters to adjust. They still take part in the back propagation, contributing their partial derivatives, but they just have no weights that can be updated.

The convolutional layers do contain weights that are updated during the process (the parameters of the filter and their bias).

  Note: I assume that what you refer to as "feed forward part", are the
  fully-connected layers that are usually placed as the final layers of
  a network. In standard CNNs, all the network is a "feed forward part", (including convolutional layers)
  it just means that the input goes thru a sequential pipeline until
  becoming the final output.


Equation related to Smoothness If you have a differentiable function $f:\mathbb{R}^d\rightarrow\mathbb{R}$ that is $\beta$-smooth (for all $v$ and all $w$, you have $\|\nabla f(v)-\nabla f(w)\| \leq \beta \|v-w\|$), how can you show the equation below?

$f(v) \leq f(w) + \langle\nabla f(w), v-w \rangle + \frac{\beta}{2}\|v-w\|^2$

Have you tried using the mean value theorem for integrals? You can relate $f(v)-f(w)$ to $\nabla f$ and use some simple estimates to get something like the above.


Running scikit-learn with large volume I need to run a Random Forest process with scikit-learn. To train the model, I have a database table with 10 million rows of features. The question is: what is the best way to approach this, should I load into memory the 10 million rows, for example with numpy or pandas or there's a better way to load the data progressively by chunks?

There are multiple possiblities from dusk, to others model etc.
Here are my 2 favorites, not to loose you in the number of possibilities:

www.h5py.org/ "It lets you store huge amounts of numerical data, and easily manipulate that data from NumPy. For example, you can slice into multi-terabyte datasets stored on disk, as if they were real NumPy arrays. Thousands of datasets can be stored in a single file, categorized and tagged however you want."

Try online learning with Cousin models of random forest (light-gbm). He has online learning capabilities.


Dot product for similarity in word to vector computation in NLP In NLP while computing word to vector we try to maximize log(P(o|c)). Where P(o|c) is probability that o is outside word, given that c is center word.

Uo is word vector for outside word
Vc is word vector for center word
T is number of words in vocabulary
Above equation is softmax. And dot product of Uo and Vc acts as score, which should be higher the better. If words o and c are closer then their dot product should be high, but that is not the case with dot product. Because of following:
Consider vectors A=[1, 1, 1], B=[2, 2, 2], C=[100, 100, 100]
A.B = 1 * 2 + 1 * 2 + 1 * 2 = 6
A.C = 1 * 100 + 1 * 100 + 1 * 100 = 300
Vectors A and B are closer compared to A and C, however dot product A.C >A.B
So Dot product is acting as distance and not as similarity measure. Then why is it used in softmax. Please help me make my understanding better

The reference points to the word2vec library (source code).
It does not use normalised vectors during training (although it indeed uses the cosine similarity metric for semantic comparisons on already trained vectors).
The reasons for using only dot product instead of cosine similarity during training can be due to:

Dot product is a variation of cosine similarity.
Length captures some semantic information in the sense that length can correlate to frequency of occurance in a given context, so using dot product only captures this information as well (although for strict similarity testing cosine metric is still used)
When vectors are normalised the two metrics coincide.
Efficiency (doing less computations)

References:

Why does the word2vec objective use the inner product (inside the softmax), but the nearest neighbors phase of it uses cosine? It seems like a mismatch.
Should I normalize word2vec's word vectors before using them?
Measuring Word Significance using Distributed Representations of Words
word2vec Parameter Learning Explained


opening Weka from command line I'm very new to computer coding and data science in general. I'm trying to open Weka and use it on a practice csv data set called weather.arff.  The manual I have says to type this into command line:

  java weka.classifiers.j48.J48 -t weather.arff

When I open the command line it goes to C:\User\Admin
When I type in the above on command line it gives me message

  Error: Could not find or load main class weka.classifiers.j48.J48

Might someone give me some advice on how to proceed to make the command line work.
Thank you

I think your manual is for an older version of Weka.  I have version 3.9.3. In some version the Weka developers have perhaps changed some internal structures and class names.

In a terminal window with a command line prompt, go to the directory where you have Weka installed, that is where the weka.jar file is located. There should be a data/ subdirectory.

cd c:\.....

There, run this command.

java -cp weka.jar  weka.classifiers.trees.J48 -t data/weather.nominal.arff
(Your data file name could be different)

Then I get this output:

=== Classifier model (full training set) ===

J48 pruned tree
------------------

outlook = sunny
|   humidity = high: no (3.0)
|   humidity = normal: yes (2.0)
outlook = overcast: yes (4.0)
outlook = rainy
|   windy = TRUE: no (2.0)
|   windy = FALSE: yes (3.0)

Number of Leaves  :     5

Size of the tree :  8

Time taken to build model: 0.33 seconds

Time taken to test model on training data: 0 seconds

=== Error on training data ===

Correctly Classified Instances          14              100      %
Incorrectly Classified Instances         0                0      %
Kappa statistic                          1
Mean absolute error                      0
Root mean squared error                  0
Relative absolute error                  0      %
Root relative squared error              0      %
Total Number of Instances               14

=== Detailed Accuracy By Class ===

                 TP Rate  FP Rate  Precision  Recall   F-Measure  MCC      ROC Area  PRC Area  Class
                 1.000    0.000    1.000      1.000    1.000      1.000    1.000     1.000     yes
                 1.000    0.000    1.000      1.000    1.000      1.000    1.000     1.000     no
Weighted Avg.    1.000    0.000    1.000      1.000    1.000      1.000    1.000     1.000

=== Confusion Matrix ===

 a b   <-- classified as
 9 0 | a = yes
 0 5 | b = no

Time taken to perform cross-validation: 0.02 seconds

=== Stratified cross-validation ===

Correctly Classified Instances           7               50      %
Incorrectly Classified Instances         7               50      %
Kappa statistic                         -0.0426
Mean absolute error                      0.4167
Root mean squared error                  0.5984
Relative absolute error                 87.5    %
Root relative squared error            121.2987 %
Total Number of Instances               14

=== Detailed Accuracy By Class ===

                 TP Rate  FP Rate  Precision  Recall   F-Measure  MCC      ROC Area  PRC Area  Class
                 0.556    0.600    0.625      0.556    0.588      -0.043   0.633     0.758     yes
                 0.400    0.444    0.333      0.400    0.364      -0.043   0.633     0.457     no
Weighted Avg.    0.500    0.544    0.521      0.500    0.508      -0.043   0.633     0.650

=== Confusion Matrix ===

 a b   <-- classified as
 5 4 | a = yes
 3 2 | b = no


why my pytorch liner regression failed? I am new to pytorch, i want start from a simple example-linear regression:
I created some random training and test sample.
here is my code:
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as opt
from numpy import random
from util import *

np.random.seed(100)

class MLP(nn.Module):
  def __init__(self):
    super(MLP, self).__init__()
    self.l1 = nn.Linear(100, 100)
    self.l2 = nn.Linear(100, 1)

  def forward(self, x):
    x = (self.l1(x))
    x = (self.l2(x))
    return x

mlp = MLP().float()

target = nn.MSELoss()
o = opt.SGD(mlp.parameters(), lr=0.02, momentum=0.9)

w = torch.tensor(np.random.rand(100, 1) * 3).float()
x = torch.tensor(np.random.rand(100, 100) * 100).float()
y = torch.mm(x, w) + 2

test_x = torch.tensor(np.random.rand(100, 100) * 10).float()
test_y = torch.mm(test_x, w) + 2

for epoch in range(100):
  op = mlp(x)
  if epoch == 10:
    print(op)
    sys.exit(1)
  o.zero_grad()
  loss = target(op, y)
  loss.backward()
  o.step()
  test_pred = mlp(test_x)
  print(test_pred.shape)
  print(test_y.shape)
  print('%dth: loss=%.4f os_loss=%.4f'%(epoch, loss.item(), target(test_pred, test_y).item()))

I found it loss become nan, when several rounds passed.
i cant find out why, i think my netual network framework is correct, can you help on this?

The problem seems to come from your learning rate and the non-normalization of your data. Here your network is clearly unstable and thus gets to sky high values (10^20) which lead to NaN values. A typical learning rate for SGD is 0.001, but this is for normalized datas (inputs-outpus between 0 and 1). Here your inputs and ouputs have high values, that are amplified even more by MSE (which squares the error). So this is why there is an issue with the learning rate here, the resulting gradient is way too strong.
This is how I understand the network behaviour, that is clearly unstable with lr=0.02.
A way to solve it is greatly diminish your learning rate (lr=0.00000001 worked for me). Another way around is to Normalize your data (inputs and ouputs)
Here is your modified code, you can try changing the learning rate and see how it changes the network behaviour (from stable to unstable).
import torch
import torch.nn as nn
import torch.optim as opt
import numpy as np
import matplotlib.pyplot as plt
import math
import os

os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE"  # This line may be useless

class MLP(nn.Module):
    def __init__(self):
        super(MLP, self).__init__()
        self.l1 = nn.Linear(100, 100)
        self.l2 = nn.Linear(100, 1)

    def forward(self, x):
        x = self.l1(x)
        x = self.l2(x)
        return x

mlp = MLP().float()

target = nn.MSELoss()
o = opt.SGD(mlp.parameters(), lr=0.00000001)  #Try 0.0000001 and 0.000001

w = torch.tensor(np.random.rand(100, 1) * 3).float()
x_train = torch.tensor(np.random.rand(100, 100)*100).float()
y_train = torch.mm(x_train, w) + 2

test_x = torch.tensor(np.random.rand(100, 100)*10).float()
test_y = torch.mm(test_x, w) + 2

losslist = []
losstestlist = []

for epoch in range(100):
    op = mlp(x_train)
    o.zero_grad()
    loss = target(op, y_train)
    losslist.append(math.log10(loss))
    loss.backward()
    o.step()
    test_pred = mlp(test_x)
    loss = target(test_pred, test_y)
    losstestlist.append(math.log10(loss))

plt.plot(losslist, 'r', label='train loss')
plt.plot(losstestlist, 'b', label='test loss')
plt.legend()
plt.show()

Hope this helps.


BERT: it is possible to use it for topic modeling? I'm struggling to understand which are the full capabilities of BERT: it is possible to make topic modeling of text, like the one we can achieve with LDA?

No. Most folks use https://radimrehurek.com/gensim/models/ldamodel.html for topic modeling and https://github.com/bmabey/pyLDAvis for visualization.


LSTM Time-series classification - derived feature I have a time-series dataset and I want to derive a new feature based on a date column which I believe might improve my predictive model. The feature is if it's weekday or weekend.

I am not sure how to create this for modeling purposes. Do I represent it as a boolean function or a numerical one (1/0) or just add 2 new features (weekday, weekend) and represent them as 1 and nan?

Add two new features, dummy variables, that represent the presence or absence of weekend and weekdays. You can create the feature (weekend/weekday) and use one hot encoding to create the two new.


Hashing Trick - what actually happens When ML algorithms, e.g. Vowpal Wabbit or some of the factorization machines winning click through rate competitions (Kaggle), mention that features are 'hashed', what does that actually mean for the model? Lets say there is a variable that represents the ID of an internet add, which takes on values such as '236BG231'. Then I understand that this feature is hashed to a random integer. But, my question is:

Is the integer now used in the model, as an integer (numeric) OR
is the hashed value actually still treated like a categorical variable and one-hot-encoded? Thus the hashing trick is just to save space somehow with large data?

The a second bullet is the value in feature hashing.  Hashing and one hot encoding to sparse data saves space.  Depending on the hash algo you can have varying degrees of collisions which acts as a kind of dimensionality reduction.

Also, in the specific case of Kaggle feature hashing and one hot encoding help with feature expansion/engineering by taking all possible tuples (usually just second order but sometimes third) of features that are then hashed with collisions that explicitly create interactions that are often predictive whereas the individual features are not.

In most cases this technique combined with feature selection and elastic net regularization in LR acts very similar to a one hidden layer NN so it performs quite well in competitions.


Why does a belief network need to be represented using a directed acyclic graph (DAG)? I would have thought that it's because DAGs preserve the dependency relationships between the variables, but I am currently unsure.

Thanks

Yes, we use DAGs to represent dependency relationships.

We need directed graph because condition probability P(A|B) is not same as P(B|A).

We assume it to be acyclic to get certain properties and ease calculations. You can create cyclic graph and you are going to get a lot of contradictions.

If you ease both the restriction(directed and acyclic), we get more general model called Markov Network.


How to rank pages based on CTR I have a collection of articles and for each articles I have the number of clicks and views, so I can measure the Click-Through Rate (CTR) as the number of clicks over the number of views.

However, the CTR is not a very informative quantity because, for instance, two pages with CTR 10% can have very different values for the clicks and the views. For example, page A can have 10 views and 1 clicks and page B 10 million views and 1 million clicks.

So I was wondering if there was an algorithm that can rank pages based on their CTR that also account for the absolute value of clicks and views because in the previous example page B may have a better ranking than A.

Just CTR is not important. Also, cost-per-click should be applied. Hence, to rank sites in an ads auction, using $CPC\times CTR$.


Condense matrix of values into one column? I have a dataset where I have unnecessarily duplicated column variables that I want to condense down. I wish the output wasn't so clumsy and I've already had to do some work to transform it and make it easier to manage. I'm familiar with basic R stuff but not an expert by any means so please be patient!

Each R(1-10) corresponds to a response rating for the question (q1-10). The questions are randomised in order for each trial for each participant, so the qnum doesn't refer to the same question every time. I want to condense the matrix of response ratings down so that there is a single column called 'response' which reflects the rating given by the participant for each qnum for each trialNumber, so I don't have unnecessarily repeated tokens! Essentially going down the diagonal and reshaping it into one column. How can I do this?

You can use quite easily using the dplyr and tidyr packages as follows:
df %>%
    # convert the columns to rows
    gather(key="response", value="value", R1, R2, R3, R4, R5, R6, R7, R8, R9, R10) %>%
    # select only rows where the reponse corresponds to the question
    filter(qnum == substr(response, 2, numchar(response)))


How to fit integer output in neural network I have a data set that contains 104 input features and 96 output values to be predicted, the input features are floating values normalized to [-1,1], and each out put value would be one integer among [-1,0,1].

I am basically new to data science and machine learning, and I need to know what kind of neural network model would fit best for this kind of data. I have googled this for a quite long time, can you guys help me ? Thank you!

我的每一条训练数据具有104个[-1,1]之间的浮点型输入量和96个整数型输出量，每个输出值为[-1,0,1]之中的一个整数。
我是机器学习的新手，我想知道应该采用何种神经网络模型来解决此问题，在post本问题之前我已经自行搜索了很久，希望大家能给我一些帮助，谢谢！

You may check a multitarget model like this one. Basically Keras‘ functional API gives you a lot of flexibility to model multitarget problems.

You have three classes per target, so make sure you use 'categorical_crossentropy' as loss function with three terminal layers.


Classification or regression? Which model is more accurate if I only care about being above or under the threshold? If I have a regression problem that can also be a classification problem by converting a continuous variable to a binary depending on a threshold, which model would be more accurate if I only care about being above or under the threshold?

Example: I have a column with house prices [20, 30, 40, 70] but I only care if the house is expensive (more than 40) or cheap (less than 40)

Classification is the more direct approach and it will likely give better results. This is because the model's goal is exactly the same as your goal - i.e. predicting whether the price is above or below the threshold - and it will maximize this accuracy.

A regression model that is trying to minimize MSE, for example, could give you a model that doesn't accurately put predictions on the correct side of the threshold - which it doesn't know or care about. The model does not have the same goal as you do.

Let's construct a very simple example to see why this might be the case:

Training     Test
x      y     x      y
1.0    20    1.8    ?
2.0    20    2.4    ?
3.0    200   2.8    ?
4.0    200   3.2    ?

Here I used the rule if x<2.5: y=20, else y=200 to generate the data.

A maximum-margin classifier would get all of the test examples right, but a linear regression model fit on the training data would get all of the test examples wrong.


Prediction of a dependent variable having comma separated values I have a dataset from survey results of Genetic Testing that looks like below:

For more details sample csv file here:

https://drive.google.com/open?id=0BzFPqeNZO-6mM2VwZ1pRQ3NxYTA

I am trying to create a model to predict the column “actionspost” (the extreme right column of the above file). The column currently holds comma separated values like “2, 4, 6”. Each numerical value is a coded action that a patient would take after knowing the results of the genetic test as per the survey filled by the patient. For example, 2 indicates: Change eating habits, 4 indicates: Getting members of his/her family tested and so on. Based on the features in the csv file above, I want to create a model that would help me predict the “actionspost” values.

One way I thought, I could do this was break down comma separated values in “actionspost” to create duplicate rows for same customer Ids holding only one “actionspost” value at a time (For example: There would be 4 rows for customer ID: C00003 with first row having “actionspost” value of 2, second row with “actionspost” value 8, third with 9 and fourth with 10. Then create subset of the data set for each unique value of “actionspost”. This time, the dependent variable will not be actionspost but something like “isactionpost2?” (Yes =1 and No = 0). And train models for each of this subset for different values within the “actionspost” column.

I personally don’t like the above method as it too cumbersome and in my opinion, not the optimal one too. I was wondering if there could be a better way to address such scenario? My end goal is to try to train different models like Decision Trees, Naïve Bayes, and Neural Nets and check which one leads to better predictability.

Your problem is called multi-label classification. If you search for this term you will find a lot of literature. In short, these are the most common approaches:

Treat each label independently and build a binary classifier per label (baseline approach, Eric Lecoutre's answer)
Predict the power set, means create a target variable that represents all possible combinations and predict it using a multi-class classifier (Upper_Case's answer)
Use a classifier that can deal with multi-labels inherently, e.g. a neural net with multiple output nodes and cross-entropy as loss function, or respective multi-label variants of decision trees or SVM. For example, scikit-learn supports multi-label implementation for decision trees.
Classifier chains: Train a binary classifier per label, but use the predictions of one classifier as input to another classifier

Depending on the nature of the data, one or the other approach will work best. Start with a binary classifier per label to get a baseline. If the labels are correlated (some labels frequently occur together), there is potential for improvement using one of the other methods. Using one of the existing implementations of multi-label classifiers is probably the easiest next step. The power set approach only makes sense if the number of possible combinations is reasonable. I am not sure if classifier chains are worth the effort, if you are not doing this for a Kaggle competition :)

As always, setup a proper cross-validation strategy, define the evaluation metric (e.g. micro/macro f-measure or cross-entropy loss) so that you are able to identify the best model.


Decision tree too small I have a data set of 2300 entries, with 5 variables one of them the dependent variable which is binary.

I fitted a decision try using the rpart function in R over the 4 independent variables, and I had this tiny tree, which means the split was done using only one predictive variable.

Is it normal? How can I add the rest of the nodes?

Have you changed the max depth parameter? This parameter limits the depth of each tree. If you have set it up to max_depth=1 you wont be able to grow your tree further.

From the documentation:

  max_depthint, default=None
  The maximum depth of the tree. If None, then nodes are expanded until all leaves are pure or until all leaves contain less than min_samples_split samples.

Also, it could be that you have your target on the train and your tree does not need to develop further.

Or that you are training with a small number of rows or columns


Steps for back propagation of convolutional layer in CNN Imagine I have the following layers in a CNN:

Conv-layer 1 (without reLU) [3 filters @ 1x3x3] =>
ReLU-layer 1 =>
Maxpooling-layer 1 [2x2]

Conv-layer 2 (without reLU) [10 filters @ 3x4x4] =>
ReLU-layer 2 =>
Maxpooling-layer 2 [2x2]

What I know is that, when "deriving" the Maxpooling-layer, the maximum value of that pooling has the value of 1 (because that value is the only one that is affecting the result of the CNN), and the other values take the value of 0.

After this step, I have the reLU-layer, where the derivative of the reLU-layer is 0 when x <= 0, and 1 when x > 0.

And in the last step, I have the Conv-layer, but I don't know how to calculate the derivative. How do I calculate the whole delta for that specific weight?

If there is a way to calculate the gradients of those weights using matrix notation, please let me know.

Notation

The matrix notation that I use is the following: [number of filters - depth - rows - columns]. For example, when I say [3 filters - 1 x 3 x 3] I mean that I have 3 filters with depth 1, 3 rows and 3 columns (depth is "equivalent" to "channels").

Steps in Conv layer, using my notation

Imagine I have an input X with shape [depth:1, rows: 28, columns: 28]

Then, I have the conv-layer, where I have 3 filters (you can call the filter as W) with shape [depth:1, rows:3, columns:3]

So, what I do is, using 1 filter a time, convolve input X , so the resulting matrix has the following shape => [depth:1, rows: 26, columns: 26] (this is the result of convolving 1 filter). So, when I do the convolution using the 3 filters, the resulting matrix is shaped [depth:3, rows: 26, columns: 26].

After this, I apply ReLU to each value of the resulting matrix (the 3x26x26 matrix), and then apply non-overlapping maxpooling with a window of 2 rows x 2 columns.

Extra Note: The matrix notation that I use is the following: [number of filters - depth - rows - columns]. For example, when I say [3 filters - 1 x 3 x 3] I mean that I have 3 filters with depth 1, 3 rows and 3 columns (depth is "equivalent" to "channels").

Steps in Conv layer:

Imagine I have an input X with shape [depth:1, rows: 28, columns: 28]

Then, I have the conv-layer, where I have 3 filters (you can call the filter as W) with shape [depth:1, rows:3, columns:3]

So, what I do is, using 1 filter a time, convolve input X , so the resulting matrix has the following shape => [depth:1, rows: 26, columns: 26] (this is the result of convolving 1 filter). So, when I do the convolution using the 3 filters, the resulting matrix is shaped [depth:3, rows: 26, columns: 26].

After this, I apply ReLU to each value of the resulting matrix (the 3x26x26 matrix), and then apply non-overlapping maxpooling with a window o 2 rows x 2 columns.

If there is any doubt of what I explained, please comment me.

Note: I'm editing my own answer beacuse I don't have 50 reputation for doing comments.


I only have microsoft office, how can i write code to manipulate relational data sets? At work, I am in a role that does not have a lot of computer permissions. I do have the Microsoft office suite. I have skills with programming and would like to be able to use my experience to do more streamlined and substantial manipulation of large relational data sets. Considering I can't get python or even Bash, is there any good way to do this?

It should be possible to install Python without admin privileges. This has been discussed on Stackoverflow here.

The easiest method appears to be installing Anaconda.

And according to this answer you should also be able to use WSL without admin rights, which will get you bash.

For handling relational data you can use sqlite, which is built-in to Python. For example:

import sqlite3

conn = sqlite3.connect('example.db')
cur = conn.cursor()
cur.execute('''CREATE TABLE users(id INTEGER PRIMARY KEY, name TEXT''')
conn.commit()
conn.close()


Best image PPI for computer vision deep learning algorithms For computer vision tasks using deep learning, should I worry about image size (e.g. 256 x 256) or PPI (pixels per inch)?

I find that PPI is not discussed in the computer vision/deep learning literature.

What's the best PPI for images of size 256 x 256?

PPI is only relevant for physical objects, like representations on screen or paper. It is pixels per inch, and defines how big an image X by Y pixels appears on that object. So if you have a 256x256 pixel image, and you show it at 256 PPI on a screen it will be 1 inch square if you put a ruler up to it.

Many physical devices have natural PPI characteristics. Screens are often 72 PPI, 100 PPI, or about 300 PPI for "Retina" devices. Laser printers start at 300 PPI and higher.

So if you show a 256x256 image on a 100 PPI screen as a direct mapping of one image pixel to one screen pixel, you'll get an image 2.56" across.

If you zoom into the image you can make it bigger on the screen, and it will now cover more than 256 pixels of the screen. The underlying image data will still be an array of 256x256 values

Scanners are another area where PPI is important. If you scan a 1" image at 300 PPI, you get a 300x300 pixel image. Set your scanner to 1200 DPI and you get a 1200x1200 pixel image, that is much finer detail.

In summary, PPI is irrelevant to CV or ML algorithms because they only care about the number of pixels.


TPOT machine learning I trained a regression TPOT algorithm on Google Colab, where the output of the TPOT process is some boiler plate Python code as shown below.
import numpy as np
import pandas as pd
from sklearn.ensemble import ExtraTreesRegressor
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline, make_union
from tpot.builtins import StackingEstimator
from tpot.export_utils import set_param_recursive

# NOTE: Make sure that the outcome column is labeled 'target' in the data file
tpot_data = pd.read_csv('PATH/TO/DATA/FILE', sep='COLUMN_SEPARATOR', dtype=np.float64)
features = tpot_data.drop('target', axis=1)
training_features, testing_features, training_target, testing_target = \
            train_test_split(features, tpot_data['target'], random_state=1)

# Average CV score on the training set was: -4.881434802676966
exported_pipeline = make_pipeline(
    StackingEstimator(estimator=ExtraTreesRegressor(bootstrap=False, max_features=0.9000000000000001, min_samples_leaf=1, min_samples_split=20, n_estimators=100)),
    ExtraTreesRegressor(bootstrap=True, max_features=0.9000000000000001, min_samples_leaf=6, min_samples_split=13, n_estimators=100)
)
# Fix random state for all the steps in exported pipeline
set_param_recursive(exported_pipeline.steps, 'random_state', 1)

exported_pipeline.fit(training_features, training_target)
results = exported_pipeline.predict(testing_features)

Would anyone know what is the Sklearn pipeline process like how does this work? When I butter up the boiler plate code and run it with my data set in IPython I can see this output from the pipeline process, what is this all doing?
Pipeline(steps=[('stackingestimator-1',
                 StackingEstimator(estimator=ExtraTreesRegressor(max_features=0.6500000000000001,
                                                                 min_samples_leaf=19,
                                                                 min_samples_split=14,
                                                                 random_state=1))),
                ('maxabsscaler', MaxAbsScaler()),
                ('stackingestimator-2',
                 StackingEstimator(estimator=ExtraTreesRegressor(max_features=0.4,
                                                                 min_samples_leaf=3,
                                                                 min_samples_split=7,
                                                                 random_state=1))),
                ('adaboostregressor',
                 AdaBoostRegressor(learning_rate=0.001, loss='exponential',
                                   n_estimators=100, random_state=1))])

The results look good just curious about how the pipelines processes work, any tips or links to tutorials greatly appreciated. I thought this machinelearningmastery tutorial is also somewhat useful for anyone interested in learning more about TPOT.

The documentation for StackingEstimator is surprisingly poor, but it's relatively simple: fit the estimator on the data, and tack its predictions onto the dataset as a new feature.  Source, github issue.
So, your pipeline fits an ExtraTreesRegressor on the original inputs, and appends its predictions to the dataset going forward.  The data (original + 1st predictions) get scaled, then another ExtraTreesRegressor is fit (on scaled original and 1st preds, and with different hyperparameters), its predictions also getting tacked onto the dataset.  Finally, an AdaBoostRegressor is fit on scaled-original + scaled-1st-preds + 2nd-preds.


Containing multicomponent data in rows or columns I have been working with DNA sequences and compiled a table with features from those sequences. I have a column called Trimer, which contains strings. For some DNA sequences there is one trimer of interest so that column contains one 3 character string (i.e. "ATG"). For other rows in the table that trimer column has 2 or 3 trimers of interest so the Trimer column has multiple strings in it (i.e. "ATT, CTG, GAT"). All trimers from one sequence should be thought to have equal weight and importance.

I know I cannot analyze my data in this format. I was wondering whether to split the Trimer column into 3 columns so if a sequence only has one trimer of interest the cells in the other two columns will remain blank. When doing my analysis my worry is that these columns will be seen as different features and will be weighted differently.

I was also thinking to make multiple row entries for the same DNA sequence. But the independent variable is influenced by the combination of trimers.

Any advice on how to change my table or create dummy variables is much appreciated. Thank you!

Note: This more of a comment to help your question being answered I will delete it if needed.
It is not clear to me which analysis you want to do with this dataset thus it is hard to advice on the structure and format of the dataset.
In my experience when dealing with DNA strings and their associated features, R and in particular the Bioconductor package are the best. These packages contain already data structure, functions and operations purely design to deal with bioinformatics tasks.The GenomicAlignements could be a good start.
Hope this helps!