Relation mining of multivariant categorical timeseries without excluding the temporal nature To all:

I have been wracking my brain at this for a while and thought maybe someone here would know of a package or algorithm to handle the following:

I have nominal multivariant timeseries that look like the following:

          Time Var1 Var2 Var3 Var4 Var5 ... VarN
             0     A     A   B    C    A   ... H
             1     A     A   B    D    D   ... H
             2     B     A   C    D    D   ... H
             ..

And so on from times 0 to 1,000,000. What I would like to do is search the time series for rules of the type:

Given Var3 is in state B in the previous step and Var5 is in state D in the previous step, than Var1 will be in state B. What I want to do is have the rules that include the time interval explicitly. A simpler case of interest would simply be to reduce the time series to

               Time    Var1 Var2 Var3 Var4 Var5 ... VarN
                0        0    0    0     0   0   ... 0
                1        0    0    0     1   1   ... 0
                2        1    0    1     0   0   ... 0

Where the the variable is 1 if its state is different from the previous step and zero otherwise. Then I just want to have rules that say something like:

If Var4 and Var5 changed in the previous step than Var1 will change in the current step. Which would be easy for a lag of one, as I could just make the data into something like:

   Var1 Var2 Var3 Var4 Var5 ... VarN Var1_t-1 Var2_t-1 Var3_t-1 ...

and then do sequence mining, but if I want to have rules that aren't just a single lag but could be lags from 1 to 500 than my data set begins to be a little difficult to work with.

Any help would be greatly appreciated.

Edit to respond to comment:
Each column could be in one of 7 different states. As far as a target, it is non-specific, any rules between the columns would be of interest. However, predicting columns 30-40 and 62-75 would be particularly interesting.

This problem is one of estimating the lag. Once that is estimated, you could create additional features representing the lagged values and move forward with "sequence mining" as you have already suggested in the question itself.

For each variable, Var_i, you will have to estimate its lag l_i. This lag can be calculated by estimating the order of a Markov chain with seven symbols (you could use either BIC or AIC to estimate this order; both would require calculating likelihood of candidate orders and pick the order that maximizes either of these criteria). Once you are done calculating the order of the Markov chain for each of the variables, then you could represent your dataset such that each row will have the current value of Var_i and its preceding values, all the way back to its estimated lag l_i. While this methodology is laborious, it pays rich dividends as its automated and parsimonious way of representing the necessary information.


Can I apply feature selection before splitting by requiring selection occurs > 90% of time I want to move the feature selection step to before splitting to save time and allow bigger input dataset. If, in repeated subsamples, a feature is selected in over X percentage of cases I will keep it. Alternatively use very low X to remove features that will clearly never be selected. I have read warnings against doing this including on this forum because of information leakage. Feature selection: Information leaking if done before CV-split?  But if the feature would have been selected in almost all post split cases then where is the problem?
Edit: it does involve the target features.

As explained in the post you link, it depends how you select the features: if it doesn't involve the target variable, then it's probably fine. I'll assume the most common case, that is that the selection relies on the target variable.
There are two parts in this problem:

About the ensemble method for feature selection: let's say a particular feature is selected 90% of the time. This means it's not selected 10% of the time, so if you were using a single training set it would have a 10% chance not to have this feature selected. So in theory your model is likely to be better than a "regular model", because it includes a few features which wouldn't have been selected in the "regular model". Note however that this method could also have negative side effects, because the features are not selected as a whole but individually (i.e. it might not be the optimal subset of features).
About the risk of data leakage: by definition, applying this method on the whole data means using information from the test set, so a potential bias in the evaluation. It's true that the ensemble method decreases the risk of selecting a feature by chance, but for every feature there's a 10% chance that it wouldn't have been selected in the "regular model". Since this selection is based partly on the test set, you can't be certain that the evaluation is reliable.

Assuming your goal is to use cross-validation and that the feature selection process is computationally expensive, I can think of two ways to do this properly:

Select a random N% of the data (where N is the size of the training set, e.g. 90% if using 10 fold CV), do the feature selection on this data, then use this predefined set of features every time, independently from the training set. The idea here is that the selected set of features cannot take advantage of a selection on the whole data, so it's not particularly  optimized for the test set or any CV split. That should be enough to obtain a fair evaluation, although technically there is still data leakage in this way.
Do an additional split in your data: with the training set you apply exactly the process that you described for feature selection, then run CV on it. After this, apply the final model to the test set. For example take 80% of the data for the training set, do feature selection and CV based only on this training set, and after that use the 20% left of unseen instances as test set. There's no data leakage in this way, but the final evaluation comes from a single test set, not from CV (here the CV stage can be used to study performance variation or tune hyper-parameters).


Is there any implementation of Extended Isolation Forest algorithm in R/Python? I am using isofor package for regular Isolation Forest but I came by an article about Extended Isolation Forest and need your advise which package has this functions implemented in R/Python.

There is a package on Github called "Extended Isolation Forest for Anomaly Detection", I used it a couple months ago and it seemed to work. For how accurate or how buggy it is, I'm not sure but if anything seems off you can check the source code for errors in the implementation of the paper Extended Isolation Forest by Harir et.al.


Examples of the Three V's of Big Data? What are some examples of the Three V's of Big Data? The three V's stand for: volume, velocity, variety.

Reference:
Three V's of Big Data, provided by Norwegian University of Science and Technology.
https://www.ntnu.edu/ime/bigdata/what-is

Volume:
Simply stated, big data is to big to work on one computer. This is a relative definition, as what can't work on today's computer will easily work on computers in the future.
 - One Google search uses the computing power of the entire Apollo space mission.
 - Excel used to hold up to 65k rows in a single spreadsheet. Now it holds over a million.

Velocity:
Data is coming in extremely fast. Traditional scientific research methods of a few hundred cases could take weeks, months or even years to analyze and publish.
 - Iris flower data set
 - Statistical Programming Language R
 - Twitter Firehose (6,000 tweets per second)

Variety:
Big Data that is contained in one specific data type or does not fit well within the format of a relational database. This data often comes in the form of unstructured text.
 - Estimated 80% of all enterprise data is unstructured
 - Open Data(Government)
 - noSql Databases

Iris Data Set: https://en.wikipedia.org/wiki/Iris_flower_data_set
Open Data: https://www.data.gov/open-gov/


What is difference between Bayesian Networks and Belief Networks? While reading some articles about Bayesian Networks, I came across many occurrences of Belief Networks.

Do both of these terms mean the same thing or is there any difference between Bayesian Networks and Belief Networks?

Both are literally the same. A Belief network is the one, where we establish a belief that certain event A will occur, given B. The network assumes the structure of a directed graph. The term Bayesian was coined after the name of Thomas Bayes.


Using a combination of gradient boosting with LSTM for classification? I am presently using an LSTM model to classify high dimensional tabular data which is not text/images (dimensions 21392x1970). I also tried XGBoost (Gradient boosting) in Python separately for the same classification task (classify into one of 14 categories of different categorical values). I have come across the provision of using feature_selection_ method in XGBoost, which can provide me the F1 scores of the most relevant features in prediction.

I would like to create a hybrid model that combines the LSTM with the XGBoost but am confused as to how I can do something such as using the most important features for classification (probably getting these through XGBoost) and then feeding to an LSTM ?)by a combined approach. Any ideas, suggestions, and comments are appreciated!

I'm not aware of any streamlined way of doing this without writing the proper code to connect the two models.  It should be fairly straight forward to run XGBoost, grab the feature importance and use this as a way to identify the features to use for the LSTM.  This part should be not so hard, just have a list of features with some criterion of whether to keep or drop it.  Once you have the list of features you want to use for the LSTM, just filter the whole dataset, and build out the LSTM model using that set.


Which Algorithm did OpenAI used to create a hide and seek playing Agent? I just saw this video on youtube: https://www.youtube.com/watch?v=kopoLzvh5jY&t=9s
Which Algorithm did OpenAI used to create a hide and seek playing Agent?
Was it Genetic Algorithm or Policy Gradients or something else?
If it was Policy Gradient method, then which Policy Gradient method did they used?

This is specified in the original paper that led to that video:

Policies are optimized using Proximal Policy Optimization (PPO) (Schulman et al., 2017) and Generalized Advantage Estimation (GAE) (Schulman et al., 2015)

They, nevertheless, used concepts often used in evolutionary algorithms, specifically competitive co-evolution, as the goal of the seeking agents is contrary to the goal of the hiding agents, and this drives the reinforcement signal.


Data augmentation for the inputs of CNNs to identify flowers I want to make a neural network to identify flowers from images like this:

or similar images e.g as on https://www.pexels.com/search/flowers/

I want to use a CNN for this as in https://stackoverflow.com/questions/52463538/reducing-memory-requirements-for-convolutional-neural-network

My questions are:

Which will be better: to put whole images into training set or to divide each image into 4 parts (cutting in midline horizontally and vertically)?
Will it help if I rotate/tilt these images and put them also in training set?
Will it help if I blur these images and put them also in training set?

The main aim of network is to identify which kind of flower it is. For this, it will be trained with images of about 20 flowers. Hence, classification is the objective.

Edit: we can assume there will be only one kind of flowers in one image.

As you've mentioned your task is classification and due to using images, it is better using convolutional neural networks. First I have a suggestion, try to find an appropriate size with the same dimensions for all the images and feed them to your network. you can keep aspect ratio or not depending on the environment you are going to test your model. You can also take a look at Why do we scale down images before feeding them to the network.

  Which will be better: to put whole images into training set or to divide each image into 4 parts (cutting in midline horizontally and vertically)?

I guess you attempt to do data augmentation. If so, it depends. If you do that, you may have images that do not contain the flower or they may contain a part of a flower which is common among different classes. Consequently, it may increase the Bayes error. If you are sure you do not have these problems, you can use it albeit I do not think so.

  Will it help if I rotate/tilt these images and put them also in training set?

Yes, it is better than the previous technique. You have to be aware that this is dangerous in some cases. Basically, you should train your network using the training set which is given i.i.dly from the real distribution that your test data has. Suppose that while testing your model, your test data all are given from flowers which are vertically placed in the scene. In such cases, if you train your network with just rotated versions, you may not have a good test result.

  Will it help if I blur these images and put them also in training set?

Again like the previous answer, it depends on your test data. If it is something that happens while testing, it is legitimate.

I didn't notice to the images. Based on the comment of our friend, I update the answer. Your classes can be one these things. They can be exhaustive or not and they can be mutually exclusive or not. if the former is satisfied, it means that the input should at least belong to one class. The latter means that if you have mutually exclusive classes, the inputs should contain only one class. If they are not mutually exclusive, you can have multiple classes in a single input.

To add an update for the answers above you should consider that data augmentation with reduction can be difficult because it needs an expert to label the inputs by hand which is time consuming.


How does the "skip" method work for upsampling? (fully convolutional NN) I'm studying fully convolutional neural networks for image segmentation, so far i've study and kind of understood the deconv network.
Following this tutorial (Upsampling) i can't really understand how the "skip" method works.

I understood how to apply max unpooling and the transpose convolution for upsampling,but how does the "skip" method "relate" to those 2 methods?

Why do we use this "skip" method? Max unpooling and transpose convolution are not enough to upsample and give back a better resolution map?

So as you already know, the convolutions extract some local information from the image, but for some decisions you need more global info. This is what pooling is supposed to do, it basically erases some locality information. In plain classification this is obviously necessary, because the question, if there is a cat in a picture, is a question about the entire image, not about one or a few pixels. This is why in the end of the classifier you get rid of all locality by adding fully-connected layers.

For semantic segmentation it is not quite as simple, because the output is supposed to still have locality information in it. You actually are just doing a pixel-wise classification and for each pixel the broad context matters. To decide, if a pixel is part of the cat in an image requires information about many pixels within a certain distance from the pixel in question. So this is why pooling layers are used for semantic segmentation as well. But pooling also reduces the image size, that's why upsampling and transpose convolutions are used to get back to the original size.

But there is a downside. As we said, pooling erases some information about the locality, in other words, it reduces the resolution. Once this information is gone, no combination of upsampling and transpose convolutions is going to get it back. Transpose convolutions can only learn to do this the least bad, kind of like a pseudo-inverse matrix. This limits the resolution/detail you can expect on your segmentation mask.

This problem is solved by skip connections by providing the relevant locality information where it is needed. Whenever you do upsampling, you find the last layer before the pooling, where the image still had the same size and simply add it pixel-wise to the upsampled image. This pixel-wise connection is implemented by using a 1x1-convolution in the link you posted. This way, the now upsampled feature map has both the locality detail from before pooling, but also the broad context info from after pooling. This allows for much better detail in the segmentation mask.


Should i pick this as a career? I'm mainly interested in creative things, but also interested in the science behind viewership and marketing a piece of media. Think about having a stable career in data science, where do you think I should begin? Are media analytics as big as others like business intelligence? Ive checked online courses, like google analytics academy and others like these:

https://www.edx.org/xseries/data-science-analytics-context

https://www.coursera.org/specializations/jhudatascience

Is this the right path to take? And is it easy to find jobs in such a field even if the person is a beginner in programming?

Data Science is all about creativity and do/see things what makes sense.

I would describe Data Scientist career in couple words: Take this path only and only if you are passionate about science, working with data and are curious. Do not take this way if you are thinking about money or job security ... this field is not so simple and you stay weeks and months experimenting and try to find something in that datasets ...
One example: as data scientist you are mostly starting to look at data and try to find something inside and solve problems, some dta does not provide any info and after weeks of "magic" is put on the side and moved to other project, after a while you returning and look into data to find information on other business problems.

Look at Data Science as pure science and only after that about money, also most of the data scientist's primary goal is not remuneration ... media is promoting salaries but not the real work what we doing ... how much we love data ...


Create a single time series plot of multiple devices I have a dataset where there are stored the measurements of 30 devices. Each device has about 4000 values and it is structured as well:
device01 0.02;0.13;1.15;0.10;8.30;........;4.20
device02 0.06;0.13;1.40;0.03;7.40;........;6.30
........
device30 0.03;0.24;1.10;0.43;4.40;........;2.30

Each value is got each X seconds (I don't have the timestamp reference, I only know they are got through time). How can I plot them to get a single time series plot like the one in this picture?

As you can see lots of the values do overlap, how can I make them consecutive in the plot?

You can create overlapping plots by looping over your dataset and plotting onto the same axes:
ax, fig = plt.subplots()
n = 30 # number of device
for i in (number+1 for number in range(n)):
    ax.plot(data[i], label='device_0' + i)

plt.show()

It might be best to first put your data into a pandas Dataframe.
In regard to the X-axis, you will need to convert the seconds to datetime, by adding an estimate of the starting date to each value in the dataset.


can we make a word2vec NN of more than 3 layers using tensorflow? To the best of my understanding , word2vec crated using gensim is of 3 layers only.
I was wondering can we customize word2vec NN and create word2vec NN of more than 3 layers to experiment with it using tensorflow ?

Technically, any Word2vec is based on an encoder-decoder Neural Network architecture with a hidden layer that learns the word embedding. In theory it is perfectly feasible to implement a "deeper" word2vec model in TensorFlow.

You are going to find some practical problems, though. Word vectors are super long (for a small corpus, we are in the order to thousands). Training a deep Network with, say, 10.000 input nodes is going to be practically infeasible without an appropriate infrastructure.

To overcome this problem, word2vec libraries (such as gensim) recur to negative sampling. The inventors of word2vec (Mikolov et al.) found that the results from training a simple word2vec model can be approximated by a sequence of logistic regressions. You can find more on negative sampling here and on Google.

But if you make the Network deeper, logistic models (and therefore negative sampling) couldn't work anymore, making this task too computationally intensive for normal computers.

In other words: yeah you can create a word2vec Neural Network with more than 3 layers, but the computational power required makes it almost impossible for individual researchers with common hardware.


AttributeError: 'numpy.ndarray' object has no attribute 'columns' import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.feature_selection import SelectFromModel
from scipy.io import arff

data = arff.loadarff("C:\\Users\\manib\\Desktop\\Python Job\\Project Work\\Breast\\Breast.arff")
df = pd.DataFrame(data[0])

df.head()

df["Class"].value_counts()

X = df.iloc[:,:24481].values
y = df.iloc[:, -1].values

from sklearn import preprocessing
label_encoder = preprocessing.LabelEncoder()

y=y.astype('str')

y= label_encoder.fit_transform(y)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=0)

sel = SelectFromModel(RandomForestClassifier(n_estimators = 100))
sel.fit(X_train, y_train)

sel.get_support()

selected_feat= X_train.columns[(sel.get_support())]
len(selected_feat)

print(selected_feat)

The problem is that train_test_split(X, y, ...) returns numpy arrays and not pandas dataframes. Numpy arrays have no attribute named columns

If you want to see what features SelectFromModel kept, you need to substitute X_train (which is a numpy.array) with X which is a pandas.DataFrame.

selected_feat= X.columns[(sel.get_support())]

This will return a list of the columns kept by the feature selector.

If you wanted to see how many features were kept you can just run this:

sel.get_support().sum()  # by default this will count 'True' as 1 and 'False' as 0


Trouble with accuracy of multiclass perceptron I have built a multiclass perceptron, but it has low accuracy (around 80%). I think I'm missing something. One possibility is that I should add a bias, but I'm not sure how to incorporate that.

The task is, given 2 dimensions, predict the class, which is between 0 and 8.

I could use some pointers as to where this code is going wrong. We are not given classes for test data, but rather need to populate it and it is checked elsewhere.

def dot_product(weights_list, training_data):
    return [sum([a * b for a, b in zip(training_data, i)]) for i in weights_list]

def predict(row, weights_list, n_classes):
    # get dimensions * weight
    activations = dot_product(weights_list, row[:-1])
    # get index of argmax
    predicted_label = activations.index(max(activations))
    return predicted_label

def train_weights(train, n_epoch, n_classes):
    # prepopulate just weights
    weights_list = [[0.0 for x in range(len(train[0])-1)] for x in range(n_classes)]
    for epoch in range(n_epoch):
        for row in train:
            actual_class = int(row[-1])
            # argmax from predicting each class
            prediction = int(predict(row, weights_list, n_classes))
            # if incorrect:
            #   lower score of wrong answer by this row's values
            #   raise score of correct answer by this row's values
            if actual_class != prediction:
                weights_list[prediction] = [a - b for a, b in zip(weights_list[prediction], row[:-1])]
                weights_list[actual_class] = [a + b for a, b in zip(weights_list[actual_class], row[:-1])]

    return weights_list

train = [[83.0, -14.0, 6.0],
        [77.0, 15.0, 6.0],
        [93.0, 35.0, 3.0],
        [86.0, -8.0, 6.0],
        [-51.0, -79.0, 1.0],
        [62.0, -73.0, 1.0]]

test = [[36.0, 27.0, -1],
        [6.0, 99.0, -1],
        [-3.0, 16.0, -1],
        [-40.0, -61.0, -1],
        [70.0, 67.0, -1],
        [86.0, -14.0, -1],
        [-92.0, 67.0, -1]]

n_classes = 9
n_epoch = 10000
weights_list = train_weights(train, n_epoch, n_classes)

for row in test:
    prediction = predict(row, weights_list, n_classes)
    row[2] = prediction

Firstly, in all the linear separator algorithms such as linear regression, logistic regression and the perceptron, adding the bias is as simple as adding a feature column consisting of all 1's. Then the third weight that will be trained will act as the bias $b$.

I have some working code for a multi-class perceptron

First let's generate some artificial data with 2 features. Each distribution of points is going to be Gaussian with a given mean and variance described as a list of lists for each dimension under the variable params.

def gen_data(params, n):
    dims = len(params[0])
    num_classes = len(params)

    x = np.zeros((n*num_classes, dims))
    y = np.zeros((n*num_classes,))
    for ix, i in enumerate(range(num_classes)):
        inst = np.random.randn(n, dims)

        for dim in range(dims): x[ix*n:(ix+1)*n,dim] = np.random.normal(params[ix][dim][0],
                                                                        params[ix][dim][1], n)
        y[ix*n:(ix+1)*n] = ix
    return x, y

params = [[[ 5,1],  [ 5,1]],
          [[ 0,1],  [ 0,1]],
          [[2, 1],  [ 2,1]],
          [[-2, 1],  [ 2,1]]]

n = 300
x, y = gen_data(params, 300)

plt.scatter(x[:,0], x[:,1])
plt.show()

Alright so now we have 4 distributions with different labels. Let's split the data for sanctity's sake.

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.33)

Let's train the weights using the training data

def get_weights(x, y, n_epochs, verbose = 0):
    # Append a ones column to the feature for the bias
    data = np.ones((x.shape[0], x.shape[1]+1))
    data[:, 0:x.shape[1]] = x
    # Set the targets as integers for comparison
    targets = y.astype(int)

    # Initialize the weights as a matrix
    # number of classes by number of features
    weights = np.ones((len(set(y)), x.shape[1]+1))
    for epoch in range(n_epochs):
        for i, target in zip(data, targets):
            temp = np.dot(i, weights.T)
            pred = np.argmax(temp)
            # If wrongly predicted update prediction
            if pred != target:
                weights[target, :] = weights[target, :] + i
                weights[pred, :] = weights[pred, :] - i

        if verbose == 1:
            print('Iteration: ', epoch)
            print(weights)
            print('---------------------------------------------')
    return weights

weights = get_weights(x_train, y_train, n_epochs = 30, verbose = 1)

This converges to approximately this

  [[  23.62752045   16.03867499 -111.        ]
  [  -3.96545848  -8.66924406   47.        ] [  -0.94290763   -1.84413793   33.        ]  [ -14.71915434   -1.52529301   35.        ]]

We get an accuracy calculated using the score

def predict(x, weights):
    data = np.ones((  x.shape[0], x.shape[1]+1  ))
    data[:, 0:x.shape[1]] = x
    predictions = np.argmax(np.dot(data, weights.T), axis = 1)
    return predictions

def score(x, y, weights):
    pred = predict(x, weights)
    return sum(pred == y_test)/len(pred)

score(x_test, y_test, weights)

  0.8686868686868687

We can check our results using a confusion matrix. For the training set

from sklearn.metrics import confusion_matrix

predictions = predict(x_train, weights)
plt.imshow(confusion_matrix(y_train, predictions))
plt.show()

And the testing set

predictions = predict(x_test, weights)
plt.imshow(confusion_matrix(y_test, predictions))
plt.show()

So we see that in fact our algorithm is performing quite well.

We can then plot our points to see how it is classifying them. I will plot the training points as small circles, and the testing points as larger ones. The dark points are those which are misclassified

colors = ['y', 'r', 'b', 'g', 'k']

# Predict training set
predictions = predict(x_train, weights)
for i, t, p in zip(x_train, y_train, predictions):
    if t == p: plt.scatter(i[0], i[1], c=colors[int(t)], alpha = 0.2, s=20)
    else: plt.scatter(i[0], i[1], c=colors[int(t)], alpha = 1)

# Predict test set
predictions = predict(x_test, weights)
for i, t, p in zip(x_test, y_test, predictions):
    if t == p: plt.scatter(i[0], i[1], c=colors[int(t)], alpha = 0.2)
    else: plt.scatter(i[0], i[1], c=colors[int(t)], alpha = 1)

# Plot the linear separators
x1 = np.linspace(np.min(x[:,0]),np.max(x[:,1]),2)
x2 = np.zeros((weights.shape[0], 2))
for ix_w, weight in enumerate(weights):
    x2 = 1 * ( - weight[2] - weight[0]*x1) / weight[1]
    plt.plot(x1, x2, c = colors[ix_w])

plt.xlabel('Feature 1')
plt.ylabel('Feature 2')

plt.xlim([np.min(x[:,0]), np.max(x[:,0])])
plt.ylim([np.min(x[:,1]), np.max(x[:,1])])

plt.show()

This code generalizes to the binary classification task as well

params = [[[ 5,1],  [ 5,1]],
          [[ 0,1],  [ 0,1]]]

Stop training on convergence

If you want to stop the algorithm based on convergence you can use a stop criteria. For example you can stop training once every weight in your matrix changes by less than a very small number. The very small number we usually choose is machine epsilon 2.220446049250313e-16, which is essentially zero. Sometimes this requirement is too stringent so it can be replaced by any number of significant values.

Change the get_weights code to include the break criteria as

from copy import deepcopy

def get_weights(x, y, n_epochs, verbose = 0):
    # Append a ones column to the feature for the bias
    data = np.ones((x.shape[0], x.shape[1]+1))
    data[:, 0:x.shape[1]] = x
    # Set the targets as integers for comparison
    targets = y.astype(int)

    # Initialize the weights as a matrix
    # number of classes by number of features
    weights = np.zeros((len(set(y)), x.shape[1]+1))
    past_weights = np.zeros((len(set(y)), x.shape[1]+1))
    for epoch in range(n_epochs):
        for i, target in zip(data, targets):
            temp = np.dot(i, weights.T)
            pred = np.argmax(temp)
            # If wrongly predicted update prediction
            if pred != target:
                weights[target, :] = weights[target, :] + i
                weights[pred, :] = weights[pred, :] - i

        if np.abs(weights - past_weights).all() < np.finfo(float).eps:
            break
        past_weights = deepcopy(weights)

        if verbose == 1:
            print('Iteration: ', epoch)
            print(weights)
            print('---------------------------------------------')
    return weights


Convolutional neural network fast fourier transform I've read that some convolution implementations use FFT to calculate the output feature/activation maps and I'm wondering how they're related. I'm familiar with applying CNNs, and (mildly) familiar with the use of FFT in signal processing, but I'm not sure how the 2 work together

When I think of convolutions, I imagine taking a kernel, flipping it, multiplying (and adding) the elements of the kernel with the overlapping input, shifting the kernel and repeating the process. How does a FFT fit into this process?

By transforming both your signal and kernel tensors into frequency space, a convolution becomes a single element-wise multiplication, with no shifting or repeating.

So you can convert your data and kernel into frequencies using FFT, multiply them once then convert back with an inverse FFT. There are some fiddly details about aligning your data first, and correcting for gain caused by the conversion.

If you have a good FFT library, this can be very efficient, but there is overhead cost for running the Fourier transform and its inverse, so your convolution needs to be relatively large before it is worth looking at FFT.

I have explored this a while ago in a Ruby gem called convolver. You can see some of the code for an FFT-based convolution here and the project includes unit tests that prove that direct convolution gets same numerical results as FFT-based convolution. There is also code that attempts to estimate when it would be more efficient to calculate convolutions directly by repeated multiplications or use FFT-based solution (that is rough and ready guesswork though, and implementation-dependent).


R: Error when using Aggregate function to compile monthly means into yearly means Disclaimer: I'm extremely new to R and have been getting by with using google as my professor.

I have a somewhat large collection of monthly values over a period of several years from several different locations. I am attempting to use the aggregate function to calculate the yearly means for each location so that yearly rates of change can be calculated. However, when I run the code

read_csv_filename <- function(filename){
  ret <- read.csv(filename)
  ret$Source <- filename #EDIT
  ret
}

import.list <- ldply(filenames, read_csv_filename)
by1 <- import.list$Source
by2 <- import.list$Result
by3 <- import.list$Year
Yearly_Mean <- aggregate(import.list, by==list(by1, by2, by3), FUN= "mean")

I get an error like this

> Yearly_Mean <- aggregate(import.list, by==list(by1, by2, by3), FUN= "mean")
Error in by == list(by1, by2, by3) :
  comparison (1) is possible only for atomic and list types

I've spent quite a bit of time looking here and elsewhere for similar issues, but haven't found a case that helped me out at all. Any advice on how to fix this (or a completely new, easier method) would be appreciated.

Thanks!

You can use the group_by() and summarize function from the dplyr package to achieve the above easily:

library(dplyr)
import.list <- ldply(filenames, read_csv_filename)

# No need to create b1, b2 and b3
#by1 <- import.list$Source
#by2 <- import.list$Result
#by3 <- import.list$Year

# here you want to group by the data based on 'Year' as you mentioned you want the yearly mean and you want to calculate mean of 'Result' right?.

import.list %>%
  group_by(Year) %>%
  summarize(Mean = mean(Result, na.rm=TRUE))->Final_Output

The above will group_by the data based on Year and calculate mean of Result column.


How to reverse engineer a logarithmic equation I am trying to reverse engineer the parameters of a human-designed logarithmic equation. Here are the facts:

The equation is of the type a = x * ( y ^ b )

a and b are known, x and y are unknown and need to be determined

I have a table of values giving b as an exact value and a as the rounded integer result. For example, b might be 7 and a might be 68 where the 68 is a value rounded to the nearest integer. The number of values in the table is around 12-15 or so.

x will be some number which is human selected, so it will be a relatively round number to +/- 0.05. So, for example, x might be 53 or 52.75, but it would not be 51.1938434.

y is some positive rational value like "1.093534".

So, given the table of values of a and b and the facts above, how can I do an estimation to determine the most likely values of x and y?

You can determine the most likely values for $\log x$ and $\log y$ by linear regression since your relationship implies:
$\log a = b.\log y + \log x$
So run regression on $b$ as input and $\log a$ as output. The gradient will be $\log y$ and the intercept $\log x$


p-value of binary results I have a list of ages and then a binary value associated with it (e.g. is the person overweight)

I would like to calculate from this the p-value.

I have plotted the values and then calculated the probability that at a certain age the person is overweight, but I don't think this is the p-value.

P-values are used to provide insight into the solidity or likeliness of your results within the framework of a pre-determined hypothesis. Therefore, in order to determine the p-value, first you'll need to identify what you are curious about within the scope of the problem and develop a hypothesis test that mirrors that question.

For example, let's re-frame your data such that we have a clear question that we are trying to answer:

We have a sample of individuals and are given their ages and whether or not they are considered overweight based on their weight being above or below some threshold. Given this data, I would like to know if people who are overweight tend to be of a higher age than people who are not overweight. Given this question, I can then construct the following hypothesis in which $\mu(x)$ is the population mean age of overweight individuals and $\mu(y)$ is the population mean age of non-overweight individuals:

$$H_{0}: \mu_{x} = \mu_{y}$$

$$H_{A}: \mu_{x} > \mu_{y}$$

From there, based on the distribution of your data, you will be able to find the p-value, which is the "probability of observing a result equal or more extreme in your sample, assuming that the null hypothesis is true. If we observe a very small p-value (usually smaller than 0.05), then it is unlikely that our null hypothesis was true. Conversely, with a large p-value, we would "fail to reject" the null hypothesis that the mean ages are equal.


Keras: Merged/Concatenated model perform worst than separate models memes recognition I have a dataset of memes, and I'm trying to predict if a certain meme is sexist or not, using image and text together.

Right now I have two models, a VGG16 fine tuned CNN for images and a LSTM model for text, each of them with Keras.

Both models perform well alone (~0.8-0.9 accuracy), and I'm trying to merge them to see if I can get a better result.

I'm concatenating the output of each model like this:

CNN

input_tensor = layers.Input(shape=(image_size,image_size,3))

vgg_model = VGG16(input_tensor = input_tensor, weights = 'imagenet', include_top=False)

for layer in vgg_model.layers:
    layer.trainable = False

l2_strength = 1e-5
dropout_prob = 0.5

x = vgg_model.output
x = layers.Flatten(input_shape=vgg_model.output_shape[1:])(x)
x = layers.Dense(256, activation='relu', kernel_regularizer=regularizers.l2(l2_strength))(x)
x = layers.Dropout(dropout_prob)(x)
x = layers.Dense(64, activation='relu', kernel_regularizer=regularizers.l2(l2_strength))(x)

vgg_model = models.Model(vgg_model.input, x)

LSTM

input_text = layers.Input(shape=(1, 512))
x = layers.LSTM(32, dropout=0.5, name="LTSM")(input_text)
x = layers.Dense(10, kernel_initializer='normal', activation='relu')(x)

text_model = models.Model(input_text, x)

MERGED

x = layers.concatenate([vgg_model.output, text_model.output])
out = layers.Dense(1, activation='softmax', name='output_layer')(x)

merged_model = models.Model([vgg_model.input, text_model.input], out)

merged_model.compile(loss='binary_crossentropy',
                     optimizer='adam',
                     metrics=['accuracy'])

and I train the model like this:

merged_model.fit(
    [train_image, train_text],
    y = train_y,
    epochs=50,
    batch_size=64,
    validation_split=0.2)

where train_images are the images and train_text are the associated text, and the label are a 1 or 0 if a meme is sexist or not.

The problem is that both train and validation accuracy are always 0.5, with a loss of ~8.0-9.0, and never improve during training, so it's not able to learn, but it's weird seen that the model alone give good performance.

I was wondering if someone encountered the same problem and/or do you have any suggestion on why this happen.

I had the same problem and I solved it adding a BatchNormalization layer.
For example:

# create model ConvLSTM
input_convlstm = Input(name='convlstm', shape=(n_steps, 1, n_length, n_metadata))
branch_convlstm = BatchNormalization()(input_convlstm)
branch_convlstm = ConvLSTM2D(filters=64, kernel_size=(1,3), activation='tanh', input_shape=(n_steps, 1, n_length, n_metadata))(branch_convlstm)
branch_convlstm = Flatten()(branch_convlstm)
branch_convlstm = Dense(128, activation='tanh')(branch_convlstm)

# create model metadata
input_metadata = Input(name='metadata', shape=(21,))
branch_metadata = Dense(neurons[0], activation='tanh')(input_metadata)
branch_metadata = BatchNormalization()(branch_metadata)
branch_metadata = Dropout(dropout_rate)(branch_metadata)

# create final model
concat = Concatenate()([branch_convlstm, branch_metadata])
out = Dense(n_outputs, activation='softmax')(concat)
model = Model(inputs=[input_convlstm, input_metadata], outputs=out)


Policy gradient on data only, without emulators It is too costly for my team to emulate the agent (executing the action and assessing the reward), meaning our only option is to learn the optimal policy on our dataset. The good thing is that we have a lot of data, that represents a sequence of state, action, reward. We can train our agent on this data.

We also need continuous actions, as the set of actions is big. Policy gradient is therefore the way to go but it generally uses an actor-critic that requires an emulator. We can not emulate, what would be the other options?

I think your best approach is to use Imitation Learning. Many techniques in imitation learning use Supervised Learning so you do not need to use emulator. Check DAGGER which is used in continuous action scenarios or the recent AggreVated algorithm (just ignore the theoretical parts of the paper).

As a start you can use Supervised Learning just for experimentation and then use the above algorithms. I would suggest though to use even a poor simulator just to have an idea of how your implementations behave after the. Bare in mind that RL tries to solve an optimization problem (maximize an utility/cost function) whereas the Supervised Learning methods try to optimize the difference between model's prediction and ground truth. Just be cautious of the algorithm's behavior before you go live.


how to improve searching index in dataframe Given a pandas dataframe with a timestamp index, sorted.
I have a label and I need to find the closest index to that label.
Also, I need to find a smaller timestamp, so the search should be computed in the minor timestamps.
Here is my code:

import pandas as pd
import datetime

data = [i for i in range(100)]
dates = pd.date_range(start="01-01-2018", freq="min", periods=100)
dataframe = pd.DataFrame(data, dates)

label = "01-01-2018 00:10:01"
method = "pad"
tol = datetime.timedelta(seconds=60)
idx = dataframe.index.get_loc(key=label, method="pad", tolerance=tol)

print("Closest idx:"+str(idx))
print("Closest date:"+str(dataframe.index[idx]))

the searching is too slow. Is there a way to improve it?

This probably doesn't take into account that dates are sorted and thus performs as O(n). Try using binary-search on dates, that would perform as O(log(n)) (avoid implementing your own, look for a standard module, maybe it exists in numpy).


How to choose the order in which to split a decision tree? I know that a decision tree recursively splits along each attribute, greedily minimizing the wrong classifications/deviance at each split.

But, what is the order in which the attributes are split?  In other words, for a regression tree in N dimensions, what determines which attribute gets split first?

One option is to use a simple approach, like just choosing random attributes, or taking one at a time.

Another option is to use a metric to decide, at each step, the attribute that best splits the data. Different tree algorithms apply different metrics, like for instance Gini impurity (CART algorithm), or information gain (ID3 and C4.5 algorithms).

You can read a brief description of these methods in the "Decision tree learning" entry on the Wikipedia.


training model on random samples from a large dataset I have a huge data set(More than 1 million data points).My dataset is text. i am doing NER on it to identify few entities. if i randomly choose 100 data points from the total data set and train my model(LSTM), will this yield good results? i will be running for 20k random batches. Does this approximate the data properly or do i need to run for more number of batches than the total number of datapoints?

Depends entirely on your data, if your variables are mostly numerical then you can get by with small samples. If however you have a lot of categorical variables you need to make sure that each category of every variable is well represented in the subsample. If they are all numerical I would go for 1000 data points repeated 1000 times.


Neural net without hidden layers should be a simple linear model: why do I get so different results? I’m reading „Elements of Statistical Learning“ where Hastie et al. describe in Section 11.3 on neural nets (p. 394), that (in short) if there are no hidden layers in a neural net (so without non-linear transformation of features), then the model collapses to a linear model.

I tested this in a Keras model with only the output layer included in the model. I noticed that the results (mean absolute error, mae) differ a lot contingent on the choice of the learning rate and the number of training epochs. For some combinations of training epochs and learning rate, I get a mae which is very close to the one of an ordinary least square (OLS) model.

However, in many cases, results are hugely different (and of course worse than under OLS).
I don’t understand why this happens. Can someone point me to an answer?

Here is my playcode (using Boston housing data):

library(keras)
library(tibble)
# Data
boston_housing <- dataset_boston_housing()

c(train_data, train_labels) %<-% boston_housing$train
c(test_data, test_labels) %<-% boston_housing$test
paste0("Training entries: ", length(train_data), ", labels: ", length(train_labels))
train_data[1, ] # Display sample features, notice the different scales

column_names <- c('CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE',
                  'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT')
train_df <- as_tibble(train_data)
colnames(train_df) <- column_names
train_df
train_labels[1:10] # Display first 10 entries

############################################
# Feature engineering

# Test data is *not* used when calculating the mean and std.
# Normalize training data
train_data <- scale(train_data)

# Use means and standard deviations from training set to normalize test set
col_means_train <- attr(train_data, "scaled:center")
col_stddevs_train <- attr(train_data, "scaled:scale")
test_data <- scale(test_data, center = col_means_train, scale = col_stddevs_train)

train_data[1, ] # First training sample, normalized

######################################
# OLS
olstrain = data.frame(cbind(train_labels, train_data))
olstest  = data.frame(cbind(test_labels, test_data))
colnames(olstrain) = c('y', 'CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE',
                       'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT')
colnames(olstest) = c('y', 'CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE',
                      'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT')
ols = lm(y~.,data=olstrain)
summary(ols)
olspred = predict(ols, newdata=olstest)

library(Metrics)
mae(test_labels, olspred)

##################################
# Keras

#lrate=0.02
lrate=0.1

# OLS-like results are sensitive to choice of LR and EPOCHS (both must match)

epochs = 100
build_model <- function() {
  model <- keras_model_sequential() %>%

    # OLSisch
    layer_dense(units = 1, input_shape = dim(train_data)[2])

    # Simple NN
    #layer_dense(units = 64, activation = "relu", input_shape = dim(train_data)[2]) %>%
    #layer_dense(units = 1)

    model %>% compile(
      loss = "mse",
      optimizer = optimizer_rmsprop(lr=lrate),
      metrics = list("mean_absolute_error")
    )

  model
}

model <- build_model()
#model %>% summary()

# Fit the model and store training stats
history <- model %>% fit(
  train_data,
  train_labels,
  epochs = epochs,
  validation_split = 0.2,
  verbose = 1
)

c(loss, kmae) %<-% (model %>% evaluate(test_data, test_labels, verbose = 0))

print("======================================================")
print(paste0("Learning rate: ", lrate))
print(paste0("Keras: Mean absolute error on test set: $", sprintf("%.2f", kmae*1000)))
print(paste0("Keras mean val_mae:                     $", sprintf("%.2f", mean(history$metrics$val_mean_absolute_error[30:100])*1000)))
print(paste0("O L S: Mean absolute error on test set: $", sprintf("%.2f", mae(test_labels, olspred)*1000)))
print(paste0("Diff.                                 : $", sprintf("%.2f", kmae*1000 - mae(test_labels, olspred)*1000)))

I would say your way of comparing this is rather incorrect. There are two main problem from what you did those are :

R linear model (lm) is solved using deterministic solution whereas
keras model optimized with gradient descent. In the end you expect
the weights and result to be close but not similar. For OLS, it
attains the lowest theoretical training error(by definition)
averaged over all training sample. Your keras model will converge
towards this minima but it might have difficulty converging to
similar minima(due to the nature of GD).
you are not training your keras model with full data. Notice you put
validation split to be 0.2 i.e. you are training with 80 percent of
your data.

Finally, if you want to test whether single layer NN is similar to OLS You can try setting validation split to 0 (or simply don't declare it) and training it with the whole dataset as your batch.


Predicting invoice data of 12 month using only 1 month data I have only 1 month of historical data of invoices can I predict next 12 month of data with good accuracy if it is possible then which model should I used for prediction?
Thanks

Since Invoice analysis comes under trend analysis. It's better to take a look at the data from a statistical point of view. Plot graphs, manually check the relation and so on. if there is a trend which is not monotonic, or if it's recurring weekly it's better to take a look at (Recurrent Neural Network)RNN's to predict based on the day or week.


Is there a method that is opposite of dimensionality reduction? I am new to the field of machine learning, but have done my share of signal processing. Please let me know if this question has been mislabeled.

I have two dimensional data which is defined by at least three variables, with a highly non-linear model way too complicated to simulate.

I have had varying level of success at extracting the two main components from the data using methods like PCA and ICA (from the python library Scikit-Learn), but it seems these method (or at least, these implementation of the methods) are limited to extracting as many components as there are dimensions in the data, for example, 2 components from a 2D point cloud.

When plotting the data, it is clear to the trained eye that there are three different linear trends, the three color lines show the directions.

When using PCA, the main component is aligned to one of the color lines, and the other is at 90°, as expected. When using ICA, the first component is aligned with the blue line, and the second is somewhere in between the red and green ones. I am looking for a tool which could reproduce all three components in my signal.

EDIT, Additional info: I am here working in a small subset of a bigger phase plane. In this small subset, each input variables produce a linear change on the plane, but the direction and amplitude of this change is non-linear and depends on where exactly on the bigger plane I am working. At some places, two of the variables can be degenerate: they produce change in the same direction. for example, say the model depends on X, Y, and Z. A change in the variable X will produce a variation along the blue line; Y causes a variation along the green line; Z, along the red one.

The short answer is yes.

Essentially you will be performing some sort of feature engineering. This means constructing a series of functions of your data, often:

$\phi_j(x): \mathbb R^p \rightarrow \mathbb R \ ,\ \  j=1,...,K$

Which, strung together, define a transformed data vector $\boldsymbol \phi(x)$ of length $K$.

There are a number of ways, better and worse, of doing this. You may want to look up terms like:

Splines and generalised additive models.
The kernel trick (how to make a model where $K\rightarrow \infty$).
Feature engineering (of the manual variety, e.g. adding an $x^2$ column to your data).
Deep learning, representation learning

As you might guess from such a varied bag of techniques, this is a large area. It goes without saying really but care has to be taken to avoid overfitting.

This paper Representation Learning: A Review and New Perspectives deals with some of the issues around what makes a particular set of features 'good', from a deep learning perspective.


Aspect based sentiment analysis using machine learning approach I am very new in machine learning. I have annotated data with category, aspect, opinion word and sentiment. for example, for the bellow text

"The apple was really tasty"

I have category->food, aspect-> apple, opinion word ->tasty and sentiment->positive. I have training data like this format.

How can I train a SVM classifier using this format of training set?
How to extract features like n-gram, POS and sentiment word to train the classifier?
Could you please suggest any beginning step for this aspect based sentiment analysis using machine learning algorithms?

Aleksandar Blekh has given some really nice links about the big picture of how to do sentiment analysis; I'll try to provide some links to software and talk about the nitty-gritty of how to make it work. I'll point you to the example using scikit-learn (http://scikit-learn.org/stable/ ), a machine learning library in Python.

You would first want to take your dataset and load it into scikit-learn in a sparse format. This link (http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html ) gives examples of how to load text in a bag-of-words representation, and the same module (scikit-learn.feature_extraction.text) can also count n-grams. It then describes how to run Naive Bayes and SVM on that dataset. You can take that example and start playing with it.


Confusion with euclidean distance implementation I have to implement the k means algorithm from scratch using python on this data set that has 29 columns and 476 rows. With all these different data points I am confused on how I can calculate the distance between the centroids and the data in order to make sure the data is in the right cluster. I know I have to use the euclidean distance, but I am not sure how I can use it with this data set. All the examples I've seen online usually deal with something that is only 2 dimensional. I need someone to explain to me what kind of approach I should take for this. This is the dataset:

First of all, you need to convert your categorical columns to numerical if you like to take them into account (e.g. column $T_m$).
You need to normalize your features as there is variance in scale (e.g. column $G$ vs $BLK$)
You probably need to reduce dimensionality as k-means MIGHT get stuck in Curse of Dimensionality.

The rest is fine. Euclidean distance generalizes for any number of dimensions.
$$\sqrt{(x_{1} - x_{2})^2 + (y_{1} - y_{2})^2 + (z_{1} - z_{2})^2 + ...}$$

So, when you calculate your k-means, you will get $k$ points each of them in $d$ dimensions ($d$ is the number of columns). Then for each data point you calculate the distance above for data point and the center i.e.

$$\sqrt{(x_{data} - x_{center})^2 + (y_{data} - y_{center})^2 + (z_{data} - z_{center})^2 + ...}$$

Python does it for you easily.


Getting 0 accuracy and NaN mae for all epochs training my NN Background:
I made a simple game using python library 'Turtle' in which there is a long plank with a ball balanced on top of it. I can press right or left arrow keys to rotate the plank (either clockwise or anticlockwise) which makes the ball roll to the either side.
Issue:
I wanted to train a neural network to make this whole process automated(to balance the ball on plank). So I myself played the game and recorded the states(conditions) and my corresponding actions(buttons pressed) for a period of 2 minutes in the form of a csv. At the end, I got a dataset of 1300 conditions( eg. velocity of ball, direction of velocity, inclination of plank, etc) and the corresponding 1300 actions i took to balance the ball on the plank. Now when I try to train my neural network on this data, I get 0 accuracy and 'nan' mae for all the epochs. How do I resolve this?
I've attached the code below
#all necessary imports

df=pd.read_csv('train.txt')

model= Sequential([
               Dense(units=16,activation='relu',input_shape=(5,)),
               Dense(units=32,activation='relu'),
               Dense(units=10,activation='relu'),
               Dense(units=1,activation='softmax')
            ])

model.compile(optimizer=Adam(learning_rate=0.0001),loss='sparse_categorical_crossentropy',metrics=['accuracy','mae'])

X_train=df[['x','y','theta','u','v']]
Y_train=df['action']

X_train= X_train.astype('float32')/10
Y_train= Y_train.astype('float32')/10

model.fit(X_train,Y_train,batch_size=40,epochs=20)

And this is what I'm getting for all the epochs :
loss: nan - accuracy: 0.0000e+00 - mae: nan

Maybe I'm not interpreting this correctly (I don't use Keras), but why does your softmax layer only have a single unit? The softmax function normalizes a vector to represent a probability distribution: $ f : ℝ^n \rightarrow (0,1)^n  $ such that $\sum f(\vec x)=1 $. I would have assumed the softmax layer to use the same number of units as the previous, but again, maybe I'm not understanding Keras.
Usually, gradient explosions cause nan. If you are unfamiliar with it, you could read about the related vanishing gradient problem here. However, in your case, I think the final layer is more suspicious.


Variation in output of Logistic Regression when using SMOTE I am working on a logistic regression case with an imbalance in the target variable. To fix this I am using SMOTE (Synthetic Minority Oversampling Technique), but each time I run my regression model, I get different numbers in my confusion matrix. I have set random_state parameters while invoking SMOTE as well as Logistic Regression still to no avail. Even my features are the same in each iteration. I was able to get the best value for recall as 0.81 and AUC value as 0.916 once but they are not coming anymore. On some occasions, the value of False Positives and False Negatives shoots up very much indicating that the classifier is very bad.

Please guide what I am doing wrong here, below is the code snippet.

# Feature Selection
features = [ 'FEMALE','MALE','SINGLE','UNDER_WEIGHT','OBESE','PROFESSION_ANYS',
            'PROFESSION_PROF_UNKNOWN']

# Set X and Y Variables
X5 = dataframe[features]

# Target variable
Y5 = dataframe['PURCHASE']

# Splitting using SMOTE
from imblearn.over_sampling import SMOTE
os = SMOTE(random_state = 4)

X5_train, X5_test, Y5_train, Y5_test = train_test_split(X5,Y5, test_size=0.20)
os_data_X5,os_data_Y5 = os.fit_sample(X5_train, Y5_train)
columns = X5_train.columns

os_data_X5 = pd.DataFrame(data = os_data_X5, columns = columns )
os_data_Y5 = pd.DataFrame(data = os_data_Y5, columns = ['PURCHASE'])

# Instantiate Logistic Regression model (using the default parameters)
logreg_5 = LogisticRegression(random_state = 4, penalty='l1', class_weight = 'balanced')

# Fit the model with train data
logreg_5.fit(os_data_X5,os_data_Y5)

# Make predictions on test data set
Y5_pred = logreg_5.predict(X5_test)

# Make Confusion Matrix to compare results against actual values
cnf_matrix = metrics.confusion_matrix(Y5_test, Y5_pred)
cnf_matrix

I do not have enough reputation to make a comment.

For each run, you use different data for training and testing. To obtain stable results you should also fix the partition of the data into train/test set by providing random_state for the train_test_split:

X5_train, X5_test, Y5_train, Y5_test = train_test_split(X5,Y5, test_size=0.20, random_state=4)

A completely different question, however, is why with a small/large? change in training data you have drastically different classification performance.


Parzen and k nearest neighbor I have this formula for the density estimation.
$$p_n(x) = \frac{k_n / n}{V_n}$$

I have been told that with a Parzen window approach you can specify $V_n$ as a function of $n$. So if $V$ decreased when $n$ increased it is clear that it is a fixed volume.

I have also been told that with a knn approach you specify $k_n$ as a function of $n$. So if you increase as $n$ is raised, it is clear that the volume is dependent of the volume.

So can anyone explains me the above statements. I think it is somewhat clear to me how knn and Parzen works. (knn count the $k$ nearest neighbor, at the new sample is assigned to the class which has most votes. In Parzen the volume is fixed).

I also do not understand the two formulas in the figure. The figure illustrated two methods for estimating the density at a point $x$ at the center of
each square. The top knn, bottom Parzen

The probability that a vector $x$ is drawn from $p(x)$ in some region $R$ of a sample space is given by $ P = \int_{R} p(x')dx'$.  Given a set of N vectors drawn from the distribution; it should be obvious that the probability k of these N vectors fall in $R$ is given by $P(k) = \binom{N}{k} p^{k} (1-p)^{N-k}$.  From the properties of a binomial p.m.f the mean and variance of the ratio $\frac{k}{N}$ are ${E}[\frac{k}{N}] = P$ and ${var}[\frac{k}{N}] = \frac{P(1-P)}{N}$.  Therefore, as $N  \rightarrow \infty$ the distribution becomes more defined and the variance smaller.  Hence, we can expect a decent estimate of the probability P to be obtained from the mean fraction of points that fall within the region $R$.  Hence $P \cong \frac{k}{N}$,

Now consider if the region $R$ is small such that $p(x)$ does not vary considerably within it, then $\int_{R} p(x')dx' \cong p(x)V $.  Combining this result with the one above.  We see that $p(x) \cong \frac{k}{NV}$.

That's where the formula you found basically comes from.  Therefore if we want to improve $p(x)$ we should let V approach 0.  However, then $R$ would become so small that we would find no examples.  Thus we really only have two choices in practice.  We have to let V be large enough to find examples in $R$ or small enough such that p(x) is constant within $R$.

The basic approaches include using KDE (parzen window) or kNN.  The KDE fixes V while kNN fixes k.  Either way, it can be shown that both methods converge to the true probability density as N increases providing that V shrinks with N and that k grows with N.

The formulas used in the picture are just arbitrary examples that fulfill this requirement.


Cluster directed graph into DAG I am looking for an algorithm that clusters a directed graph into a set of clusters that form a directed acyclic graph.

For example, given:
nodes: {A, B, C, D}
with edges: {(A,B), (B,A), (A,C), (C,D), (D,C)}

A valid output would be: {{A,B},{C,D})
An invalid output would be: {{A,C},{B,D}} (since the node edges imply a cycle between the clusters)

Thanks for any pointers or suggestions!

What you are exactly looking for is a modification to the DEDICOM Algorithm (page 4). the DEDICOM itself gives you measure for relation between different components of a directed graph. You just need to be a bit creative to use it for converting a graph into DAG. Read the paper and if further help needed just drop me a comment.


How to create a parquet file from a query to a mysql table Updating a legacy ~ETL; on it's base it exports some tables of the prod DB to s3, the export contains a query. The export process generates a csv file using the following logic:

res = sh.sed(
    sh.mysql(
       '-u',
       settings_dict['USER'],
       '--password={0}'.format(settings_dict['PASSWORD']),
       '-D', settings_dict['NAME'],
       '-h', settings_dict['HOST'],
       '--port={0}'.format(settings_dict['PORT']),
       '--batch',
       '--quick',
       '--max_allowed_packet=512M',
       '-e', '{0}'.format(query)
    ),
    r's/"/\\"/g;s/\t/","/g;s/^/"/;s/$/"/;s/\n//g',
    _out=filename
)

the mid term solution with more traction is AWS Glue, but if I could have a similar function to generate parquet files instead of csv files there would be much needed big short term gains

I can think of a few ways besides using Apache Spark.

There is a python script doing just that: https://github.com/rondunn/odbc2parquet. It utilizes pyodbc though.
Depending on your use case you may find that https://github.com/blue-yonder/turbodbc would deliver a much speedier experience querying the data. turbodbc can also directly emit arrow arrays.
There is a command line tool written in Rust by yours truly. It is also called odbc2parquet. https://github.com/pacman82/odbc2parquet


Churn definition for non-contractual services I am computing churn definition i.e, the number of days after which we will say a customer has churned in fashion retail, etc. Currently, I am using the transaction dates to get the average days between two purchases of each customer, other stats such as number of days he has been customer.
If anyone has any idea on this , kindly give some pointers.

Perhaps you can define an "active user" (e.g., someone who has made a purchase in the last 3 months, etc. Choose a period sensible for the typical purchase cycle) and calculate the something along the line of DAU or MAU, etc. Say, if a typical active user will on average buy something from your store every 3 months, assume a user has churned if she fails to make a purchase in the last avg + 1sd days, etc.?

The other potential KPI to use is probably the CLV, but similar to DAU and MAU, it is not without its criticisms.


Evaluating loss for non classifying convolutional neural network Sorry if my question is kind of dumb, I am very new to this field.

I am trying to create a CNN that plays a variant of chess (for the examples, we'll use chess as it is close enough).

My network , which is a policy network, outputs a vector of planes of probabilities. (e.g 1st pawn layer, 2nd rook layer etc..) each layer contains scalars determining how "good" would a move of the piece to this case be, according to the network. My question is, given the input channels I. And expected outpus (0s everywhere and 1 for the move that was played), how to calculate loss for gradient descent ? (e.g : how to evaluate the "closeness" of a move to another.).

My wild guess is to weight each layer a "wrongness" factor (e.g if instead of wanting to move the first pawn , it tries to move the queen , it is labelled as really "wrong" and then apply some kind of spacial locality (e.g if it doesn't move the first pawn but the second and on the very right case, it is not very wrong). But is it correct ? And in general , how to compute the loss of a non classifying convolutional neural network ?

My wild guess is to weight each layer a "wrongness" factor (e.g if instead of wanting to move the first pawn , it tries to move the queen , it is labelled as really "wrong" and then apply some kind of spacial locality (e.g if it doesn't move the first pawn but the second and on the very right case, it is not very wrong). But is it correct ?

You can relatively simply teach a policy network to predict human-like moves in your scheme using a database of moves. Actually your "wrongness" is probably well-enough represented by classification (your positive class might be "This is a good move") and the usual log loss that goes with it.

After you have trained a policy network, you will want to look in depth at the literature for game-playing bots. Your policy network might work quite well alongside Monte Carlo Tree Search, provided you have some kind of evaluation heuristic for the resulting position.

A reinforcement learning approach to learn from self-play would take you further, enabling the bot to teach itself about good and bad moves/positions, but is too complex to explain in an answer here. I suggest look into the subject after training your network and seeing how good a player you can create using just the policy network and a move search algorithm.

  And in general , how to compute the loss of a non classifying convolutional neural network ?

There are a few common options available for regression, such as mean square error (MSE) i.e. $\frac{1}{2N}\sum_{i=1}^{N}(\hat{y}_i - y_i)^2$ where $\hat{y}_i$ is your prediction and $y_i$ is the ground truth for each example. If you use this loss function, and want to predict values outside of range 0-1, remember to use a linear output layer (i.e. no activation function after the last layer), so that the network can actually output close to the values you need - that's about the only difference in network architecture you need to care about.

In the more general case of game-playing bots, it is usual (but not required) to calculate a predicted "return" or "utility" which is the sum of all rewards that will be gained by continuing to act in a certain way. MSE loss is a good choice for that. Although for zero-sum two player games where the reward is simply win/lose, you can use a sigmoid output layer (predicting chance of a win) and cross-entropy loss, much like a classifier.

For your specific case, you can treat your initial policy network as a classifier. This immediately gives you some probability weightings for the predicted move, which you can use to pick the predicted best play, or maybe to guide Monte Carlo Tree Search.

The kind of network that predicts utility or return from a position (or combination of position and action) is called a value network (or an action-value network) in reinforcement learning . If you have both a policy network and a value network, then you are on the way to creating an actor-critic algorithm.


Cluster Similar Images into Folders I wasn't sure where to ask this question so forgive me if the question seems out of place (and please guide me where to ask it !)

I have an archive of 9GAG images and I want to Cluster them based on their content and their similarity... 9GAG images are mostly memes so it's natural that you'd find many of them pretty similar to each other...

I couldn't find any application which does this out of the box (if there is, could you please refer me to it ?) and what I found was a vast number of Papers about Image Clustering but no real application based on them...

I was wondering if there is a Ruby, Python or Java Program which could simply get the directory of the Images and Cluster them into groups (folders) based on their Similarity to each other ?

Thank you very much...

I would impressed if there is already a dedicated 9gag clustering :P

However, you can read this blog post about Hierarchical clustering in Python for images, which is close to what you want. The problem is that the author uses the average color of the image as a feature and it could proved crude and not inefficient. You may find something more interesting to use. But in the end, you need to experiment a lot with your own dataset.


What is the difference between concept class and hypothesis Formal definition that I have seen of concept class is

class of all true functions

mathematically :
$f:X \rightarrow\{0,1\}$
and that of hypothesis is:
$h:X \rightarrow\{0,1\}$
But most of the times they are used together. For example in definition of PAC

A concept class  is PAC learnable by a learner  using hypothesis space  if for all concepts ∈, distributions over , true error probability 0≤≤1/2, failure probability 0≤≤1/2, learner  outputs a hypothesis ℎ∈ such that
True error less than or equal to
Computational time is polynomial in 1/,1/, representation size of data object, and representation size of concept

What is the difference?

If one requires that $H = C$, then this is called the "proper PAC" framework compared to "PAC prediction" where we don't care about the representation of $h$ as long as the prediction error is small enough (i.e. we allow $H$ to be the class of all time-polynomial programs).

You can think of a concept as the set of inputs that produce the same label (e.g. all the images that show a chair form the concept "chair" or all the points in the same half space form the concept "true/false").


Chi square distribution for feature selection In one paper on ML I read that chi square distribution is used to reduce the number of features. In that paper, features are words. That paper is related to Sentiment Analysis, so we have "positive", "negative" and "neutral" category.

How to calculate chi square distribution in that case?
In Python there is scipy.stats.chisquare which gives chi_square value and a p_value. What do we do then with these two pieces of information?
What to do for example with word "good" as a feature?
How to calculate chi square distribution, and what to do with that?
What does it mean to exclude some feature from the set of features, because in that paper it is mentioned that we take n of them with top chi square.

I really don't know how to do it. If there is any paper or book or link to learn that, please tell me.

There are different ways for feature selection. A very good read in machinelearningmastery, to recap:

Univariate Selection.
Recursive Feature Elimination.
Principal Component Analysis.
Feature Importance.

Chi-Squared test For Feature Selection goes under the Univariate Selection method for non-negative features. My favorite explanation of chi-Squared in one photo taken from this blogpost is:

As you can see scikit-learn has an implementation for feature_selection using chi2 (perhaps according to scipy.stats.chisquare) as was shown very briefly in the above-mentioned blog post.

If you want a more thorough explanation and details how test ranks features based on statistics according to chi2 distribution and p-value etc., and also how to build your own chi2 class for feature selection in Python see this great post. Obviously one can read about the basics of chi2 distribution and test in wikipedia.


VC-dimension of the infinite set of convex bodies? I try to find and prove the VC-dimension of the infinite set of (uni-directional) convex bodies. From intuition, it's clear to me that it goes to infinity, but I don't know the correct way to prove it formally.
Any clue for direction?

For a binary class of data and any number of instances, as you have an infinite number of convex bodies, you can put all instances of a class inside the body and leave others. In this way, you can shatter any set with any arbitrary size. Hence, the VC dimension of an infinite number of convex bodies is infinite.


Population stability Index vs Population Accuracy Index Can anyone explain to me the difference between Population Stability Index(PSI) and Population Accuracy Index(PAI)?

Population stability refers to whether the  the distribution
of explanatory variables is changing over time. When this distribution changes  there is more concern over whether the model is currently fit-for-purpose since the data used to develop the model differs from the data the model is being applied to.
The Prediction Accuracy Index (PAI) is defined as the average variance of the estimated mean response at review divided by the average variance of the estimated mean response at development. In PSI we don't take into account the predictive ability.
To summarise,the Population Stability Index (PSI), interprets whether the distribution of development data and the distribution of review data are similar.  The PAI checks the predictive ability of the model on current data is not significantly worse than the predictive ability at development.
To know more refer this paper


Approaching data science I'm totally naive to Data Science - that is, the relatively new, somewhat hyped-field that is so popular at the moment. But I'm not naive to data ... as a scientist and researcher I've worked with all sorts in different roles in the past.

Now I'm in the lamentable position of having dug a lot of shallow holes, using different software systems and different types of data, and not really being professionally competent in anything.

My question is, if I want to "get up to speed" with data science, and perhaps leverage the different experiences I've had, how do I approach it? Ideally I'd like to make my research skills marketable - that is, become a data scientist of sorts, but with a greater emphasis on the research/reporting side.

Assuming I'm coming from scratch but have demonstrated capacity - I say that because, for example, I've used R before for some projects but after a break of a year or so I need to relearn it every time... Where do I start; how do I unify all these bits and pieces?

And what claim can I make to work in this field? (I've worked on all sorts of data, from gigabytes of climate data and earth science, to health registers to longitudinal surveys ... but none of it under the moniker of a data scientist).

Specifically, what tool(s) do I learn and what theory do I need to grasp? (Keeping in mind that all my coding and statistical competencies are mostly self-taught.)

Unlike this (fascinating) question, I don't have a business background and don't necessarily want to move towards the business analyst path - I still want to play with psychical (earth science) or social data. Neither do I want to work on the data management side so much - I want databases and coding to be a means, not an end. And finally, I'm not inclined much towards the theory and mathematics. Perhaps the best way to summarise my inclination and position is that I don't want to become a data science expert, but want to be able to become an expert in given subjects through data science.

My inclination is perhaps to concentrate on something like Python, and use it to exploit R and other functionality?

Tools I've used in the past (in order of exposure) -

SAS (for statistics and research, not the warehouse side)
VBA/VB6/Excel/Access (data manipulation, reporting)
GIS (ArcGIS for analysis/research, not database management)
R (stats...)
Some HTML/JS
Some Python

One thing I find is that my existing competencies don't provide me with a useful tool for bringing together different data and getting it into the state I want for analysis (ETL I suppose?), hence the inclination to re-learn Python.

Thanks for your thoughts!

The Udacity Data Analyst Nanodegree gives a very gentle introduction using Python and R for mostly exploratory Data Analysis which I think is what you are looking for. (It's not tailored towards business analysis. The courses are free and you can just take the ones you think are interesting. In your case, I would skip the ones about data visualization with javascript and data wrangling with MongoDB.) THey provide a bunch of resources to go from there. Of course there are plenty of other online courses.

If you are more into books you could check out Tukey's Exploratory Data Analysis. A classic that is still highly relevant.

In terms of tools, the most commonly used are R and Python and I would focus on them for now.


ways to represent document by its keyword vectors I have documents, say for example, D1, D2, D3... Dm.

Every Di has its individual components or keywords k1, k2, k3,... kn, where ki is an n-dimensional vector. The number of individual components varies between documents.

What are the ways to find how close Di are? Or what is the best way to represent a document using its keyword vectors? Please note that I'm using a custom embedding here.

If you have the vectors for your keywords, you can aggregate those to get the document vectors. The simplest(and probably one of the most effective) way is to average out your keyword vectors to form the document vector. Once you have the vectors for each of the documents, you can use similarity measures like cosine similarity to see how close your documents are.
A typical example of such a method is this :

Edit: Another interesting point would be to evaluate how you are getting your word vectors. Please look at pretrained word embeddings like word2vec, Glove or Fast Text.


How to investigate what has been learnt in CNNs? My CNN was trained using close-up images of dogs. For testing, we input an image with a dog next to its owner and it was classified as a dog. How is that possible?
Isn't this image a different vector?

There is a famous paper about tracking what a CNN network has learnt. You can visualise to see which parts are more engaged in the classification using DeConvNet. In this paper, it was officially observed that first layers attempt to find simple lines and edges while deeper layers try to put the previous things together to make abstract concepts, like mouth, eye and such meaningful things. As an example take a look at the following image:

I guess there are implementations of this paper that you can replace your pre-trained model with the one already exists and see what exactly is learnt by your network.

  Isn't this image a different vector?

They are different vectors. ML and DL models are for generalisation which means they should be good at test time.


What are the limitations while using XGboost algorithm? Will XGBoost pose any problem while dealing with categorical variables with more than 2 levels.
For example, occupation variable can have values like doctor, engineer, lawyer, data scientist, farmer e.t.c.
If so what would be a better method to use in that case?

I think you should be more specific about what you mean by "fail". As an example, a practitioner could consider an xgboost model as a failure if it achieves < 80% accuracy.

Nevertheless, there are some annoying quirks in xgboost which similar packages don't suffer from:

xgboost can't handle categorical features while lightgbm and catboost can.
xgboost can be more memory-hungry than lightgbm (although this can be mitigated).
xgboost can be slower than lightgbm.


Creating better features for clustering I am trying clustering for the first time trying to separate my user into three categories (or the categories I though that they will should fall in).

First of all I have two tables that describe states and times.
The state table looks like that:

    [[7059992, 7082352, 7083172, 7085480, 7095742],
     [7443343, 7453591, 7463815, 7474055, 7484295],
     [3122958, 3133217, 3143449, 3153658, 3163918],

each row is a user going through some states. This type of features can not be used for clustering directly so I had to create some features.

What I did was to mark the no change between states with a zero
and any change with a timestamp between the last step when a transition was made.

so this gave me a table like that

    [ 3205.,     0.,     0.,     0.],
    [    0.,     0.,     0.,     0.],
    [10005., 16620.,  4626., 26503.],

so user 1 goes through 3205,0,0,0 which means that at the beginning there was a transition that happened 3205ms time before the last change.

user 2 goes through 0,0,0,0 that means that no change happened

and user 3 goes through 10005, 16620,  4626, 26503 which means that the user went through four changes with their respective timestamps.

I thought then that these can be now features for clustering and I tried k-means to get some labels. (I also did normalization to reduce the effects of zero values). Last column is the assigned cluster id.

 0.567415  0.347368  0.673178  0.185788         1
 0.000000  0.000000  0.000000  0.000000         0
 0.413984  0.000000  0.000000  0.439690         2

I want my clustering technique to try to find three categories that have these distinct characteristics.

First class: no transitions, pure zeros
Second class: few transitions (one or max two nonzero values, but each of them is a large value.)
Third class: many transitions (three or four non zero values, but small values for each of them).

K-means seems to merge many time second and third class since the zeros and the large values seem many times the same with many transitions and small values.

If there any clustering technique that can work better with this type of data?
If not can I also denote the no transition instead of 0, with some other value, like NaN? That might help to differentiate better the second and third class.

Setting missing values to 0 does not work well. Because this causes huge deviations.

K-means cannot handle NaN values well, at least in the standard implementations. You could remove the all zero vectors, and then use a nan-toletant implementation, but this will likely also cause undesired effects. Because the absence of a transaction is not the same as an unknown timestamp.

In the end, I believe you have not understood your problem yet. K-means likely is the wrong tool, so it does not matter how you prepare the data - it's answering a different question, but you haven't understood what you are asking yet (and no, "what are the clusters" is not a well defined question - phrase it mathematically, then you have at least a formal version of the question).


Missing Categorical Features - no imputation I've been reading about how to approach missing categorical features in test data, and the most common approach is to use imputation - for example using the last known value or getting the majority feature in given row/column.

Is there a better way to approach missing data? Why can't the classifier just ignore the missing feature, and rely on the known features? Why is imputation necessary?

I'm using scikit learn, and am trying to feed in NaN to a classification model (naive bayes, logistic regression, decision tree, random forest), to see what happens.

Certain models are able to deal with missing values 'naturally', like certain tree based models. Most models however are just a mathematical function which is shaped after the training data. A very easy example would be:

$$f(x) = \alpha x_1 + log(2x_2)$$

What do you do if one of them is NaN? The function is undefined and no prediction can be made. By imputing the values you make a reasonable guess where this sample lies on the data manifold.


Is it good practice to always remove highly correlated variables? 1- Would it always be beneficial to remove highly correlated features prior to training a model? If not, why not.

2- Would you perform One Hot encoding where applicable, prior to removing highly correlated features ? (taking into account a dummy variable may be highly correlated with another variable)

Thanks

(1) No. For example, if you have computer vision problems, then each pixel of the image is a feature. most neighboring pixels are highly correlated.

For example, see the following image from The HASYv2 dataset:


Why should I normalize also the output data? I'm new to data science and Neural Networks in general.
Looking around many people say it is better to normalize the data between doing anything with the NN. I understand how normalizing the input data can be useful.

However I really don't see how normalizing the output data can help.

I've also tried both cases with a easy dataset, and I achieved the same results. The only difference is that in some weird problems, it is really hard to then re-convert the output back.

Can you give me some intuition on why we should also normalize the output?

Or maybe why it is indifferent?

That paper gives a nice answer, where i quoted from. Search for Should I standardize the target variables (column vectors)? in that page.

  Standardizing target variables is typically more a convenience for getting good initial weights than a necessity. However, if you have two or more target variables and your error function is scale-sensitive like the usual least (mean) squares error function, then the variability of each target relative to the others can effect how well the net learns that target. If one target has a range of 0 to 1, while another target has a range of 0 to 1,000,000, the net will expend most of its effort learning the second target to the possible exclusion of the first. So it is essential to rescale the targets so that their variability reflects their importance, or at least is not in inverse relation to their importance. If the targets are of equal importance, they should typically be standardized to the same range or the same standard deviation.


How to properly compare these two confusion matrix? I have used Vader, a sentiment analysis tool for social media, on a database of movie reviews. These two confusion matrices differ in the vader.py algorithm, as the first one is from nltk:

The second one is deriving from Vader's original code on github and includes fixes to negation words, etc.

I was wondering how could I properly compare the two, as I'm not really able to read them. It seems there is not a big difference between them and I don't understand what could be the sources of the errors here.

First, about interpreting these confusion matrices: the sum of every row is 1, which implies that every value is a conditional probability p( predicted label | true label ), i.e. the probability of a given true label to be a particular predicted label. Example: the top left cell in both matrices is 0.01, which means that when the true label is 5 the probability that the system predicts label 1 is 1%.
The two confusion matrices show the results of the predictions by two different systems. This means that in general there is not one which is correct and the other which makes errors, just two different ways to predict labels which might lead to different types of errors.

The diagonal shows the True Positive cases, i.e. cases where the predicted label is the same as the true label (this is important since these are the "correct" cases). The probabilities show very little difference between the two systems on the diagonal, the top one being slightly better for labels 3 and 4 and the bottom one slightly better for label 5.
In general it's important not to look only at the diagonal especially for ordinal values, because if the predicted value is far from the true value it's a more serious error than if it's close to it. Example: if the true value is 4 it's better for a system to predict 3 than 1 (both are errors but the latter is worse). However here again there is very little difference between the two systems, they appear to have a very similar behaviour.

In order to quantitatively summarize and compare the performance of the two systems, confusion matrices are too complex. Typically one would use an appropriate evaluation measure, for instance micro or macro f-score (classification evaluation) or Mean Absolute Error (regression evaluation).


Using the Datumbox Machine Learning Framework for website classification - guidelines? A short while ago, I came across this ML framework that has implemented several different algorithms ready for use. The site also provides a handy API that you can access with an API key.

I have need of the framework to solve a website classification problem where I basically need to categorize several thousand websites based on their HTML content. As I don't want to be bound to their existing API, I wanted to use the framework to implement my own.

However, besides some introductory-level data mining courses and associated reading, I know very little as to what exactly I would need to use. Specifically, I'm at a loss as to what exactly I need to do to train the classifier and then model the data.

The framework already includes some classification algorithms like NaiveBayes, which I know is well suited to the task of text classification, but I'm not exactly sure how to apply it to the problem.

Can anyone give me a rough guidelines as to what exactly I would need to do to accomplish this task?

You can make use of the text classification class for your task.first make sure how you are going to classify the websites.(i.e as sports site,health and wealth site etc).Get some training data and train them & Done


Imbalanced dataset: how to deal with test data? I plan to use many methods to solve the imbalanced dataset problem on the training set. But I couldn't find any paper that describes how they dealt with the test dataset? I assume that they just tested on the original dataset without any adjustments? Will I need to adjust the threshold on the test set with original imbalanced ratio?

update. Thanks for everyone's response!
I've found a paper that discusses how we should adjust the posterior probability and threshold.
Calibrating Probability with Undersampling
for Unbalanced Classification
However, I also believe that in practice we can just fit test set directly.

You should use the testing set without any change, as answered by others. But it is very important to understand the difference between average accuracy and overall accuracy.
In overall accuracy you find ( number of samples predicted correctly/ total number of samples) in average accuracy, you find the overall accuracy per class and then you find the average of these overall accuracies.
When you know that you are working with imbalanced database, where all classes are important, you should use the average accuracy

To understand what  this means: imagine you have two classes, class A and class B , and the ratio is 90 to 10 .
If you are sampling randomly for the training and testing, then the ratio is still 90:10 in the testing set. If your model is very biased , that predicts all the samples to be class A , then:
Overall accuracy = 90%
Average accuracy = 50 %
( 100% for class A + 0% for class B) / 2

The overall accuracy is really high but it does not reflect the actual quality of the model. The average accuracy gives you a better indication of the quality


What is the difference between reconstruction vs backpropagation? I was following a tutorial on understanding Restricted Boltzmann Machines (RBMs) and I noticed that they used both the terms reconstruction and backpropagation to describe the process of updating weights. They seemed to use reconstruction when referring to the links between the input and the first hidden layer and then backpropagation when referring to the links to the output layer.
Are these terms used interchangeably or are they different concepts?

Reconstruction is used for the concept Restricted Boltzmann Machine (RBM), it describes a phase where the structure reconstructs (generates) visible samples from the hidden states of the layers. For more detail, you can refer to:
https://stackoverflow.com/questions/4105538/restricted-boltzmann-machine-reconstruction

Backpropagation is something different entirely; you can find backpropagation in Deep Neural Nets, Convolutional Nets, RBMs (in some way) and so on. Assume a Deep Neural Net with N hidden layers. Upon training we feed the input forward through the randomly initialized weights up to the last neuron. From the output of the last neuron, we calculate our loss using a cost function that calculates the error between predicted and the true output. This is forward propagation.

Knowing the loss and the loss function, we start to take derivatives backwards to find gradients of weights and biases of each layer using the chain rule, all back to the input side. This is called backpropagation. After backpropagation, we update all of the weights and biases of N layers with the gradients for each layer we calculated. Then we do forward propagation again, backpropagation again, update again and all again until we achieve our goal (which is usually minimizing the error as solving hopefully a convex optimization problem). You can also refer to:

https://www.youtube.com/watch?v=x_Eamf8MHwU for backpropagation

Also a final note: Gradients of RBM algorithm are not able to be calculated via classic backpropagation, a method called as contrastive divergence is used for the calculation. You can also have a look at the RBM paper on this matter and more, by Geoffrey Hinton:

https://www.cs.toronto.edu/~hinton/absps/guideTR.pdf

Hope I could help, good luck!


R: lattice equivalent of density2d in ggplot? What would be the equivalent of geom_density2d in lattice? In essence I'm trying to create this graph with lattice:

I don't think contourplot or levelplot is what i want and when trying it, it gives me a blank plot?

Use a custom panel function in which the density is estimated (e.g. using MASS::kde2d()):

library("lattice")
library("MASS")

set.seed(1);
dat <- data.frame(x=rnorm(100), y=rt(100, 5))

xyplot(y~x, data=dat,
       panel=function(x, y, ...) {
         dens <- kde2d(x=dat$x, y=dat$y, n=50)
         tmp <- data.frame(x=dens$x,
                       y=rep(dens$y, each=length(dens$x)),
                       z=as.vector(dens$z))
         panel.levelplot(tmp$x, tmp$y, tmp$z, contour=TRUE,
                         subscripts=1:nrow(tmp), region=FALSE, ...)
         panel.xyplot(x, y, ...)
       })


Discriminator of a Conditional GAN with continuous labels OK, let's say we have well-labeled images with non-discrete labels such as brightness or size or something and we want to generate images based on it.  If it were done with a discrete label it could be done like:

def forward(self, inputs, label):
    self.batch = inputs.size(0)
    h = self.res1(inputs)
    h = self.attn(h)

        ...

    h = self.res5(h)
    h = torch.sum((F.leaky_relu(h,0.2)).view(self.batch,-1,4*4), dim=2)
    outputs = self.fc(h)

    if label is not None:
        embed = self.embedding(label)
        outputs += torch.sum(embed*h,dim=1,keepdim=True)

The embedding can be made to match any shape to be added to a hidden layer and summing the embeddings to the latent, which forces the discriminator to recognize the class that's it's discriminating to make a better judgment on the discrimination.  That's cool and all, but this method uses embeddings which is the discrete method.  How about a continuous label?  I can't really find a method for doing this except for some semisupervised methods whereas I have exact labels.  Is anyone able to help me?

I think the answer is, you have to either create a very large number of categories, or do it completely differently than with the code I provided.  The paper as a good solution to the problem.


High multi class classification with small data set I am working on a multi-class classifier for data set with 240K samples and ~1880 classes, the most  populated class is 4% of the dataset and large number of classes are less then 1% of the samples.

I am trying to find a method to decide for which class I have enough information to correctly predict or to reject the prediction due to classifier performance for that particular class.

Also, are there any good methods to deal with that kind of situations of high multi class with very low sample size?

Yeah. There have been scenarios like that. Imagenet has more than 1000 classes. Same could be said for certain cifar datasets.

If deep learning, go with one-hot encoding for the labels.

Stick with something like multi-class AUC as metric instead of believing on Validation accuracy.

Go with a keras DNN, then you could try advanced architectures, with the last Dense net to have 1880 neurons each representing one label. Also stick with categorical_crossentropy for loss.

Hope this helps.