A bump in CV score curve! What does it mean? My learning curve is behaving strangely and I don't know if I'm doing anything wrong or it is because of the dataset nature.
I'm using a Neural network with (30,30,15,1) layers and 'ReLU' activation function for training and linear for the output layer.
My learning curve is showing weird behavior while learning. There is a bump like increase and decrease in the validation curve around 100 epochs which I don't have any idea about the source or how to resolve it(or even if it is needed to be resolved). Can anybody help me with this? (Any related comments are welcomed)

That pattern is common in neural networks training.

Train performance is an estimate of bias, and validation performance is an estimate of variance. Initially both go down. Bias continues to go down, but variance goes up. That is the classic bias-variance tradeoff. However, in neural networks variance will start to go down again. This is called the “double descent curve”.

It is not known why neural networks training show this “double descent curve.” One idea is that this pattern might be a concatenation of the two curves. One curve might be from optimization and the other curve might be from sampling. The paper "A Modern Take on the Bias-Variance Tradeoff" goes into more detail about it.


Cannot predict the prices using the date parameter #importing the libraries
import pandas as pd
import matplotlib as plt
import numpy as np

#getting the dataset
ds = pd.read_csv("data.csv")
ds['Date'] = pd.datetime.date(ds.Date,format ='%m-%d-%Y')
X = ds.iloc[:, 0].values
Y = ds.iloc[:, 1].values

#spliting the data into train and test
from sklearn.model_selection import train_test_split
X_train,X_test,y_train,y_test = train_test_split(X,Y,test_size=0.2,random_state=0)
X_train = X_train.reshape(-1,1)

#scaling the features of the data
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
y_train = sc.fit_transform(y_train)

from sklearn.linear_model import LinearRegression
regressor = LinearRegression()
regressor.fit(X_train,y_train)

#predicting the value
y_pred = regressor.predict(y_train)

#plootting the graph
plt.scatter(y_train,color ="red")
plt.plot(regressor.predict(y_train), color ="blue")
plt.title("prediction ")
plt.xlabel("date")
plt.ylabel("stock prices")
plt.show()

Your problem is with the date field?

In Machine Learning algorithms you have to feed the model with numerical data and date is not numerical. You can see in this posts what are the common techniques of encoding time: https://stats.stackexchange.com/questions/311494/best-practice-for-encoding-datetime-in-machine-learning

So you have either two options: encoding date time or drop it.

ds = ds.drop('Date', axis=1)


Batch-mode active learning: How to select the batch that will bring the largest model improvement I am facing a problem where I want to use active learning to improve my classifier. Basically, I can choose data from one (and only one) data set among a set of candidate data sets. The question is which one to choose?

In other words, given a set of candidate data sets that I can use improve my classification model, which one is going to improve the model most?
Can I use the some metric (eg., average) from the objective functions of the batches? Do I need to normalise the objectives across all data sets? Have metrics to infer the magnitude of model improvement been proposed?

At this stage, my objective function looks like:

a  Uncertainty + (1-a) Diversity

where a is a weight factor, Uncertainty is the uncertainty in the model prediction for a given data point and Diversity is a measure of distance between the data point and the training population.

Any help would be greatly appreciated.

Cheers

Franz

Your question is actually the whole point of active learning. You probably need to read about existing approaches in active learning in order to find the one which suits your needs.

I'm not up to date at all on the topic but a traditional approach was to train several models on available data, make them predict on all the unannotated instances then use majority voting: instances for which the models tend to agree are "easy" to predict, whereas those for which models make different predictions are "hard" so potentially more valuable to improve performance.


How to apply Stacking cross validation for time-series data? Normally stacking algorithm uses K-fold cross validation technique to predict oof validation  that used for level 2 prediction.

In case of time-series data (say stock movement prediction), K-fold cross validation can't be used and time-series validation (one suggested on sklearn lib) is suitable to evaluate the model performance. In this case no prediction shall be made on first fold and no training shall be made on last fold. How do we use stacking algorithm cross validation technique for time-series data?

TL;DR

Time-series algorithms assume that data points are ordered.
Traditional K-Fold cannot be used for time series because it doesn't take into account the order in which data points appear.
One approach to validate time series algorithms is with Time Based Splitting.

K-Fold vs Time Based Splitting

The two graphs below show the difference between K-Fold and Time Based Splitting. From them, the following characteristics can be observed.

K-Fold always the all data points.

Time Base Splitting uses a fraction of all data points.

K-Fold lets the test set be any data point.

Time Base Splitting only allows the test set to have higher indexes than the training set.

K-Fold will use the first data point for testing and the last data point for training.

Time Base Splitting will never use the first data point for testing and never use the last data point for training.

Scikit-learn implementation

Scikit-learn has an implementation of this algorithm called TimeSeriesSplit.

Look at their documentation, you find the following example:

from sklearn.model_selection import TimeSeriesSplit
X = np.array([[1, 2], [3, 4], [1, 2], [3, 4], [1, 2], [3, 4]])
y = np.array([1, 2, 3, 4, 5, 6])
tscv = TimeSeriesSplit(n_splits=5)

for train_index, test_index in tscv.split(X):
   print("TRAIN:", train_index, "TEST:", test_index)
   X_train, X_test = X[train_index], X[test_index]
   y_train, y_test = y[train_index], y[test_index]

>> TRAIN: [0] TEST: [1]
>> TRAIN: [0 1] TEST: [2]
>> TRAIN: [0 1 2] TEST: [3]
>> TRAIN: [0 1 2 3] TEST: [4]
>> TRAIN: [0 1 2 3 4] TEST: [5]


using sklearn class weight to increase number of positive guesses in extremely unbalanced data set? Hi I have a poorly correlated and unbalanced data set I have to work with. The set is 2 classes, 0 has 96,000 values and 1 has about 200. When I run random forest or other methods I get an output like:

    precision    recall  f1-score   support

          0       1.00      1.00      1.00     38300
          1       1.00      0.01      0.02        90

avg / total       1.00      1.00      1.00     38390

Precision is very high but it only classified one row as positive?

I tried using {class_weight = 'balanced'} in the random forest parameters and it provides:

   micro avg       1.00      1.00      1.00     38390
   macro avg       1.00      0.51      0.51     38390
weighted avg       1.00      1.00      1.00     38390

But still not many positive guesses? Should I look into oversampling?

You can try to compute class weights and assign these values to model via weight classes function. One more reminder about weights; probably major classes weight will be less than 1 so you need to round it to 1 otherwise model won't learn major class this time.


Where can I download the toy benchmark dataset for RNNs? I have read the paper:
Simple Way to Initialize Recurrent Networks of Rectified Linear Units

Where can I download the toy benchmark dataset for RNNs this paper mentions?

I need addition problem benchmark.

In https://arxiv.org/abs/1504.00941 the authors describe the toy dataset:

  We generated a training set of 100,000 examples and a test set of 10,000 examples as we varied T.

They give you values, e.g. $T = 400$. The start of $\S~4.1$ said that values in each example are uniformly drawn from the unit interval. Publishing the generated values on an FTP server would be silly. I suppose they might have published code, or the name of a PRNG algorithm together with a seed value, but supporting such code is not what they were focused on. They did supply hyperparameters, so you have enough information to generate similar examples.

The MNIST pixels, billion word corpus, and TIMIT datasets each come from their own sources, but clearly none of those is a "toy" problem as you asked about.


semantic relation or semantic relatedness between terms or phrases I am looking for a method that can realize if there is any semantic relation between two terms using ontologies. For example given two terms "kitchen" and "Chef" it can return s.th like chef works in a kitchen and a correlation value (for example 0.7). What about proper nouns or noun phrases? The relatedness of two phrases such as "the biggest defense against the US Navy's Laser Weapon System" and "China's crippling pollution problem".
I would be appreciated if you could give me some references.

Other-than Ontologies do check word2vec, this tool provides an efficient implementation of the continuous bag-of-words and skip-gram architectures for computing vector representations of words and can be used to find the closest words for a user-specified word


How to i get word embeddings for out of vocabulary words using a transformer model? When i tried to get word embeddings of a sentence using bio_clinical bert, for a sentence of 8 words i am getting 11 token ids(+start and end) because "embeddings" is an out of vocabulary word/token, that is being split into em,bed,ding,s.
I would like to know if there is any aggregation strategies available that make sense apart from doing a mean of these vectors.
from transformers import AutoTokenizer, AutoModel
# download and load model
tokenizer = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
model = AutoModel.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")

sentences = ['This framework generates embeddings for each input sentence']

#Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, max_length=128, return_tensors='pt')

#Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

print(encoded_input['input_ids'].shape)

Output :
torch.Size([1, 13])
for token in encoded_input['input_ids'][0]:
  print(tokenizer.decode([token]))

Output:
[CLS]
this
framework
generates
em
##bed
##ding
##s
for
each
input
sentence
[SEP]

I'm not sure there is a need for aggregation, or in other words you may have a pipeline mismatch. BERT sentencepiece tokenization is specifically meant to be passed to some set downstream pipelines, with the aim of the sentencepiece thing being to be able to cater to OOV words. By aggregating the sentencepiece tokens, you might be doing away with the benefit of being able to cater to OOV in your later pipeline.
If you are looking for whole word vector tokens, and want to work with OOV words, I would recommend looking at FastText instead. This algorithm more or less uses subwords, and it will also build tokens for OOV words by pretty much aggregating the subword information for that OOV word during a custom training step. The benefit here is that the aggregation step need not be part of your pipeline, and you can use these new vectors in any downstream task (except, of course, the pipelines that accept BERT sentencepiece tokens)


Simple question related to an ANN diagram showing all of the matrices' dimensions: inputs, weights and output I've been reading a few Neural Networks articles for the past week and one thing that I am still trying to grasp is the dimensioning of the matrices on an ANN training. I have created a diagram (based on the example given by the same article) to try to figure it out.  I have inserted the weights related to the neurons that are painted on the diagram.

I am hoping that someone can validate my reasoning, as it is one of my pain points right now.

I appreciate any help.

PS: Was this post supposed to be created on this forum or on Cross Validated? I can see that this topic appears on both.

Yes. reasoning presented here is correct. Mind that $x$ input vector is a row vector (not a column one): $[x_1, x_2, x_3]$. Multiplying $x\cdot w^1$ provides you an another row vector $y_1 = [y_{11}, y_{12}, y_{13}, y_{14}]$ which represents output of neurons 1 to 4 in 1st hidden layer.
Then you multiply $y_1\cdot w^2$ and gain another row vector and so on until the outlayer. At last performing an multiplying 1x4 $\cdot$ 4x1 provides you a scalar which will be a net output after using output's neuron's transfer function on that scalar.


Does converting continuous variable to discrete(categorical) variable increases accuracy of a tree based model? I've read other questions regarding if a continuous feature should be converted to categorical or not. But I'm interested in case of tree based classifiers such as Decision Tree, Random Forest, Gradient Boosted etc.

My intuition is that since tree based classifiers try to find the optimal split or the best test at each node, providing a categorical feature would make the splits more accurate than a providing a continuous feature.

My question is, doing the aforementioned pre-processing of data will lead to high accuracy in case of tree based models or the opposite? or it depends on the data?

I actually agreed with the now-deleted answer: you're unlikely to see improvement (at least with any consistency).

As you point out, binning a continuous variable just reduces the allowable split points for a tree model.  This reduces the capacity of the overall model.  That may be beneficial or detrimental, depending on how it affects the bias-variance tradeoff: I would expect it to help a lot on a fully-grown single decision tree, but probably hurt a little in a well-tuned ensemble of trees.

There are two nice examples in the sklearn docs illustrating their binning preprocessor:
https://scikit-learn.org/stable/auto_examples/preprocessing/plot_discretization.html
https://scikit-learn.org/stable/auto_examples/preprocessing/plot_discretization_classification.html

Extending the second one to (1) also include a full decision tree and (2) add binning-then-nonlinear-models produces the following:

At least in these simplistic examples, my suggestions above hold.


What are features in the context of reinforcement learning? In machine learning, "feature" is a synonym for explanatory variables. I know what a feature is. However, in the specific case of RL, it's not clear to me what features are. What are "features" in the specific case of RL? What are examples of features in RL? Which RL algorithms require the specification of features? Why do we need "features" in RL?

What are "features" in the specific case of RL?

In RL, the supervised learning component is used to learn a function approximation for either a value function or a policy. This is usually a function of the state, and sometimes of the state and action. The features are therefore explanatory variables relating to the state (or state and action) and need to be sufficient to explain the value or optimal action of that state.

In RL literature, there is often no difference between terms "state representation" or "features". Although you may need to process the state representation using feature engineering, into suitable features for whatever supervised learning you are using. With neural networks, this may involve normalising values that might otherwise cause problems.

  What are examples of features in RL?

In OpenAI's gym, CartPole-v1 has the following features describing state:

Cart Position
Cart Velocity
Pole Angle
Pole Velocity At Tip

The positions and speeds are measured in arbitrary units in this case, but you might consider them to be in SI units - metres along the track, metres per second, radians, metres per second.

Typically in RL, you will want features that:

Describe the state (and possibly the action, depending on implementation)
Are feasible to collect by observation of the environment
Together with the action choice, can adequately explain expected rewards and next state. In RL terms, they should in aggregate possess the Markov property

  Which RL algorithms require the specification of features?

None of them*. However, without descriptive features of state, you are limited to enumerating all possible states, and using tabular methods. Working with tabular methods constrains you in practice by the time it takes to fully explore all the states, and the memory space required to represent all the individual estimates of value.

If you are looking into solving your problem using DQN, A3C or other "deep" RL method that uses neural networks, then you need to be thinking in terms of a state representation that is composed of features, and to treat those features as if they were inputs to the same neural networks used in supervised learning.

  Why do we need "features" in RL?

To work with complex environments with large state spaces that cannot be solved without some form of approximation and (hopefully) generalisation.

* This is slightly more complex with policy gradient methods, as they do require that you use a function approximator, such as a neural network. However, it is possible to make the "features" in that case a one-hot coded vector of the state enumeration, which is the same representation as tabular value-based methods. There is little reason to do that in practice, but it is possible in principle.


Column sum in SPSS (with filter and grouped by date)? device      date     act power 1     react power 2
-------------------------------------------------
M1         05-02            2              3
M2         05-02            4              2
M3         05-02            3              4
M1         06-02            1              2
M2         07-02            3              4
                     -------        -------
                    need sum       need sum

Say that I only need the sum of M1 and M2 from that table. How could I add a variable that contains the sum of power group by date and device?

I don't know if it is desired to have something like this? Or how can I use descriptive statistics to filter away the other devices just to have M1 and M2 only.

device      date     SUM (actpower1)     SUM(reactpower2)
-------------------------------------------------
M1+M2      05-02            6              5

```

Normally such operations are done in Excel (click into an empty cell and type =Sum( select the column of which you want to calculate the cumulative sum and then close the bracket ). Finally, hit enter). Note that the sum is only a global metric. It will not be a local measure for each observation that you have.

If you want to get these sums in SPSS you can use the descriptive statistics or you can also use the following syntax CREATE CUMMULATIVESUM=CSUM(OLDVAR). This will generate the cumulative sum for your observations. The entry for your last observations will be the sum of all observations.


Difference between LDA and Naive Bayes LDA: linear discriminant analysis

Suppose we have a classification problem. I understand that the data can be such that the features may have discrete values or continuous values.

Suppose our data contains continuous feature values. Then we can apply Naive Bayes using a distribution. Lets assume the data to be normally distributed and so use Naive Bayes with normal distribution. We can also apply LDA which also uses Normal distribution.

Using Naive Bayes we assume the features to be independent and by using LDA we assume the covariance to be same for all the classes.

How does these assumptions make these 2 models perform differently and which is a better model and in which conditions?

As far as I know, Gaussian LDA and Gaussian Naive Bayes both assume the features follow normal distributions. However, GNB places a stronger assumption on feature independence, while Gaussian LDA does not require features to be independent.
See this.


How to build a textual search engine? I am having an HTML string and want to find out if a word I supply is relevant in that string.

Relevancy could be measured based on frequency in the text.

An example to illustrate my problem:

this is an awesome bike store
bikes can be purchased online.
the bikes we own rock.
check out our bike store now

Now I want to test a few other words:

bike repairs
dog poo

bike repairs should be marked as relevant whereas dog poo should not be marked as relevant.

Questions:

How could this be done?
How to I filter out ambiguous words like in or or

Thanks for your ideas!

I guess it's something Google does to figure out what keywords are relevant to a website. I am basically trying to reproduce their on-page rankings.

pre-process your documents (some of the steps may be skipped)

tokenize
remove stop words
stem or lemmatize
do normalization (e.g. U.S.A. -> USA, météo -> meteo, etc) and orthographic correction
perform phonetic normalization (e.g. with Soundex)
find equivalence classes (using thesauri, e.g. WordNet)

use a Vector Space model to represent documents (you may use TF, aforementioned TF-IDF or other models)
do the same with the query: preprocess and represent it in the vector space
find the most similar documents by computing the vector similarity (e.g. using the cosine similarity)

That's an outline of the Information Retrieval process

Introduction to Information Retrieval by Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze is a very good book to get started in IR.

Or just use Apache Solr to get everything you need out of the box (or Apache Lucene, that is used by Solr, to build your own application)


Best way to search for a similar document given the ngram I have a database of about 200 documents who's ngrams I have extracted. I want to find the document in my database that is most similar to a query document. In otherwords, I want to find the document in the database that shares the most number of ngrams with the query document. Right now, I can go through each one and compare it one by one, but this will take O(N) time and is expensive if N is very large. I was wondering if there are any efficient data structures or methods in doing efficient similarity search. Thanks

You could use a hashing vectorizer on your documents. The result will be a list of vectors. Then vectorize your ngrams in the same way and calculate the projection of this new vector on the old ones. This is equivalent to the database join on an index, but may have less overhead.


Tensorflow regression model giving same prediction every time import tensorflow as tf

x = tf.placeholder(tf.float32, [None,4])    # input vector
w1 = tf.Variable(tf.random_normal([4,2]))   # weights between first and second layers
b1 = tf.Variable(tf.zeros([2]))             # biases added to hidden layer
w2 = tf.Variable(tf.random_normal([2,1]))   # weights between second and third layer
b2 = tf.Variable(tf.zeros([1]))             # biases added to third (output) layer

def feedForward(x,w,b):                     # function for forward propagation
    Input = tf.add(tf.matmul(x,w), b)
    Output = tf.sigmoid(Input)
    return Output

Out1 = feedForward(x,w1,b1)                # output of first layer
Out2 = feedForward(Out1,w2,b2)             # output of second layer
MHat = 50*Out2                             # final prediction is in the range (0,50)

M = tf.placeholder(tf.float32, [None,1])   # placeholder for actual (target value of marks)
J = tf.reduce_mean(tf.square(MHat - M))    # cost function -- mean square errors
train_step = tf.train.GradientDescentOptimizer(0.05).minimize(J)     # minimize J using Gradient Descent
sess = tf.InteractiveSession()             # create interactive session

tf.global_variables_initializer().run()    # initialize all weight and bias variables with specified values
xs = [[1,3,9,7],
      [7,9,8,2],                           # x training data
      [2,4,6,5]]

Ms = [[47],
      [43],                                # M training data
      [39]]

for _ in range(1000):                      # performing learning process on training data 1000 times
    sess.run(train_step, feed_dict = {x:xs, M:Ms})

>>> print(sess.run(MHat, feed_dict = {x:[[1,15,9,7]]}))
[[50.]]

>>> print(sess.run(MHat, feed_dict = {x:[[3,8,1,2]]}))
[[50.]]

>>> print(sess.run(MHat, feed_dict = {x:[[6,7,10,9]]}))
[[50.]]

In this code, I am trying to predict the marks M obtained by a student in a test out of 50 given how many hours he/she slept, studied, used electronics and played the day before the test. These 4 features come under the input feature vector x.

To solve this regression problem, I am using a deep neural network with an input layer with 4 perceptrons (the input features), a hidden layer with two perceptrons and an output layer with one perceptron.

I have used sigmoid as the activation function. But, I am getting the exact same prediction([[50.0]]) for M for all possible input vectors I feed in.

Can someone please tell me what is wrong with the code above, and why I get the same result each time?

Your network design/logic is basically correct, but you are seeing some very common problems with neural network numerical stability. This results in your weights diverging and not training accurately.

Here are the fixes, any one of them might help a little, but the first two should be used for nearly all neural network projects.

1. Inputs need to be scaled to work with neural networks.

This is called input normalisation. Usually you would do this in data preprocessing, but for your simple network we can include the scaling at the input:

x_normalised = x * 0.2 - 0.5             # Arbitrary scaling I just made up
Out1 = feedForward(x_normalised,w1,b1)   # output of first layer

The most common scaling operation is to take all the training data and convert it so that it has mean $0$ and standard deviation $1$ (typically per feature, as opposed to global scaling for all data as I have done here) - store the values used to achieve that and apply them to all following data for training, tests etc.

2. Adjust learning rate until you get a working value.

train_step = tf.train.GradientDescentOptimizer(0.001).minimize(J)

A value that is too high will cause your training to fail. A value that is to low will take ages to learn anything.

3. For small training sets, use more iterations than you might think

This is not a general thing, but specific to demos with tiny amounts of training data like your example, or the commonly-used "learning XOR function".

for _ in range(10000):                      # performing learning process

   sess.run(train_step, feed_dict = {x:xs, M:Ms})

With your very simple network actually this may cause over-fitting to the training data, so you will have to play with a value that gives you "sensible" results. However, in general how to spot and minimise over-fitting is a whole broad subject in itself, based on how you test and measure generalisation. This will need other questions if you are not sure when you learn it. It should be high on your list of things to learn though . . . it is a critical skill in producing useful neural networks that solve real problems.


Feature selection where adding features are deteriorating model k

I am training a kNN classifier with 144 features and graphed the accuracy vs number of features used and got this. What might be the reason for the drops in the accuracy at some points of the graph? I am using accelerometer-gyroscope-magnetometer fusion to recognize human activities.

The one presented is validation accuracy. Should I use training accuracy instead? And why?

I ranked the features using ReliefF feature selection algorithm.

I used time domain features such as mean, standard deviation, rms, median, variance, iqr, mad, zcr and mcr, and frequency domain features such as skewness, kurtosis and pca

Here are the top 8 features chosen. Peak accuracy occurs at 8 features.

I guess the measurements from accelerometer-gyroscope-magnetometer are noisy and redundant in some sense. This means you can find some sort of correlation among the values of the measurements, for instance a correlation between the values from the accelerometer and the gyroscope.

PCA captures the principal directions of variation on your data, removing the correlation between the measurements and also reducing the noise, therefore increasing the accuracy. From the graph it can be seen that the accuracy just diminishes a little when using all the features.

Other factor I will consider is the magnitude of the features, a feature with a very large magnitude affects the behavior of K-NN.


How to go about cooling load forecast for a district level dataset? I am trying to build a high accuracy cooling load forecast for a district-level dataset over various time horizons. My data consists of a time series of cooling load and consecutive weather data. I added a bunch of other features such as an hour of the day, day of the week, holiday or not, etc. My question is, what are some current models for the prediction of the time series load dataset like mine? I gather LSTM is the way to go. However, given the features I have identified, are there other models that are more accurate for such a forecast? If I chose to go the NN fitting route, what are some other features I can add?

For a good overview of common statistical methods, check our Rob Hyndman's Forecasting Principles and Practices.
For a comparison of different models, including machine learning and deep learning, check out the M4 competition and their companion repo. The current state-of-the-art on the M4 data set is N-BEATS as far as I know.
For a replication and extension of the M4 study in Python, check out sktime (disclaimer: I'm one of the developers) and our preprint.


why to use Scaler.fit only on x_train and not on x_test for normalizing value using MinMaxScaler? while normalising the data everone is saying that we need to fit only on x_train and not on x_test ? why is that we should not fit x_test ?

if we should not fit the scaler on x_test then why we need to apply the transformation alone on x_test ?

from sklearn.preprocessing import MinMaxScaler
Scaler=MinMaxScaler()
Scaler.fit(X_train)
x_train=Scaler.transform(x_train)
x_test=Scaler.transform(x_test)

If you would fit the scaler using x_test you would be using information from your test set and would be leaking data. This is information that you would not have if your model was in production and can therefore not use when fitting your model.


Fashion Clothing images Dataset with image description I am searching for Fashion Clothing image dataset where each image is associated with description. Basically description should contains all information about cloth. like -

image_01.jpg - red shirt has collar with long-sleeves.

Is there any good source to collect these type of dataset ??

You can check Fashion10000 dataset released by Delft University of Technology here


An ambiguity in SVM equations about misclassified data I have encountered an ambiguity in SVM equations.
As is stated in Chris Bishop's machine learning book, the optimization goal in SVM is to maximize this function:

$$C\sum\limits_{n = 1}^N {{\xi _n}}  + {1 \over 2}{\left\| w \right\|^2}$$

Subject to this constraints(*):

$${\xi _n} \ge 0$$
$${t_n}y({x_n}) \ge 1 - {\xi _n}$$

where:

$$y({x_n}) = {w^T}{x_n} + b$$

so the corresponding Lagrangian function for this problem is:

$$L(w,b,a) = C\sum\limits_{n = 1}^N {{\xi _n}}  + {1 \over 2}{\left\| w \right\|^2} - \sum\limits_{n = 1}^N {{a_n}\{ {t_n}y({x_n}) - 1 + {\xi _n}\}  - } \sum\limits_{n = 1}^N {{\mu _n}{\xi _n}} $$

and the corresponding KKT conditions are given by (**):

$${a _n} \ge 0$$

$${{t_n}y({x_n}) - 1 + {\xi _n}} \ge 0$$

$${a_n}({t_n}y({x_n}) - 1 + {\xi _n}) = 0$$

$${\xi _n} \ge 0$$

$${\mu _n} \ge 0$$

$${\mu _n}{\xi _n} = 0$$

And if we set

$${{\partial L} \over {\partial {\xi _n}}} = 0$$

we get (***)

$${a_n} = C - {\mu _n}$$

As we know, that subset of data points that have

$${a_n} = 0$$

are not support vectors. But for this data points we have (from ***):

$${\mu_n} = C$$

and therefore (from **)

$${\xi _n} = 0$$

So here lies the problem. If a data point from this subset is in the wrong side of the decision boundary, then

$${t_n}y({x_n}) \le 0$$

and we will have (from *)
$${\xi _n} \ge 1$$

which is in an obvious conflict with

$${\xi _n} = 0$$

Good point! Interesting consequence!

Problem is the $a_n=0$ assumption, i.e. assuming misclassified points are not support vectors.

Here is the flow. Slack variable $\xi_n$ is defined as
$$\xi_n := |t_n - y(\boldsymbol{x}_n)|$$
where $t_n \in \{+1, -1\}$ is the true label, and $y(\boldsymbol{x}_n)$ is the prediction. Therefore, for a misclassified point (on the wrong side) we have $$\xi_n > 1$$ by definition. Given $\mu_n \xi_n = 0$, therefore$$\mu_n=0$$
and given $a_n=C - \mu_n$, therefore $$a_n = C > 0$$ which means (given $a_n > 0$ only for support vectors)

  Every misclassified  point is a support vector.

This is a nice consequence and should have been stated in the book.

Although, a remotely! related point has been stated in the book:

  Points with $a_n = C$ can lie inside the margin and can either be
  correctly classified if $\xi_n \leq 1$ or misclassified if $\xi_n > 1$.


How to plot number of Trees and OOBs score with Grid Search I searched to find the answer but I don´t find something with Grid Search.
I create a random forest and gradient boosting regressor with grid search. Now I want to make a visualization to see if the number of trees is alright.
# Grid-Search erstellen auf der Basis eineer Random Search
param_grid = {
    'bootstrap': [True],
    'max_depth': [50, None],
    'max_features': ['auto', 50],
    'min_samples_leaf': [5],
    'min_samples_split': [2],
    'n_estimators': [100, 150, 200, 250, 300, 350]
}

# Base-Model zum Verbessern erstellen
rf_v1 = RandomForestRegressor(random_state=42)

# Grid-Search initisieren
rf_grid_search_v1 = GridSearchCV(estimator = rf_v1, param_grid = param_grid,
                           scoring = 'neg_mean_absolute_error', cv = 3,
                           n_jobs = -1, verbose = 2)

# Die Grid-Search für das Model durchführen
rf_grid_search_v1.fit(X_train_v1, y_train_v1)

But how to plot the splits of trees with gridsearch? Is there any way to make an RF/xgb trees split visualization?
I am really grateful if anybody has a tip for me.

To plot feature importance using gridsearch use:

x= X_train_v1.columns,y= rf_grid_search_v1.best_estimator_.feature_importances_


How to gather training data for simple voice commands? I'm trying to build a machine learning model for recognizing simple voice commands like up, down, left, etc.

On similar problems based on images, I'd just take the picture and assign a label to it.

I can generate features and visualize them using librosa. And I hear CNNs are amazing at this task. So, I was wondering how I'd gather training data for such audio based systems, since I can't record an entire clip considering my commands are only going to be a few milliseconds long (and while predicting, it'll only be a few milliseconds too).

My first idea was to generate an image (using matplotlib and librosa.features) for every few milliseconds and just use the ones where I'm saying the command as positive labels (and the rest negative labels), but it seems tedious and there might be a better way to do it.

Is there an effective way to do achieve this ?

What would librosa generate? If the sound clips it gives you aren't also words then I would worry about the effectiveness of the training. It'd be better to record samples of yourself saying the positive labelled words and saying a bunch of other words to use as negative labels. I recognize this is likely to be an irritating process; one approach I have read of in the past to maximise the use of small amounts of training data is to use transformations and distortions on training data to increase the effective number of samples. In this case, the training data were pictures of animals. Effectively what the researchers did was tweak the images slightly; they rotated the images various degrees, squeezed them (by lowering the height or width and not retaining aspect ratio) or stretched them and so on. You might be able to do the same;  record a smallish number of samples and then use distortions to raise the effective number of samples to a useful number.


Collinearity and Multicollinearity in the features? What are some advanced or basic methods most used by data scientists/ML Engineers to detect collinearity (or) multicollinearity between features?

One way to measure multicollinearity is the Variance Inflation Factor (VIF), which assesses how much the variance of an estimated regression coefficient increases if your predictors are correlated.

If no factors are correlated, the VIFs will all be 1.

If the VIF is greater than 1, the predictors may be moderately correlated.

A VIF between 5 and 10 indicates high correlation that may be problematic.

And if the VIF goes above 10, you can assume that the regression coefficients are poorly estimated due to multicollinearity

If multicollinearity is a problem in your model the solution may be relatively simple. Try one of these:

Remove highly correlated predictors from the model.  If you have two or more factors with a high VIF, remove one from the model. Because they supply redundant information, removing one of the correlated factors usually doesn't drastically reduce the R-squared.  Consider using stepwise regression, best subsets regression, or specialized knowledge of the data set to remove these variables. Select the model that has the highest R-squared value.

Use Partial Least Squares Regression (PLS) or Principal Components Analysis, regression methods that cut the number of predictors to a smaller set of uncorrelated components.


Hadoop Resource Manager Won't Start I am a relatively new user to Hadoop (using version 2.4.1). I installed hadoop on my first node without a hitch, but I can't seem to get the Resource Manager to start on my second node.

I cleared up some "shared library" problems by adding this to yarn-env.sh and hadoop-env.sh:

  export HADOOP_HOME="/usr/local/hadoop"

  export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

I also added this to hadoop-env.sh:

  export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native

based on the advice of this post at horton works http://hortonworks.com/community/forums/topic/hdfs-tmp-dir-issue/

That cleared up all of my error messages; when I run /sbin/start-yarn.sh I get this:

  starting yarn daemons

  starting resourcemanager,
  logging to /usr/local/hadoop/logs/yarn-hduser-resourcemanager-HdNode.out

  localhost: starting nodemanager,
  logging to /usr/local/hadoop/logs/yarn-hduser-nodemanager-HdNode.out

The only problem is, JPS says that the Resource Manager isn't running.

What's going on here?

Check your version of JPS and make sure it's the same as the version of java that you are running.  Sometimes you start out with an out of the box jdk install, upgrade, set alternatives for the java bin, but still have the original jps binary being referenced.

Run ps -ef |grep java and look for the resource manager threads.  Maybe it's actually running.  If it is, try update-alternatives --config jps to see what binary jps is pointing at and compare it with the java binary that you are using.

If the resource manager is not running, it's time to do some basic linux troubleshooting.  Check log files and barring that check actual command output.

On the system I'm looking at now, the log files for resource manager are placed in the hadoop-install/logs directory in yarn-username-resourcemanager-hostname.log and yarn-user-resourcemanager-hostname.out.  Your configuration may place them in /var/logs or what have you.  Also, have a look at the syslog.

If the logs don't yield any good information, which can happen, my process is to generally try to figure out the command line from the startup script (usually by prefixing the command line with echo), and then trying to run the command directly to see the output as it comes out.

I have actually run into this problem before, but I can't remember the specific issue.  I'm sure the same result can manifest itself from a variety of problems.  Considering that you are as far as you are in the process of getting set up, I believe it's likely to be a minor configuration issue.


How to create domain rules from raw unstructured text using NLP and deep learning? How to create domain rules from raw unstructured text using NLP and deep learning techniques ?
For example for the below text on symptoms of Dengue, all three look pretty similar but if you want to make sure a person is having Dengue you want to exact definite common minimum rules out of these raw text , so as to confirm a person is having Dengue. Can someone give reference to some research or blogs where similar problem has been solved ?

1) Symptoms of dengue fever include severe joint and muscle pain, swollen lymph nodes, headache, fever, exhaustion, and rash.

2) High fever and at least two of the following:
Severe headache
Severe eye pain (behind eyes)
Joint pain
Muscle and/or bone pain
Rash
Mild bleeding manifestation (e.g., nose or gum bleed, petechiae, or easy bruising)
Low white cell count

3) Aching muscles and joints
Body rash that can disappear and then reappear
high fever
intense headache
pain behind the eyes
vomiting and feeling nauseous

For the above three extracts commons rule would look like

1) Fever
2) Joint and muscle pain
3) Headache
4) Rash

I think your question can be solved using Case Based Reasoning .

Basic principle how it works is, you need to train the model using whole lot of different cases which you have. Based on the symptoms which you give the outcome is predicted(which disease).

Please refer to the links below for deep diving into the topic, appended couple of links with respect to health care industry:

Link -1
Link -2
Link -3
Link -4
Link -5
Link -6
Link -7

Do let me know if you need any additional information.


How can I use time stamps for classification? My data consists of entries when an event is True, namely, when a train crossing is down. So I will have entries within a day like so (just examples):

start,end,interval-length

10:43:02,10:46:02,180,
10:49:06,10:51:06,120

The state is of course True for these entries, and the times not logged throughout a day are when the state is False, aka traversable. (however I have a strong feeling I should explicitly pad these times in, so that I have full 24-hour cycles logged)

The data presented is a time series, based on which I wish to make predictions for coming days. I want to be able to ask "What are the chances that the crossing is down at time $t$, of day $d$?" (the second argument is important because I will have 7 different models, one for each day of the week due to the differing train traffic schedules in a week), taking into consideration the previously observed times.

This of course needs to take into account the fact that when the events happen, they have a certain length (as shown by my data). This is not inherently a classification problem because I am also interested in a confidence of the prediction, which is not as simple as taking a built model's performance parameters. Assuming though that this could be a promising way to tackle the problem, how can I build a model so that the times later on in a day don't wrongfully bias it?

Consider the simple example of some dataset, where days are represented as indexing numbers 1-7. An unsupervised classification algorithm would be biased to guess more towards saturday and sunday, because they are higher numbers (and in this case the solution is to have 7 different columns, each with boolean values instead - but I can't do anything similar to this in my case). I hope this example clears up my question a bit.

Not knowing your data in detail, my intuition is that you could go for dummy (one hot) encoding. You could split each day in (say) 10 min. intervals $x$ (144 columns) and attach labels (up/down) $y$. Each time interval $x$ would be one dummy encoded column (true/false).

The model would be a binary classification (logistic) like:

$$ y = \beta_0 + \beta_1 x_1 + ... + \beta_n x_n + u .$$

In case you have enough days, this could be a reasonable predictive model. It is likely, that some of the dummy encoded $x$ are irrelevant. So it could be worth to check a Lasso model, where some (not so helpful $x$) are shrunken or automatically excluded from the model.

https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html

Based on the same logic, you could also check boosting, e.g. lightGBM to make the classification.


How Does Sci-Kit Learn Train Regression Neural Networks (MLPRegressor) So Fast? Why does using the scikit-learn library's MLPRegressor result in such a boost in training time when compared to constructing the network from scratch? I tried both methods and I found that writing the code from scratch yielded an average training time of ~10 seconds while Sci-Kit Learn trained almost instantly?

Scikit-learn use numpy and cython to speed-up code.


How can be proved that the softmax output forms a probability distribution and the sigmoid output does not? I was reading Nielsen's book and in this part of chapter 3 about the softmax function, he says, just before the following Excercise, that the output of a neural network with a output softmax layers forms a probability distribution and the sigmoid output does not always forms it.
Now I've been wondering about the output of a neural network, if I have a sigmoid output layer, say for one observation the output is 0.7 for class 0, should the probability for class 1 be 0.3? Or, in this binary classification example, using a softmax output, the first output neuron would be 0.7 for class 0 and 0.3 for class 1 in that particular observation?

Softmax maps $ f:ℝ^n\rightarrow (0,1)^n$ such that $\sum f(\vec x) =1$. Therefore, we can interpret the output of softmax as probabilities.
With sigmoidal activation, there are no such constraints for summation, so even though $ 0<S(\vec x)<1$, it is not guaranteed that $\sum S(\vec x)=1$. The sigmoidal function does not normalize the outputs, so in your example where class 0 has output $0.7$, class 1 could have any value in $(0,1)$, which might not be $0.3$.

Here's an example:
$\vec x=[-5,\pi,\frac{1}{3},0] $
$ f(\vec x)\approxeq [2.6379\times10^{-4},0.9059,0.05464]$
$ S(\vec x)\approxeq [6.693\times10^{-3},0.9586,0.5826,0.5] $
Because $0<f(\vec x)<1$ and $\sum f(\vec x)=1$, the softmax output vector can be interpreted as probabilities. On the other hand, $ \sum S(\vec x) > 1$, so you cannot interpret the sigmoidal output as a probability distribution, even though $ 0<S(\vec x)<1$
(I chose the above $\vec x$ arbitrarily to demonstrate that the inputs need not be negative, non-negative, rational, etc., hence $\vec x\in ℝ^n$)


Can continuous dataset have negative values? As in the description, I want to choose continuos dataset with temperatures and the minimun value is -40 and maximum is 51. Is that OK? Can we have negative values in continuous distribution?

Yes, you can. Why would a negative temperature make no sense? Unless the temperature is measured in Kelvin...

You should know what each feature describes, and based on that, make sure the data makes sense. As of the dataset itself, with no context, it is indeed possible for any numerical feature to take negative numbers.

Hope this helps!


test accuracy of text classification is too less I have a data set of movies and their subtitles. My task is to classify them based on their ratings - [R, NR, PG, PG-13, G].

I have 13 examples for each class.
I preprocessed the subtitles in the following way:

I used word puns tokenizer to tokenize subtitles.
Removed stop words and punctuation.
Performed stemming.
Vectorized the subtitles using TF-IDF vectorizer.

The accuracy that I am getting using:

1) svm :
train accuracy is 1.0 and test accuracy is .17.

2) naive bayes:
training accuracy is 0.5 and test accuracy is .23.

I have the following questions:

1)Why is my accuracy so low and what can I do to improve the accuracy?

2)Will more training data help?

3)Should I perform feature selection?

4)What other classification algorithms can I use to improve the accuracy?

You should try to perform cross-validation using all your datasets. Maybe your test set has a completely different distribution than your training set (with such a small number of samples out of all possible subtitles, it is likely). Did you create this dataset?

What is the size of the final input vectors you are using and what size of vocabulary are you using for this?

At least in the case of the SVM, it seems that your model is overfitting and you need to increase the regularization.
More training data will always help.
Not sure what are your options here.
There are a lot of options based on the small number of samples, you can try using clustering (KNN, K-Mean).


What should be the value of batch_size in fit() method when using sgd (Stochastic Gradient Descent) as the optimizer? I am confused about the batch size of this model. I have used sgd i.e., Stochastic Gradient Descent as the optimizer (see the code). I am aware that in sgd, a single random instance from the training set is used to compute the gradient at each step. So, according to it, the batch_size should be equal to 1. Now, in the tf.keras.Sequential.fit() documentation it says:

  If unspecified, batch_size will default to 32.

So, do I have to manually set the batch_size equal to 1? It is because the default value, 32 will make it a Mini-batch Gradient Descent.

    import tensorflow as tf
    from tensorflow import keras

    fashion_mnist = keras.datasets.fashion_mnist
    (X_train_full, y_train_full), (X_test, y_test) = fashion_mnist.load_data()

    X_valid, X_train = X_train_full[:5000]/255.0, X_train_full[5000:]/255.0
    y_valid, y_train = y_train_full[:5000], y_train_full[5000:]

    model = keras.models.Sequential()

    model.add(keras.layers.InputLayer(input_shape = [28, 28]))
    model.add(keras.layers.Flatten())
    model.add(keras.layers.Dense(300, activation = "relu"))
    model.add(keras.layers.Dense(100, activation = "relu"))
    model.add(keras.layers.Dense(10, activation = "softmax"))

    model.compile(loss = "sparse_categorical_crossentropy", optimizer = "sgd", metrics = ["accuracy"])

    history = model.fit(X_train, y_train, epochs = 30, validation_data = (X_valid, y_valid))

First, using the appropriate terminology you can say batch Stochastic Gradient Descent and batch Gradient descent are in the extreme ends, where Stochastic Gradient Descent is training with $batch size = 1$ and for batch gradient descent with $batch size=n$ where $n$ denotes the number of data points.

In the appropriate terminology, what we are often using (and similar to your example as well) is called mini-batch gradient descent. Note that the term mini here does not mean it is necessarily very small like 4, 32 or 64 but instead can be anything bigger than $1$ but smaller than $n$. In practice, people use the term mini-batch gradient descent and stochastic gradient descent interchangeably. This is because in practice they behave similarly.

I personally do not think that such practice (using SGD and minibatch SGD interchangeably) is bad, because I don't think that it differs a lot such that it requires a specific new term.


How to train models using Google ML ( keras & tenserflow) Do I need to change my existing code I used to train neural network from my PC.
here is my current code.

Model = Sequential()
Model.add(Convolution2D(32,3,3, input_shape=(128,128,3),activation="relu"))

Model.add(MaxPooling2D(pool_size=(2,2)))

Model.add(Convolution2D(32,3,3,activation="relu"))
Model.add(MaxPooling2D(pool_size=(2,2)))

Model.add(Flatten())

Model.add(Dense(output_dim =128, activation= 'relu'))
Model.add(Dense(output_dim =1, activation= 'sigmoid'))
Model.compile(optimizer="adam", loss = 'binary_crossentropy', metrics= ["accuracy"])

train_datagen = ImageDataGenerator(
        rescale=1./255,
        shear_range=0.2,
        zoom_range=0.2,
        horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1./255)

traning_set = train_datagen.flow_from_directory(
        'data/train_set',
        target_size=(64, 64),
        batch_size=32,
        class_mode='binary')

test_set = test_datagen.flow_from_directory(
        'data/test_set',
        target_size=(64, 64),
        batch_size=32,
        class_mode='binary')

classifier.fit_generator(
        traning_set,
        steps_per_epoch=204,
        epochs=20,
        validation_data=test_set,
        validation_steps=101)

classifier.save('mymodel.h5')

You should provide more information about the previous framework you used and your layer setup. I imagine you also want to "translate" your trained model into a proper weight format


Using pandas get_dummies() on real world unseen data I made a ML model, trained and tested it with my data containing categorical variables.
To create dummy variables I used pd.get_dummies() before the split.
I now want to use my model on previously unseen data where, of course, I need to re create my dummies. Should I do it still with pd.get_dummies()? In this way isn't the encoding lost? Any suggestion on how to do it?
Thanks

Yes, the encoding would be lost. You should instead use sklearn OneHotEncoder and save the corresponding encoder instance so that you can re-load it on unseen data.

One can do something along these lines:

import pandas as pd
import pickle
from sklearn.preprocessing import OneHotEncoder

def get_encoder_inst(feature_col):
    """
    returns: an instance of sklearn OneHotEncoder fit against a (training) column feature;
    such instance is saved and can then be loaded to transform unseen data
    """
    assert isinstance(feature_col, pd.Series)
    feature_vec = feature_col.sort_values().values.reshape(-1, 1)
    enc = OneHotEncoder(handle_unknown='ignore')
    enc.fit(feature_vec)
    with open(file_name, 'wb') as output_file:
            pickle.dump(enc, output_file)
    return enc

which can then be loaded and applied as

def get_one_hot_enc(feature_col, enc):
    """
    maps an unseen column feature using one-hot-encoding previously fit against training data
    returns: a pd.DataFrame of newly one-hot-encoded feature
    """
    assert isinstance(feature_col, pd.Series)
    assert isinstance(enc, OneHotEncoder)
    unseen_vec = feature_col.values.reshape(-1, 1)
    encoded_vec = enc.transform(unseen_vec).toarray()
    encoded_df = pd.DataFrame(encoded_vec)
    return encoded_df

where the argument enc in the latter function is an instance of OneHotEncoder that you load through pickle.load. Of course the above is just a pseudo-code example, take care that all the objects that you are using keep the initial shapes and so on.

The problem with using pd.get_dummies is that it has no memory of the previously mapped encoding: it basically turns a column into factors, whereas OneHotEncoder actually maps categorical variables to a fixed-length vector representation that is stored and kept.


Image features (spectral bands) other than RGB for image analysis What features can I derive from the image, other than its RGB, to help me analyze an image?

Actually, I'm looking for some papers, etc about this. But I couldn't find.

I know I can use some features like its lightness, etc. But I don't want a single value as lightness, I want a 2d array just like R or G or B.

I think what you are looking for is called a colour space. There are multiple available, and each of them was created to solve a certain problem.

The RGB colour space is the most common I guess, since that is close to the way the computer displays show information (although they do apply some transformations before lighting the pixels). It is also similar to the way we see light (we have red, green and blue receptors).

Light is additive. If you keep adding colours, you get to white. Paint is subtractive. That leads to some differences into how colours mix and lead to the creation of the CMYK standard for printing. It represents colours in terms of cyan, magenta, yellow and black.

Perhaps more important for you is the HSB color space. It is rather nicely shaped, as a 3D cylinder. H stands for hue, the colour nuance, S stands for saturation, how pronounced is the colour and B stands for brightness. HSL is a similar color space, where the vertical axis is lightness instead of brightness. I find these two color spaces quite useful in analysis.

The RG cromacity space is used to analyse the color itself, and discard the brightness information. You basically represent the amounts of red and green from a normalized color (a color where the sum of r g b is 1). This way, you operate in a simpler, 2D space, and analyze the color information. See [1], [2].

The CIE colorspace is based on measurements of our color perception. It tries to mimic the way our photoreceptors react to color. The original CIE colorspace, known as CIEXYZ, was published in 1931. It has been revised in 1976 to form the CIELab colorspace, which is perceptually uniform. This means that it accounts for our non-liniar perception of color, making sure that the same increase produces the same perceptual difference across the color space. See [3], [4], [5].

The TSL colorspace (tint, saturation, lightness) was proposed in [6] for face detection.

There are many other colorspaces, but these are the ones I found most useful. There are usually conversions across different color spaces.

A publication by Logvinenko, 2014 might help with understanding the geometry of a color space and choosing a colorspace that helps your analysis [7].
Also, have a look at [8].


Could not convert string to float error on KDDCup99 dataset I am trying to perform a comparison between 5 algorithms against the KDD Cup 99 dataset and the NSL-KDD datasets using Python and I am having an issue when trying to build and evaluate the models against the KDDCup99 dataset and the NSL-KDD dataset.

Whenever I try to run the algorithms on the datasets I get the following error 'could not convert string to float: S0'

This error is produced during the during the evaluation of the 5 models; Logistic Regression, Linear Discriminant Analysis, K-Nearest Neighbors, Classification and Regression Trees, Gaussian Naive Bayes and Support Vector Machines.

Here is the code that I am using to evaluate the datasets:

#Load KDD dataset

dataset = pandas.read_csv('Datasets/KDDCUP 99/kddcup.csv', names = ['duration','protocol_type','service','src_bytes','dst_bytes','flag','land','wrong_fragment','urgent',
'hot','num_failed_logins','logged_in','num_compromised','root_shell','su_attempted','num_root','num_file_creations',
'num_shells','num_access_files','num_outbound_cmds','is_host_login','is_guest_login','count','serror_rate',
'rerror_rate','same_srv_rate','diff_srv_rate','srv_count','srv_serror_rate','srv_rerror_rate','srv_diff_host_rate',
'dst_host_count','dst_host_srv_count','dst_host_same_srv_rate','dst_host_diff_srv_rate','dst_host_same_src_port_rate',
'dst_host_srv_diff_host_rate','dst_host_serror_rate','dst_host_srv_serror_rate','dst_host_rerror_rate','dst_host_srv_rerror_rate','class'])

# split data into X and y
array = dataset.values
X = array[:,0:41]
Y = array[:,41]

# Split-out validation dataset
validation_size = 0.20
seed = 7
X_train, X_validation, Y_train, Y_validation = cross_validation.train_test_split(X, Y, test_size=validation_size, random_state=seed)

# Test options and evaluation metric
num_folds = 7
num_instances = len(X_train)
seed = 7
scoring = 'accuracy'

#  Algorithms
models = []
models.append(('LR', LogisticRegression()))
models.append(('LDA', LinearDiscriminantAnalysis()))
models.append(('KNN', KNeighborsClassifier()))
models.append(('CART', DecisionTreeClassifier()))
models.append(('NB', GaussianNB()))
models.append(('SVM', SVC()))

# evaluate each model in turn
results = []
names = []
for name, model in models:
    kfold = cross_validation.KFold(n=num_instances, n_folds=num_folds,

random_state=seed)

    #Here is where the error is spit out
{
            cv_results = cross_validation.cross_val_score(model, X_train, Y_train, cv=kfold, scoring=scoring) # Could not convert string to float happens here. Scoring uses string.
            results.append(cv_results)
            names.append(name)
            msg = "%s: %f (%f)" % (name, cv_results.mean()*100, cv_results.std()*100)#multiplying by 100 to show percentage
            print(msg)
}

# Compare Algorithms
fig = plt.figure()
fig.suptitle('Algorithm Comparison')
ax = fig.add_subplot(111)
plt.boxplot(results)
ax.set_xticklabels(Y)
plt.show()

Here is a 3 line sample from the KDDcup99 datatset:

0   tcp http    SF  215 45076   0   0   0   0   0   1   0   0   0   0   0   0   0   0   0   0   1   1   0   0   0   0   1   0   0   0   0   0   0   0   0   0   0   0   0   normal.
0   tcp http    SF  162 4528    0   0   0   0   0   1   0   0   0   0   0   0   0   0   0   0   2   2   0   0   0   0   1   0   0   1   1   1   0   1   0   0   0   0   0   normal.
0   tcp http    SF  236 1228    0   0   0   0   0   1   0   0   0   0   0   0   0   0   0   0   1   1   0   0   0   0   1   0   0   2   2   1   0   0.5 0   0   0   0   0   normal.

I have tried using label encoding and it still spits out the same error and when I was looking through the sklearn websites, I noticed that the scoring value was for the string type, is this the cause of the issue? and if not, is there a problem with the way I have loaded the dataset?

EDIT I tried removing scoring value from the code and still got the same error.

I notice you mentioned that you used Label encoding but I did it myself and the code runs just fine. I used the 10 percent version of the dataset . Just put this piece of code after you load the dataset:

for column in dataset.columns:
    if dataset[column].dtype == type(object):
        le = LabelEncoder()
        dataset[column] = le.fit_transform(dataset[column])

After label encoding you should use a One Hot Encoder to improve the performance of some algorithms. You should also avoid using cross_validation module as it is deprecated, it will be removed in version 0.20.


Custom conditional loss function in Keras I'm looking for a way to create a conditional loss function that looks like this: there is a vector of labels, say l (l has the same length as the input x), then for a given input (y_true, y_pred, l) the loss should be:

def conditional_loss_function(y_true, y_pred, l):
    loss = if l is 0: loss_funtion1
           if l is 1: loss_funtion2
    return loss

It is like a kind of semi-supervised loss funtion.

You should be able to solve this with currying. Make a function that takes the label as input and returns a function which takes y_true and y_pred as input. Note that the label needs to be a constant or a tensor for this to work.

def conditional_loss_function(l):
    def loss(y_true, y_pred):
        if l == 0:
            return loss_funtion1(y_true, y_pred)
        else:
            return loss_funtion2(y_true, y_pred)
    return loss

model.compile(loss=conditional_loss_function(l), optimizer=...)

Small working example with different loss function depending on the label:

# load pima indians dataset
dataset = numpy.loadtxt("pima-indians-diabetes.csv", delimiter=",")
data = dataset[:,0:8]
label = dataset[:,8]

X = Input(shape=(8,))
Y = Input(shape=(1,))
x = Dense(12, input_dim=8, activation='relu')(X)
x = Dense(8, activation='relu')(x)
predictions = Dense(1, activation='sigmoid')(x)

def custom_loss(l):
    def loss(y_true, y_pred):
        if l == 0:
            return binary_crossentropy(y_true, y_pred)
        else:
            return mean_squared_error(y_true, y_pred)
    return loss

# Compile model
model = Model(inputs=[X, Y], outputs=predictions)
model.compile(loss=custom_loss(Y), optimizer='adam', metrics=['accuracy'])

# Fit the model
model.fit(x=[data, label], y=label, epochs=150)

# evaluate the model
scores = model.evaluate([data, label], label)
print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))


How many epochs to run during hyperparameter search? If I'm doing a hyperparameter search and comparing two different hyperparameters (but not number of epochs), is there some established rule of thumb for how many epochs to run?  If I just compare after a few epochs, will that give me a good idea about how it will perform fully converged (say for example after 1000 epochs).  Is there any research papers where people have studied this question?

The general rule of thumb is to run the number of epochs until validation error starts to increase.
Sometimes fast initial learning will not lead to the best performance later.


what ETL technique should i use for text documents using Hadoop? I have a school Big Data project where basically the teacher is going to give us a large amount of text documents (from the Gutenberg project data set ) and he want us to give as output the document where a "keyword" is more relevant, he also wants us to divide the project in 3 parts:

Data acquisition, preprocing (cleaning, transform, join, etc), loading. ETL PROCESS.
Data processing.
User friendly application.

I need to define what technologies or methods i'm gonna user for each of the parts of the project, but i have no idea what to do on the ETL part since the documents are gonna be written in legible English (they are books), i will appreciate any information you can give me on this but also about the other parts of the project.

Thanks a million for reading.

I would suggest a data fabric. That would meet your need for data acquisition, preprocessing, data quality, master data management, etc.

Given I work for Talend, I would suggest our data fabric. =)

Here’s a case study with the Panama Papers. https://www.talend.com/blog/2017/01/17/talend-data-masters-2016-icij-decoded-panama-papers-talend/

The concept in that case study of combining a data fabric with analytic tools is the general concept to use regardless of which data fabric you use.

You can find the trial and open versions of Talend at https://www.talend.com/download/.

Edit: Here’s another example, which implements a backend and UI showing the ebooks of the Gutenberg project. It allows to import the whole gutenberg index using a camel route. https://github.com/Talend/tesb-rt-se/tree/master/examples/tesb/ebook


Difficulties training a classifier I'm a begginer in data science but I tried to build a classifier for my own bank transactions, I collected ~ 50.000 in total. My intention is to create a relationship between the statement of the transaction and the type of transaction. For example:
Statement: Payment with card number XXXXXX in Wallmart.
Type/label: Buy in a supermarket
For doing so I trained my model with 80% of the total of statements (previously labeled) and I reached a so high accuracy (85%). After that I realized that something was wrong, there are so many statements with the text:
Statement: Payment with card number XXXXX in _______ .
Because this is a "default" statement text in my bank when someone buys a product in a supermarket. Then, if the model tries to find out in what category the statement is it will be pretty simple because there will be many statements repeated and the probability of predict one of those will be high. What I did later was trying to delete all the repeated statements, since the repeated ones will not affect in the training process, but because of this my total of statements went down to ~ 2.000, and reaching then only a 10% of accuracy in the prediction.
My questions are: What is correct process to train my model? Should I let the repeated statements? (I feel like I'm cheating the prediction letting the repeated ones) What I'm doing wrong?

If I understand correctly, you noticed that a lot of correct predictions were due to a case which always has the same label and is very frequent in your data, right?
Let me say that I think you had a good reasoning: you analyzed the problem, designed a potential solution and then tried it. This kind of problem is very common, but in this case the solution is not really to delete the repeated cases in your training data.
What happens is that you have imbalanced data, that is you have a class which appears very frequently: when you deleted the very frequent instance you mentioned, your data size went from 50000 to 2000, so this instance represents 96% of your data. This instance has probably almost always the same label, so my guess is that this instance with this class as label represents 85% of your data. At first you thought your classifier was working well since you had 85% accuracy, but it's very easy for the classifier to reach 85% of correct predictions if the most frequent class represents 85% of the data: it just needs to always answer this class.
The first lesson here is that accuracy is not a good evaluation measure for imbalanced data: it's too simple, it can be very high even though the classifier doesn't do anything useful. That's why you should probably look at precision, recall and f1-score by class, and typically at macro f1 score as a global evaluation score.
Second point, I'm going to mention resampling because it's usually the standard answer to imbalanced data: the idea is simply to artificially "re-balance" the distribution of the classes in the data by either:

undersampling the frequent class, that is using only a random subset of this class. Note that your idea to delete the frequent instance is actually not so different.
oversampling the other classes, that is repeating the rare instances as many times as necessary to make the data balanced.

Importantly, the resampling must be done only on the training set, otherwise the evaluation on the test set is biased.
Resampling is quite easy, but the results are often disappointing. This is because quite often the problem is deeper: the classifier is just not able to distinguish the classes from the information provided by the features.
That brings me to my final point: the fact that you get 10% accuracy after removing the frequent case shows that the classifier doesn't have a clue what it's doing. Most of the time designing a good ML system needs some serious work with the features, especially with text data. The type of model (and parameters) can also play an important role. You didn't explain what kind of model and what kind of features you use, so there's not much I can say about this. Don't hesitate to post a new question with these details, and maybe we can help you improve this model.


How do I use recency/time as a feature after I've a word2vec model I have a word 2 vector model which works great but let's say I typed Eminem.

And it gives me:

  rap, music, revival and recovery.

How do I make another layer that shows revival first?

The purpose of the Word2vec is it will itself learn a hidden structure in your text data. If "recovery" comes after rap and music is because Eminem is more frequent with these context words than with recovery. You can't simply change that. You should avoid making a manual intervention on your model.

However, here are 2 ideas you could try if you really want "recovery" to be closer of Eminem word :

Try an other Word2Vec algorithm architecture. There are CBOW and skip-gram you can try. In my opinion CBOW could perform better because the CBOW model trains each word against its context. It asks "given this set of context words, what missing word is likely to also appear at the same time?". Skip-gram trains each the context against the word. It asks "given this single word, what are the other words that are likely to appear near it at the same time?"
Add sentences relative to recovery album in your data. Because words recovery and Eminem should dominate in these sentences, It could reconcile the 2 words. Same, you can replicate sentences containing both Eminem and recovery word and randomly change the order of words in the sentence.


Finding user similarities within informal data sets I'm new to all this and am putting together a learning project. I've decided on finding similarities between users in a data set such as http://en.wikipedia.org/wiki/Enron_Corpus. After doing a bit of research, I also came across Dataset for Named Entity Recognition on Informal Text. So I'm not short of data or a goal, I need to understand high-level techniques to get there.

One valuable comment noted that this question appears too broad. What I was hoping to find with this question was the breadth of techniques I should focus research on, not answers that are immediately implementable. Please consider vague answers as entirely appropriate!!

Expanding on the goal, I am hoping to discover which authors might have affinity toward each other, or conversely do not care much for each other. So I will definitely need to start with Named Entity Recognition and build a means to organize the documents against those entities. Beyond that, I am not so sure.

What high level concepts should I be looking at? Thanks!

As you accept vague answers:
Sranford NLP tools are strong for this kind of stuff. NER, POS Tagger maybe, Parsers, etc. Now for machine learning itself, I would try looking up at WEKA, it's got a lot of filtering, classifiers and clustering methods, including StringToWordVector filters, that are, in my opinion, fundamental to text classification. Mostly, the tags you should be looking for are Text Categorization, Natural Language Processing and even Sentiment Analysis if you will.


python///TypeError: unsupported operand type(s) for -: 'list' and 'list' ,i write the code and i get this massege N = int(input('number of traverse='))
x = []
y = []
for r in range(N):
    x.append([float(input('enter x='))])
for l in range (N) :
    y.append([float(input('enter y='))])

N=N-1
b=int(N)
area = x[0] * (y[1] - y[b]) + x[b] * (y[0] - y[b-1])

for k in range(1, N):
    area = area + x[k] * (y[k+1] - y[k - 1])
    i=i+1
area = .5 *abs(area)**strong text**
```

The problem is x[0] etc are resulting in a 1D list. You need to extract element from 1D list.. below works (I used index 0 element)

area = x[0][0] * (y[1][0] - y[b][0]) + x[b][0] * (y[0][0] - y[b-1][0])


data science / machine learning resources? In a few weeks I'm starting a new job that will be involved in machine learning and data science.

I have a masters degree in probability / mathematics but I have no knowledge of machine learning and data science.

Are there any online resources like courses or books that could help me prepare for my new job? Basically I'm trying to get a head start on my new job.

Right now the only things I really know related to this fields are regression methods like linear regression and logistic regression, and I have a decent grasp of mathematical concepts so I'd prefer something that is not totally dumbed-down

Start with the Coursera's Machine Learning course. It does a really good job in introducing the student to the domain of Machine Learning and helps you lay a solid foundation in the concepts.

In case, you feel that the math is a bit dumbed down in that course, you can take this course, taught by the same professor and is more math-intensive than the former.

Now, you would have a clear intuition about the basic concepts of Machine Learning. Take this course, which can be said as a follow-up or a supplement to the course of Andrew Ng.

This resource from IAPR has in-depth notes on a lot of ML concepts like cross-validation, regularization, etc.

You can also have a look at these amazing list of resources compiled into a blog on Quora.

Now, for diving into advanced concepts of neural networks and deep learning, you can make use of this free book.

Finally, the free e-book: Elements of Statistical Learning is a wonderful book for beginners in ML or Statistical Learning.


Feature extraction of accelerometer data for machine learning Currently, I am doing a project with the aim of classifying potholes through machine learning. The data collected is from an accelerometer in which the z-axis measures the "vertical" acceleration of the car, when a pothole is struck.

I have tried to deconstruct the signals and create features using two methods:

PACF along with a moving average to combat the noise
Calculated a periodogram for spectral analysis

Is there any other ways to differentiate a pothole from "rough" road surfaces, as I am unsure which approach to take?

Actually you could treat your acceloremeter signal like normal audio signals. There are endless possibilities for processing audio data (e.g. chroma features).

Another way would be to directly process your raw signal with the help of a neural network (1D-convolution or LSTM).


Questions on ensemble technique in machine learning I am studying the ensemble machine learning and when I read some articles online, I encountered 2 questions.

1.

In this article, it mentions

  Instead, model 2 may have a better overall performance on all the data
  points, but it has worse performance on the very set of points where
  model 1 is better. The idea is to combine these two models where they
  perform the best. This is why creating out-of-sample predictions have
  a higher chance of capturing distinct regions where each model
  performs the best.

But I still cannot get the point, why not train all training data can avoid the problem?

2.

From this article, in the prediction section, it mentions

  Simply, for a given input data point, all we need to do is to pass it
  through the M base-learners and get M number of predictions, and send
  those M predictions through the meta-learner as inputs

But in the training process, we use k -fold train data to train M base-learner, so should I also train M base-learner based on all train data for the input to predict?

Instead, model 2 may have a better overall performance on all the data
  points, but it has worse performance on the very set of points where
  model 1 is better. The idea is to combine these two models where they
  perform the best. This is why creating out-of-sample predictions have
  a higher chance of capturing distinct regions where each model
  performs the best.

It's not about training on all the data or not. Both models trained on all the data. But each of them is better than the other at different points. If I and my older brother are tying to guess the exact year of a song, I will do better in 90s songs and he in 80s songs - it's not a perfect analogy but you get the point - imagine my brain just can't process 80s songs, and his can't process 90s songs. The best is to deploy us both knowing we each have learnt different regions of the input space better.

  Simply, for a given input data point, all we need to do is to pass it
  through the M base-learners and get M number of predictions, and send
  those M predictions through the meta-learner as inputs

k-fold is still just one learner. But you're training multiple times to chose parameters that minimize error in the left-out fold. This is like training only me on all the songs showing me k-1 folds of data, and I calibrate my internal model the best I can... but I'll still never be very good at those 80s songs. I'm just one base learner whose functional form (my brain) isn't fit for those songs. If we could bring the second learner along, that would improve things.


What is a latent space vector? I do not understand this about GANs.

Apparently the Generator is supposed to receive a latent space vector as its input. Yet I couldn't find an example of how I can implement it in Pytorch. This is a problem for me, because different posts suggest different approaches.

Is it simply an image of Gaussian noise which is then served as an input to Generator's Convolutional Neural Network or is it a one-dimensional array passed through a Fully-Connected layer to the Generator?

The latent vector $z$ is just random noise.

The most frequent distributions for that noise are uniform: $z \sim U[-1,+1]$ or Gaussian: $z \sim \mathcal{N}(0, 1)$ . I am not aware of any theoretical study about the properties derived from different priors, so I think it's a practical choice: choose the one that works best in your case.

The dimensionality of the noise depends on the architecture of the generator, but most of the GANs I've seen use a unidimensional vector of length between 100 and 256.

In PyTorch, torch.Tensor.random_ and torch.randn can respectively be used to generate uniform and Gaussian noise.


Classification/Regression Problem where Response Variable is Ordinal Data Science novice here!

I'm trying to work on the white/red wine quality data set, where I 'm trying to predict the quality of the wine. All the features are numerical.

The response variable however, is ordinal with a quality score of integers 1 through 10 . I have seen tutorials try to group the scores ie: (0-4: Bad, 5-7: Good, 8-10: Great), but what if I wanted to predict the score as it is?

Should I use a regression approach where I try and minimize the error of my predicted scores versus actual scores?

Or should I use a classification model anyways and instead of calculating a F-score to evaluate the model, find the model that minimizes a cost function?

Or perhaps there is another approach that works best?

You can treat this problem as multiclass classification and use classification algorithms to solve this problem. You can use any of the classification metrics like accuracy, precision, recall etc.

Regression should be used only for the continuous data and continuous data is the one which can have infinite number of potential values within a given range. For example, for your problem the range is $1-10$, if it is continuous data, then it will have values like $1.04783,6.92838,8.2381,3.999,5.0$ etc. In this case you can opt for Regression algorithm.


What is the origin of YOLO/darknet coordinates I am aware of that darknet/yolov3 outputs relative coordinates, would the coordinate (0,0) be at the bottom left, or at the top left?

I am confused as opencv2 seems to have the y=0 at the top of the image as is usual when working with matrices. Yet mathematicaly and intuitively (for me at least) the coordinate (0,0) should be at the bottom left.

(0, 0)  is top left.

Here an a helpful blog that goes through all the features in the output vector.

This is common in image processing. There are a few reasons that this is the convention in computer vision. Check here for a list, the accepted answer states:

  This is caused in the history. Early computers had Cathode Ray Tubes (CRTs) which "draw" the image with a cathode ray from the upper left corner to the lower right.


PySpark dataframe repartition What happens when we do repartition on a PySpark dataframe based on the column. For example

dataframe.repartition('id')

Does this moves the data with the similar 'id' to the same partition? How does the spark.sql.shuffle.partitions value affect the repartition?

The default value for spark.sql.shuffle.partitions is 200, and configures the number of partitions that are used when shuffling data for joins or aggregations.

dataframe.repartition('id') creates 200 partitions with ID partitioned based on Hash Partitioner. Dataframe Row's with the same ID always goes to the same partition. If there is DataSkew on some ID's, you'll end up with inconsistently sized partitions.


Metric to punish positive errors I'm involved in a ship transit time prediction project. Is about prediction the time that a ship's cargo takes to go from port A to B in order to contract the fastest carrier company and tell the client how long it going to take, because a several times a the carrier company takes more or less (most of the times more).
We have historical database with predictive variables and stuff, the point is if the real transit time of cargo is 10 days is worse to the model estimate 9 days than eleven days, even that the absolute error is 1.
So a need a custom metric to punish positive errors more than negative ones and punish absolute large errors as well, calling error = actual - predicted.
I thought in using a weighted mean squared error, but I don't know how to balance the weights. Do you guys may help me? Do you know some other metrics that meet that requirements?
Thanks very much!

How about using the difference of an exponential and a linear or a polynomial functions?
For example, if $x = actual - predicted$, then
$$L = w_1 (e^x-1) - w_2 x$$
The cost of error increases faster for positive values than for negative ones, and is a differentiable function of $x$. You can tweak $w_1$ and $w_2$ depending on your typical errors and your learning algorithm. This is an example, for $w_1 = w_2 = 20$:


XGboost with group-specific and individual-specific data My dataset consists of a combination of two databases. One database consists of individual-level data on the characteristics and compensation of the top five executive officers of big American companies. The second database consists of balance sheet information on the companies of these executives. I use 4 individual-specific variables (e.g. executive age) from the first database and about 15 to 20 company-specific variables from the second database (e.g. total company profit). This means that for executives of the same company, only 4 variables are unique to the specific executive and all company-specific variables are the exact same for all 5 executives. I want to use XGboost to predict executive compensation using these individual-specific and company-specific variables. Is it okay to use 5 executives from the same company or should I use only 1 executive (CEO) for each company? This would decrease the sample size from 8926 to 1703 observations, but it would ensure that all of my observations are truly “unique”. Thank you very much!

You can use all executives if you believe their compensation is governed by the same function. That is, all things equal, you expect all executives to have equal pay.
If not, perhaps you should add a feature that tells the function of each executive (CEO, CFO...).
Anyhow, the fact that part of the features is the same for multiple examples is not a problem.
It would be a problem if you had very many examples from one company and you're looking for a model that applies to all companies. If you used loads of examples of one company, your model would be biased towards the pay at that company, but since you have 5 from each company, I don't see the problem.


GANs (generative adversarial networks) possible for text as well? Are GANs (generative adversarial networks) good just for images or can they be used for text as well?
Like training a network to generate meaningful text from a summary.
UPD - quotes from the GAN inventor Ian Goodfellow.

GANs have not been applied to NLP because GANs are only defined for
real-valued data. (2016) source
It is not a fundamentally flawed idea. It should be possible to do at least one of the following... (2017) source

Yes, GANs can be used for text. However, there is a problem in the combination of how GANs work and how text is normally generated by neural networks:

GANs work by propagating gradients through the composition of Generator and Discriminator.
Text is normally generated by having a final softmax layer over the token space, that is, the output of the network is normally the probabilities of generating each token (i.e. a discrete stochastic unit).

These 2 things do not work well together on their own, because you cannot propagate gradients through discrete stochastic units. There are 2 main approaches to deal with this: the REINFORCE algorithm and the Gumbel-Softmax reparameterization (also known as the Concrete distribution). Take into account that REINFORCE is known to have high variance so you need large amounts of data to get good gradient estimations.

As an example of REINFORCE for textual GANs you can check the SeqGAN article. An example of Gumbel-Softmax you can check this article.

Another completely different option is not having a discrete stochastic unit as output of the generator (e.g. generating tokens deterministically in embedded space), hence eliminating the original problem of backpropagating through them.


How to create full graph from vertex data frame with igraph? Here is a DF which I take as a representation of my future vertices:

Id   Name
01   Alpha
02   Beta
03   Delta
04   Epsilon

I want to get an igraph representation of a full graph with these nodes in R. As far as I know igraph's graph.full function cannot take DF as input and I cannot really want to create an intermediary DF with all possible conection combination. Any clever idea?

All full graphs of the same order and direction are the same, so just make a full graph of the right size and then put your vertex data on the vertices:

> d = data.frame(Id=c("01","02","03","04"),Name=c("Alpha","Beta","Delta","Epsilon"),stringsAsFactors=FALSE)
> d
  Id    Name
1 01   Alpha
2 02    Beta
3 03   Delta
4 04 Epsilon
> g = graph.full(nrow(d))
> V(g)$Name = d$Name
> V(g)$Id = d$Id


How to run a saved TensorFlow Model? (Video Prediction Model) I was reading this paper. The code is available in GitHub. In the README.md file, they've mentioned how to train the model

python prediction_train.py

with optional parameters.
Can anyone please explain how do I use this model to predict a video sequence? I'm new to deeplearning and tensorflow. So, I'm not able to understand the code properly. My current task is to just run the code and see the output (i.e. the videos predicted by this model).

All I could understand was, it uses tensirflow saver to save the checkpoints. I'm guessing these checkpoints are the intermediate trained model after few epochs (2000 in this case). How to use these models to predict the next frames of a video?

Any help is greatly appreciated :)

The TensorFlow saver is used to save the weights of a specific model at some given point. When you want to use a trained model, you must first define the model's architecture (which should be similar to the one used for saving the weights), then you can use the same "saver" class to restore the weights:

with tf.Session() as sess:
    # Restore variables from disk.
    saver.restore(sess, "../my_saved_model.ckpt")

Regarding your initial question. I think that if you are just starting with deep learning and TensorFlow, this is the wrong place to start and you should first understand how TensorFlow works in general by applying it at easier tasks like image classification (start with MNIST).

From what I understand, you need to use the "construct_model" function and pass it your initial image sequence (video) and some action tensor, and it should output a predicted frames:

def construct_model(images,
                actions=None,
                states=None,
                iter_num=-1.0,
                k=-1,
                use_state=True,
                num_masks=10,
                stp=False,
                cdna=True,
                dna=False,
                context_frames=2):
"""Build convolutional lstm video predictor using STP, CDNA, or DNA.
  Args:
    images: tensor of ground truth image sequences
    actions: tensor of action sequences
    states: tensor of ground truth state sequences
    iter_num: tensor of the current training iteration (for sched. sampling)
    k: constant used for scheduled sampling. -1 to feed in own prediction.
    use_state: True to include state and action in prediction
    num_masks: the number of different pixel motion predictions (and
           the number of masks for each of those predictions)
    stp: True to use Spatial Transformer Predictor (STP)
    cdna: True to use Convoluational Dynamic Neural Advection (CDNA)
    dna: True to use Dynamic Neural Advection (DNA)
    context_frames: number of ground truth frames to pass in before
                feeding in own predictions
  Returns:
    gen_images: predicted future image frames
    gen_states: predicted future states


How do Bayesian methods in machine learning help with the problem of limited data? Can this be used for image classification? When reading about machine learning, I've often come across information stating that Bayesian methods in machine learning are effective when you only possess a limited amount of data. As someone who is planning to start learning machine learning, I was curious how it accomplishes this? Lastly, and more specifically, I was especially wondering how and/or if this can be applied to problems such as image classification, when there is a limited amount of data (limited number of images to train the algorithm on)?
Any textbook, research, or other references with regards to the latter would also be greatly appreciated.
Thanks.

The discussion of Bayesian reasoning and its advantages over frequentist reasoning is very wide. I refer you to the book “Pattern recognition and machine learning” by Cristopher Bishop which is a great book on Bayesian reasoning.


How to drop row where all columns BUT the timestamp are zeros in python I have a dataset that like any other has zeros and i need to get rid of them. The problem is that I want to delete rows where all the columns values but the timestamp are zeros

You can use difference to not look at the Timestamp column. Then sum over the horizontal axis looking for non-zero sum rows:

df.loc[df[df.columns.difference(['Timestamp'])].sum(axis=1) != 0]


Can a decision in a node of a decision tree be based on comparison between 2 columns of the dataset? Assume the features in the dataframe are columns - A,B,C and my target is Y

Can my decision tree have a decision node which looks for say, if A>B then true else false?

Yes, but not in any implementation that I am aware of.

The idea is mentioned in Elements of Statistical Learning, near the end of section 9.2.4 under the heading "Linear Combination Splits."  But this is not implemented in the popular CART or Quinlan-family of decision tree algorithms*, it is not done in sklearn's trees, and I do not know of any other python or R package that use it.

Some R packages do a more limited version, where a split can be made on two features, but these splits are of the form "$x_1>\alpha\text{ and }x_2\leq\beta$" as opposed to direct comparisons of the variables.  See https://stats.stackexchange.com/questions/4356/does-rpart-use-multivariate-splits-by-default

An obvious problem is computational requirements: just checking over all pairs of features is now quadratic, and allowing arbitrary linear combinations of two features is potentially much much larger.  On the other hand, if you want to restrict to direct comparisons $x_1\geq x_2$ (with no coefficients), that should be tractable (if substantially slower than CART).  The Elements authors suggests Hierarchical Mixtures of Experts model instead if incorporating linear combinations is desired.

Oh, one more comment.  If you really want splits like $x_1\geq x_2$, you could just generate all the features $x_i-x_j$; then a more common implementation of decision trees will be able to make your splits, when considering these new features.  (Probably there will be some side effects, and still the computational problem arises: you've added $\binom{m}{2}$ features.)

* I've found a comment that suggests that CART does support multi-feature ("surrogate") splits?:
https://stackoverflow.com/a/9996741/10495893


How to return states of LSTM in MXNet? I used MXNet previously to beat keras+tensorflow accuracy in CNN regression models.
Now I am trying to implement LSTM, which in keras runs fine:

from keras.layers import LSTM,Flatten,Input
from keras import Model
import numpy as np

def make_keras_stacked_lstm():
    inp = Input(shape=(100,1))
    lstm1 = LSTM(16, return_sequences=True)(inp)
    lstm2 = LSTM(1, return_sequences=True)(lstm1)
    outp = Flatten()(lstm2)
    return Model([inp], outp)

def keras_main():
    ins = np.random.uniform(size=(1000,100,1))
    outs = np.random.uniform(size=(1000,100))
    model = make_keras_stacked_lstm()
    model.compile(optimizer='sgd', loss='mean_squared_error')
    model.fit(ins, outs, epochs=1, validation_split=.1)

if __name__ == '__main__':
    # main()
    keras_main()

How can I translate this to MXNet, in either Symbol or gluon dialect? I found no "return_sequences=True" analogue.

The best emulation (although I didn't check full equivalency):

import mxnet as mx

SEQUENCE_LENGTH = 100

stack = mx.gluon.rnn.SequentialRNNCell()
stack.add(mx.gluon.rnn.LSTMCell(16, prefix='first')
stack.add(mx.gluon.rnn.LSTMCell(1, prefix='first')

x, _ = stack.unroll(length=SEQUENCE_LENGTH,
    inputs=x,
    layout='NTC',
    merge_outputs=True,
)

Here layout='NTC' is for the case when input shape is (samples, timesteps, channels).


improving accuracy of classification I have data with 95 numeric variables and 5 categorical variables. My Y has 2 values. I built a decision tree and my accuracy was 81.8%. Then I created 3 new variables as follows. It improved accuracy to 84.3%

Normalize numeric variables and for training data, find mean vector for Y=1 and Y=0
for each data     point, find euclidean distance from each mean vector - distance0 and distance1
third variable will be 0 if distance0 is <= distance1

I was wondering if there is any other new variables that i could create to improve the accuracy

I used a decision tree as it is fast to build and gives me indication whether a newly created variable is useful or not.

Please let me know if you have any thoughts

You could cluster your data with something like k-means and use the assigned cluster as a new feature.

import numpy as np
import pandas as pd
from sklearn.datasets import load_iris
from sklearn.cluster import KMeans
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split

data = load_iris()

X = pd.DataFrame(data['data'], columns=data['feature_names'])
y = data['target']

X_train, X_test, y_train, y_test = train_test_split(X.copy(), y, test_size=0.2)

km = KMeans(n_clusters=3)
km.fit(X_train)
training_dist = km.transform(X_train)

X_train['dist1'] = training_dist[:,0]
X_train['dist2'] = training_dist[:,1]
X_train['dist3'] = training_dist[:,2]

clf = DecisionTreeClassifier()
clf.fit(X_train, y_train)

# new data
test_dist = km.transform(X_test)

X_test['dist1'] = test_dist[:,0]
X_test['dist2'] = test_dist[:,1]
X_test['dist3'] = test_dist[:,2]

clf.predict(X_test)