Understanding genetic algorithms What is a genetic algorithm, and what are its practical advantages over other algorithms? Is it similar to any commonly used machine learning algorithm like linear/logistic regression, neural networks, or tree-based methods like gradient boosting and random forests? I've heard it's based on "mutated" combinations of other models. Does this make it more like an ensemble?

A genetic algorithm is an algorithm based on biological evolution, on how nature evolved. It does exactly that, evolve the algorithm so that "it" finds the best solution to the problem at hand. You can use a genetic algorithm to find a solution to a problem which you don't know the answer, you know the answer but want to know a different one, or you are just lazy. The common steps for a genetic algorithm are:

Generate a random population of elements
Evaluate fitness of each element (how good are they against the
solution)
Take the best elements for the next generation
Generate child elements using the above elements
Mutate (randomly) each child. This can also be done before step 4 with parents.
Repeat from step 2 for n number of generations until the solution is found, the maximum number of generations have been reached or the fitness is not changing anymore (local minima).

Two basic problems with GAs:

To get trapped in a local minima (or local maximum depending on the point of view). This means that you might find a good answer (or not even a good one) but it will never reach the best answer because fitness values are not changing or not getting better. This is fought using Mutation in step 5 to keep diversity in the population so it doesn't get stuck.
They are generally slower than other approaches.

If you can, take a look at this book. It is not free, but worth a look.
Alternatively, take a look at this online book for free, it is a great source and he has his own youtube channel. It's more basic than the other book I recommend, but will help you get started with GAs.

To answer the other part of your question, the GA should be used to find a model, but will not act as a model. For example, if you have a neural network you can train it using the backpropagation method, but you could also train it using a GA. The GA will not use anything of the backpropagation mathematics, it could be used to generate weights in its neurons, and evaluate the answer (last layer). Weights will evolve to get closer to the solution. In this scenario, the model will still be the NN, but you used a different algorithm to find the best NN.

Hope this helps.


How not to see the output in the console when adding a new column in a table? I have a data table called td and it contains 14k obs and 5k variables.
When I run the command

td[,fv_mc2:=1]

I also get the whole data table as output in my console, which floods my console. Any ideas how to fix it ? (It might also be a setting in R, but I am not aware of it.

R by default returns the updated dataframe to console.You can assign it to variable by avoiding it.

td <- td[,fv_mc2:=1]

By this way you can avoid the console of 14k data.


How are weights represented in a convolution neural network? I have been trying to develop a convolution neural network following some guides online. However, most guides I have encountered gloss over an important detail, which is how to programmatically represent the weights in a CNN.

As far as I understand, in a "regular" neural network, the weight of a connection is a numerical value, which is adjusted in order to reduce the error; then back-propagation is used to further update the weights, reducing thus the error, etc.

However, in a CNN, the input is an array of numbers (the image), and a subset of those (the filter) to calculate the mean error, by multiplying the filter pixels by the original pixels.

So, is there a weight neuron for each filter (kernel or feature map) of the image? Or is a single weight neuron represented by the sum of all the mean error's calculated from convolving the filter over the receptive field, such that you have one value, in the end, that is the total error for the entire image?

In convolutional layers the weights are represented as the multiplicative factor of the filters.

For example, if we have the input 2D matrix in green

with the convolution filter

Each matrix element in the convolution filter is the weights that are being trained. These weights will impact the extracted convolved features as

Based on the resulting features, we then get the predicted outputs and we can use backpropagation to train the weights in the convolution filter as you can see here.


Cross Correlation Between Input-Output Sine Waves I am writing an algorithm to estimate the frequency transfer function of the system. For this, I want to use the Cross-correlation Between Input-Output Sine Waves method. There are a few things I don't understand about method:
I - What does capital N mean?
II - Is and Ic scalar or vector(depend on N)?
These are my thoughts :
I - N represents the number of outputs I collected for a specific w value with a sampling periode.
II - Scalar.
Could you help?

These formulas are discrete fourier transform.

It's very standard procedure when you analyze wave-alike processes. What it does it transforms the signal from the time-space to the frequencies-space. The idea is that you take sin-wave with specific frequency, multiply it by signal and if the value is big then you have something similar to the wave of this frequency.

$\frac{1}{NT}$ term is just overall time of the sequence (N samples with length T)

The $I_{s}$ and $I_{c}$ is amplitude(intensity) for the sin and cos, respectively. You get them for the specific frequency $\omega$ and it's a single number for each. From this you can get full amplitude (G) and phase $\phi$. Why do you need phase? Because it's important - if you have two waves with similar frequency and $\phi = \pi$, then they would cancel each other, and if they have same phase, they would double.


Classify phrases as biomedical or non-biomedical Words like myotonic are biomedical, but words like new appear in regular English texts. I can imagine a few ways to classify words into biomedical and non-biomedical. Has this problem been tackled before? Does anybody know any ready-to-use solutions for this problem?

Yes. There has been work going on in the healthcare domain which involves NLP.

Resources like Pubmed have been put together for the exact same purpose.

You can make use of the freely available Pubmed vectors for training your models like maybe a sentence tagging style model for identification. We(my team at work) use them for identification of medical terms like diseases, medicines, etc.


Clustering of variants of similar news articles We have data of several news sites, having quite literally millions of entries. As each news site publishes their own version of the news (also each news site may publish several different version of the same news), we have several entries that are variants of a single news. I am currently working on separating out "Unique" news from our repository. That means if a single news has several variants, only a single variant (most likely the one reported earliest) will be considered.

I believe, clustering of the news articles can be used to group together similar news. I am currently exploring DBSCAN, and Hierarchical clustering (Ward's Method). I am wondering whether am I moving in the right direction, is clustering the best solution for solving our problem? If yes, which other algorithms and techniques should I explore?

You don't want clustering.

What you are looking for is near duplicate detection.

Use minhash. Apparently that is what Google News uses for exactly this purpose.


How to predict whether the client will renew the subscription or not based on given data structure I have a requirement where I want to predict whether the client will renew the subscription or not.
And the data is something like below.

Basically client's subscription end date can be anything. And I want to predict whether client A will renew the subscription in Oct or not.
And Similarly client B will renew it in December or not.
And I want to run the model every day because every client's subscription end date can be anything.
Along with this I have various features of the client i.e. Feature 1, 2, 3 etc which can be used to train the model.
My question is, how do I structure the data so that I can run the model everyday and what would be my output variable. What kind of model I can fit.
I was thinking of survival analysis but not sure if  the data structure I have can be used for survival analysis or not.
Need your suggestion to approach this problem.

That data structure is fine. You need to have a dataset of historic subscriptions (I.E. subscriptions that have now finished), where the output variable is "Renewed/ Not renewed". You can train your model on that dataset and then make predictions each day as to whether or not each "Active" subscription is likely to be renewed when it reaches the end date.

As to what kind of model you can train, the "usual suspects" should be fine for this. LogisticRegression, XGBoost, RandomForests and so on can all handle this kind of problem.

EDIT

So lets say for example I have a 3 month subscription that runs from 1st Jan to 31st March. I decide to renew for another 3 Months until 30th June. I then renew again until 30th September. YOU have the same subscription from 1st Jan to 31st March, and you also renew until 30th June. You then decide NOT to renew. In that case, your data should look like this:

The Status column shows subscriptions which are Finished, and which are active. You should train the model against Finished subscriptions only and make predictions against Active ones. The target variable is the "Renewed" feature. For testing purposes you should split the Finished subscriptions into a 75/25; train the model on the 75% and test it against the 25%

Hope that clarifies.


Getting results of folds in orange canvas I am using Orange Data mining 3.8 to classify a dataset using leave one out cross validation.

I know that the results ($AUC$, $CA$, $F1$) are averaged; How can I get the results ($AUC$, $CA$, $F1$) of every fold?

For AUC, you can not. Leave-one-out uses only one data instance in the training set. Computation of AUC requires more data.

For general cross-validation (say, 10-fold cross-validation), there are several ways to examine what happened in each fold. Test & Score widget that performs cross-validation outputs a data table with predictions for each of the data instances in the test set. Together with predicted probabilities, predicted class, original class, the data also include information about the fold. You can then select the results by the fold and analyze it further, say, in a box plot.

To compute evaluation statistics for each of the folds, use Data Sampler, chose "Cross Validation" for sampling, select the desired fold, and wire the connection between Data Sampler and Test & Score widget as shown below. Notice that in this setting the Data Sampler outputs the fold data as a sample (to be used as test data), and out-of-fold data as remaining data (to be used as training data).


Best forecast model for insurance policies volumes I am new in forecasting and I am studying a dataset from an insurance company that contains the volume on a monthly basis of new policies, renewals & cancellations. New policies of a given month are renewed in intervals (3 months, 6 months, 12 months) but could be canceled as well at any time. For instance, new policies of January with 3-months duration are renewed after 3 months in April.
I would like some help in what direction to study in order to build a forecast model that will predict the amount of renewals (R) and cancellations (C) of a given month based on the input of new policies (N) of previous months.
Thank you

I assume Renewals/Cancellations are your expected output, so it can be considered as a binary classification.
From where I stand, all you mentioned (policies) in your first paragraph is meaningless because I cannot see how I can use them to build any model. Rather, you better study what the clients who used to renew/cancel respectively look like. And that is the mighty "feature"! For example, you might take into consideration their gender, age, ethnicity, and annual income, etc. That would be more helpful than thinking only about your policy.
Or, you can also try a probabilistic solution: think about the probability of the occurrence of cancellation of your company in the past, and the probability of the occurrence of cancellation of other companies given that they execute the same policy. What can you predict?


receive value error decision tree classifier after one-hot encoding I am trying to build a decision tree model. After one-hot encoding, it seems somehow the data still has a problem. When I run the following code, I receive this error:

ValueError: Number of labels=172 does not match number of samples=540

#Code:
import pandas as pd
import numpy as np

df = pd.read_csv("https://library.startlearninglabs.uw.edu/DATASCI420/Datasets/Bank%20Data.csv", sep=",")
df.columns=['age', 'sex', 'region', 'income', 'married', 'children', 'car',
       'save_act', 'current_act', 'mortgage', 'pep']
df.info()

(nrows, ncols) = df.shape
colnames = list(df.columns.values)
string_encoding = {}
df_encoded = df.copy()
for i in range(ncols):
    levels = list(set(df.iloc[:, i]))
    num_levels = len(levels)
    string_encoding_i = dict(zip(levels, range(num_levels)))
    string_encoding[colnames[i]] = string_encoding_i
    for j in range(nrows):
        df_encoded.iloc[j, i] = string_encoding_i[df.iloc[j, i]]

print(string_encoding)
print(df_encoded.head())

# One Hot Encoding Categorial Variables
from sklearn import preprocessing
enc = preprocessing.OneHotEncoder()

non_categorial_features = ['age',
                          'income',
                          'children',
                          'pep']

for categorical_feature in list(df.columns):
    if categorical_feature not in non_categorial_features:
        df[categorical_feature] = df[categorical_feature].astype('category')

df_with_dummies = pd.get_dummies(df, sparse=True)

df = pd.concat([df, df_with_dummies], axis=1)

df.head(5)

df = df.drop(['sex', 'region', 'married', 'car',
       'save_act', 'current_act', 'mortgage', 'pep_NO', 'pep_YES'], axis=1)

df.head()
df.info()

from sklearn import tree
import numpy as np
from sklearn.model_selection import train_test_split

# prepare for decision tree
np.random.seed(101)

title_names =['age', 'income', 'children', 'counts', 'age', 'income',
       'children', 'sex_FEMALE', 'sex_MALE', 'region_INNER_CITY',
       'region_RURAL', 'region_SUBURBAN', 'region_TOWN', 'married_NO',
       'married_YES', 'car_NO', 'car_YES', 'save_act_NO', 'save_act_YES',
       'current_act_NO', 'current_act_YES', 'mortgage_NO', 'mortgage_YES', 'pep']

df = df[title_names]

X = df.iloc[:,0:23]
Y = df.iloc[:, 23]
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size = 0.1, random_state = 99)

# decision tree
from sklearn.tree import DecisionTreeClassifier

# Use entropy = no limit on samples for split
model_ent = DecisionTreeClassifier(criterion='entropy').fit(X_train, y_train)
y_ent_pred = model_ent.predict(X_test)

# Use information gain (default) limit min_samples to 4
model_gini = DecisionTreeClassifier(min_samples_leaf=5).fit(X_train, y_train)
y_gini_pred = model_gini.predict(X_test)
```

It looks like Y is a SpareSeries as well as y_train and y_test. So when that is passed to the decision tree fit method, it only interprets those entries with label 1 as existing. According to the pandas documentation:

  We have implemented “sparse” versions of Series and DataFrame. These are not sparse in the typical “mostly 0”. Rather, you can view these objects as being “compressed” where any data matching a specific value (NaN / missing value, though any value can be chosen) is omitted.

I'm not sure why it is a sparse data structure, but you can use the to_dense method to densify it:

Y = df.iloc[:, 23].to_dense()

Edit: Danny below mentions you could just remove Sparse=True from get_dummies.


Can I apply Hidden Markov Models this way? I have just gotten my feet wet with Hidden Markov Models. Now I want to apply them to tell whether a transaction from an ATM is suspicious or not. I have great confusion in defining my Hidden States.

My attempt: A transaction is either suspicious or non-suspicious. Since I cannot tell, ("this is hidden to me"), whether a transaction is suspicious or not before analyzing it, therefore my Hidden States(Suspicious, Non-Suspicious). Does this hold?

My great confusion: When I read about Hidden Markov Models on Weather example, I understood it this way: "weather can be rainy or sunny - I mean its the 'same' weather that is switching between the two states". In my transaction case - one transaction cannot switch between being suspicious and non-suspicious. So, does my hidden states hold?

Questions: Does my hidden states hold? Can I apply the HMM that way? Please advice, thanks.

P.S: I had my Observations as location, time and amount of money but as of now I want clarification on Hidden states.

This is a reasonable application for a hidden markov model only if you have reason to believe that the transactions have the markov property - that the next state depends only on the present state, and not on any of the previous ones. This often makes sense for models in time series (like in the weather example, in which the fact that it's already raining makes it more likely that in the next time step it will still be raining). Of course, the model can be useful even if it's not a perfect fit.

The way you've formulated the model is saying that the hidden state (suspicious/non-suspicious) of an ATM may be predictive of the emitted, or observed states (location, time, and amount of money). At least for the first two, this feels a bit off, particularly if there are multiple ATMs involved, as is suggested by the presence of a location variable. I'll try to explain what I think the way you've formulated the model suggests in terms of observed variables and how it might be nonsensical, and suggest some alternatives.

First, this seems to suggest that suspicious/non-suspicious is a shared state of all ATMs, and that that state somehow influences where an ATM is being used (each hidden state has a particular distribution from the location is drawn). It may make more sense to have a separate model for each ATM location. Maybe this is already what you were intending. If that is the case, location wouldn't really be an observed variable. Instead of thinking of the transactions switching between states, which doesn't seem to make much sense, as you say, perhaps you want to be thinking of the ATMs switching states.

Second, it's not clear how time would be a random variable that would be emitted based on a state of suspiciousness. Instead, time would probably make more sense as the way of ordering the observations you feed into the model.

If you do want to combine all transactions into a single model, maybe an SVM would be more straightforward as a binary classifier.


Why does a horizontal cross section of a squared error surface yield ellipsis? Also, can someone please explain why the descent happens in a direction perpendicular to the contour lines?

Hinton refers to a shape like this, where clearly the horizontal cross-sections are ellipses (or circles).

(In 3D space things can be perpendicular to each other in two ways)
So no matter how steep the gradient at one point is, we need to travel perpendicular (in one way) to the contour lines in order to reach the minimum of the bowl. Think about it :D


Why is the decoder not a part of BERT architecture? I can't see how BERT makes predictions without using a decoder unit, which was a part of all models before it including transformers and standard RNNs. How are output predictions made in the BERT architecture without using a decoder? How does it do away with decoders completely?

The need for an encoder depends on what your predictions are conditioned on, e.g.:

In causal (traditional) language models (LMs), each token is predicted conditioning on the previous tokens. Given that the previous tokens are received by the decoder itself, you don't need an encoder.
In Neural Machine Translation (NMT) models, each token of the translation is predicted conditioning on the previous tokens and the source sentence. The previous tokens are received by the decoder, but the source sentence is processed by a dedicated encoder. Note that this is not necessarily this way, as there are some decoder-only NMT architectures, like this one.
In masked LMs, like BERT, each masked token prediction is conditioned on the rest of the tokens in the sentence. These are received in the encoder, therefore you don't need an decoder. This, again, is not a strict requirement, as there are other masked LM architectures, like MASS that are encoder-decoder.

In order to make predictions, BERT needs some tokens to be masked (i.e. replaced with a special [MASK] token. The output is generated non-autoregressively (every token at the output is computed at the same time, without any self-attention mask), conditioning on the non-masked tokens, which are present in the same input sequence as the masked tokens.


Prediction based on more dataframes I'm trying to predict a score that user gives to a restaurant.

The data I have can be grouped into two dataframes

data about user (taste, personal traits, family, ...)
data about restaurant(open hours, location, cuisine, ...).

First major question is: how do I approach this?

I've already tried basic prediction with the user dataframe (predict one column with few others using RandomForest) and it was pretty straightforward. These dataframes are logically different and I can't merge them into one.

What is the best approach when doing prediction like this? What are the most common methods used for task like this?

My second question is what is the best way to handle categorical data (cuisine f.e.)?

I know I can create a mapping function and convert each value to index, or I can use Categorical from pandas (and probably few other methods). Is there any prefered way to do this?

First question - predicting user score:
One approach addressing this problem is using collaborative filtering.
Where the basic idea is 'placing' each entity type(users, restaurants) in a vector space, where a distance function can be utilized.
This way you can learn that if user1 is similar(short distance) to user2 and restaurant1 is similar to restaurant2 then its probable that both users will give similar score to both restaurants.
 One popular algorithm that uses collaborative filtering is the 'Matrix Factorization' - the matrix factorization example is in the bottom of the article.

Second question - handling categorical variables:
Using one-hot-encoding for handling categorical data is a common methodology.
I personally love using pandas.get_dummies function.


Unable to save the TF-IDF vectorizer I'm workig on multi-label classification problem. I'm facing issue while saving the TF-IDf verctorizer and as well as model using both pickle and joblib packages.

Below is the code:

vectorizer = TfidfVectorizer(min_df=0.00009, max_features=200000, smooth_idf=True, norm="l2", \
                         tokenizer = lambda x: x.split(), sublinear_tf=False, ngram_range=(1,3))
x_train_multilabel = vectorizer.fit_transform(x_train)
x_test_multilabel = vectorizer.transform(x_test)

classifier = OneVsRestClassifier(SGDClassifier(loss='log', alpha=0.00001, penalty='l1'), n_jobs=-1)
classifier.fit(x_train_multilabel, y_train)
predictions = classifier.predict(x_test_multilabel)

Error Message while saving the TF-IDF vectozier.

Any suggestions ?
Thanks in advance.

The issue is due to your lamda function with the tokenizer key word argument.

>>> from sklearn.feature_extraction.text import TfidfVectorizer
>>> from joblib import dump
>>> t = TfidfVectorizer()
>>> dump(t, 'tfidf.pkl')
['tfidf.pkl']

No issues. Now let's pass a lambda function to tokenizer

>>> t = TfidfVectorizer(tokenizer=lambda x: x.split())
>>> dump(t, 'tfidf.pkl')

Which throws the following error:

  _pickle.PicklingError: Can't pickle  at 0x100e18b90>: it's not found as main.

To get around this, create a function to split the text:

>>> def text_splitter(text):
...     return text.split()

Try dumping again:

>>> t = TfidfVectorizer(tokenizer=text_splitter)
>>> dump(t, 'tfidf.pkl')
['tfidf.pkl']

Now you can you save the vectorizer.


With a MLP (regression), is it appropriate to initialize bias in the final layer to be a value near the expected mean? For instance, when predicting IQ in a population you would expect the mean to be 100. If you initialize the bias in the final layer you are basically giving the network a head start, telling it in what range of values it should be guessing. Another way of getting at this would be to predict a scaled outcome value with a mean of 0, which would make the standard bias initializer value of 0 be spot-on.

Are there any guiding rules/norms when it comes to this situation? Are both methods appropriate?

Cool idea! Looks like it really helps.

Training starts with smaller error, so you might be able to train the model in shorter time with this trick. You should be okay as long as you initialise the weights properly.

I trained model with different biases and plotted error at the start and the end of of the training. Expected mean of the data is 100. See code below.

from keras.models import Sequential
from keras.layers import Dense
from keras.initializers import Constant, Zeros, Ones
from keras.metrics import mean_squared_error
import matplotlib.pyplot as plt
import numpy as np

def getData():
    n = 200
    X = np.random.randn(n, 2)
    Y = 20 * X[:,0] + 10 * X[:,1] + 100
    return X, Y

def getModel(bias):
    m = Sequential()
    m.add(Dense(1,
        input_shape=(2,),
        bias_initializer=bias))
    m.compile('adam', loss='mse')
    return m

X, Y = getData()

constants = [0, 1, 10, 50, 100, 150, 200]
loss_at_start = []
loss_at_end = []

for c in constants:
    m = getModel(Constant(c))
    m.fit(X,Y,
        epochs=20,
        validation_split=0.2,
        validation_steps=20,
        steps_per_epoch=1000)
    loss_at_start.append(m.history.history['loss'][0])
    loss_at_end.append(m.history.history['loss'][-1])

plt.plot(constants, loss_at_start)
plt.plot(constants, loss_at_end)
plt.xlabel('Bias')
plt.ylabel('Loss')
plt.legend(['Loss at training start','Loss at training end'])
plt.title('Expected mean: 100')
plt.show()


Train/Test size and bias I'm running a classifier (logistic regression). The information on my dataset are the following:
dataset size= 279 observations

(80/20 rule)
train size= 233
test size = 56

# of events in train = 31
# of events in test = 8

I think my classifier and results may be affected due to this not equal proportion. Is there any way to avoid bias issues and improve accuracy? What do you personally think of such data?

If you're referring to the fact that your dataset is small:

You should use k-fold cross validation. This will let you evaluate your model on all 279 instances

If you're referring to the class imbalance being 31:202 in train and 8:48 in test:

Use AUROC and PRC to eliminate bias in thresholding
Also see MCC


image recognition: fully connected network vs CNN To recognize handwritten digits, I have a fully connected network, containing only 2 layers: input layer (all pixels of the image) and output layer (0 or 1). I use the simplest linear regression for training and got excellent results. So I wonder if CNNs aren't necessary for this purpose if a double layer fully connect network can do the job well?!
For example, I'd like to recognize digit '1'. I use only 4 or 5 images that resemble '1', and other 4 or 5 images that look like anything else. Every time I take~2000 pixels from each image. It turns out this code with this tiny amount of training data can recognize correct and incorrect digits really well.

You might be able to get pretty good results on a simple task, but the fact of the matter is that taking random pixels (or indeed just flattening out ALL pixels) essentially destroys any structural information that was contained within the original image.

This was the insight behind convolution networks (from original author Yann LeCun), as they really find areas of correlation in the position/structure of the image input across the dataset. So they understand, for example, the a "1" offers high correlations of pixels in a vertical line, normally in the center of the image input.

This information is no longer included with randomly selected pixels and has been almost destroyed by flattening all pixels into a single vector.

If your use-case requires a certain accuracy and you are reaching that with your simple neural network (or otherwise), then of course it is perfectly valid and you can be happy  :-)


is perceptron based implementation of Logical conjunction (AND) regression or classification A perceptron model could be used to implement a Logical conjunction (AND) operator.

In this case, is the problem regression or classification?

AND operator gives you answer of True or False, it is binary in nature.

Hence, it is classification in nature.


Logic in sentence : tree representation I have sentences telling me to who a shop is opened to:

"cats, dogs or birds" (1)
"young dogs with collar" (2)
"old cats or yellow birds" (3) etc...

I would like to design an algorithm that will change this sentences to a tree representation of the logic in it:

(1) = (cat) or ((dog) or (bird))
(2) = (young) and (dog) and (collar)
(3) = ((cat) and (old)) or ((bird) and (yellow))

What do you think will work the best? LSTM maybe? How can I have this representation as a result?

Syntaxnet parser could surely help you in parsing the sentences and in tree represenation.

If you plan on solving using RNNs, I believe Tree LSTM will be a better choice than LSTM, as it also preserves dependency information.
Full paper.

Use Tree LSTM, if you need a vector embedding for the whole sentence. For use cases like classification, sentiment analysis. It works and there is a good probability that the vector could have all the information but you may not be certain and thats why its still a black box. But, if your use case is a clear representation of tree structure and logic among the terms (which you wanted), better go with parsers like Syntaxnet and try rule-based models for the use case mentioned.


Drastic increase in accuracy while using pickle file with sklearn I trained a xgboost classifier and it gave an accuracy of 49.99 % and i saved that model into a pickle file. When i ran the same data with pickle file (.pkl) it's giving an accuracy of 88.99 percent. I don't know why it's happening. Please help me out from this situation.
bank_dataset = pd.read_csv(r"dataset.csv")

missing_val = pd.DataFrame(bank_dataset.isnull().sum())

bank_dataset[' Balance'] = bank_dataset[' Balance'].fillna(bank_dataset[' Balance'].mean())

from sklearn.preprocessing import LabelEncoder

le = LabelEncoder()

objList = bank_dataset.select_dtypes(include = "object").columns

for feat in objList:
    bank_dataset[feat] = le.fit_transform(bank_dataset[feat].astype(str))

correlation = bank_dataset.corr()
print(correlation['Outcome'].sort_values(ascending = False), '\n')

k = 10
cols = correlation.nlargest(k, 'Outcome')['Outcome'].index
print(cols)
cm = np.corrcoef(bank_dataset[cols].values.T)
f, ax = plt.subplots(figsize=(14,14))
sns.heatmap(cm, vmax = .8, linewidths = 0.01, square = True, annot = True, cmap = "coolwarm", linecolor = "white",
           annot_kws = {'size':12}, xticklabels = cols.values, yticklabels = cols.values)

X = bank_dataset.iloc[:, [7,8,12,24,11,16,4,18,20]].values
y = bank_dataset.iloc[:, -4].values

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 1)

from xgboost import XGBClassifier

xg = XGBClassifier()
xg.fit(X_train, y_train)

y_pred = xg.predict(X_test)

from sklearn.metrics import confusion_matrix
from sklearn import metrics
print(metrics.accuracy_score(y_test, y_pred))
cm = confusion_matrix(y_test, y_pred)

import pickle

filename = 'xg.pkl'
pickle.dump(xg, open(filename, 'wb'))

The above code for training and saving the model into .pkl
bank_dataset = pd.read_csv(r"dataset.csv")

from sklearn.preprocessing import LabelEncoder

le = LabelEncoder()

objList = bank_dataset.select_dtypes(include = "object").columns

for feat in objList:
    bank_dataset[feat] = le.fit_transform(bank_dataset[feat].astype(str))

X = bank_dataset.iloc[:, [7,8,12,24,11,16,4,18,20]].values
y = bank_dataset.iloc[:, -4].values

file = open('xg.pkl', 'rb')

data = pickle.load(file)

X_ = data.predict(X)

The second code is running the code with pkl file with the same data used for training.

In the first code you split the data X randomly with this line:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 1)

Then after training the model you correctly apply it to the test set, which is 20% of the instances:
y_pred = xg.predict(X_test)

Whereas in the second code you apply the model to the full data X instead of only to the test set:
X_ = data.predict(X)

So in this second code you are testing on the whole dataset which includes the 80% training set. It is of course incorrect to test on the training set, this is why you obtain artificially high performance: the model "is cheating" since it has seen 80% of the instances during training.

[answer to comment]
There are at least two possibilities to split the data properly and use the saved model only on the test data:

option 1: split the data before running the training code into two different files, one for the training set and one for the test set. The training code should be provided only the training data as input, it doesn't perform any splitting and just outputs the model. The testing code takes the model and the test data as input and outputs the predictions (or performance).
option 2: apply exactly the same splitting in the testing code as in the training code. This can be done thanks to the random_state parameter in the call to train_test_split: by using the same value for this parameter the function will always separate the data exactly the same way. Then of course use only the test set for predicting and evaluating (ignore the training set part of the data).

Option 2 is much easier to implement given your current code (you just have to copy the train_test_split call). Option 1 is more general and makes the ML design cleaner, but it's less convenient.


Pros/Cons of stop word removal? What are the pros / cons of removing stop words from text in the context of a text classification problem, I'm wondering what the best approach is (i.e. to remove or not to remove)?

I've read somewhere (but can't locate the reference) that it may be detrimental the the performance of a model in the case of sentiment analysis to remove stop words.

In the context of sentiment analysis, removing stop words can be problematic if context is affected. For example suppose your stop word corpus includes ‘not’, which is a negation that can alter the valence of the passage. So you have to be cautious of exactly what is being dropped and what consequences it can have.


Adding a custom constraint to weighted least squares regression model I am trying to run a weighted least squares model that looks something like this (but could be different):

$y = \beta_0 + \beta_1 x + \beta_2 log(x) + \epsilon$

with weights $w_1, w_2, ..$

However, I know, from external knowledge, that whatever the model the outcome must asymptotically converge to a constant for large values of $x$. How can I get an OLS estimate with this constraint.

As an example, let's say if I knew the asymptote $c$, then I can add two fake data points to my model, with very high values of $x$ and very high weights $w$ and $y=c$, and run the normal WLS model and it would give me what I need - except I don't know the value of $c$. Is there a way to impose this constraint - maybe through adding a custom error term to the model?

The model you are looking for is this:

$Y=\frac{A}{1+e^{-(\beta_0+\beta_1x+\beta_2log(x))}}$, this could not be obtained but a very similar was obtained.

This code in R might work:

R=data.frame(X=c(1,2,3,4,5,6,7,8,9),Y=c(1,2,3,3,3,3,3,3,3)) # Data in which X is a line, and Y has an still unknown limit.
model=nls(formula = Y~A/(1+exp(-(b0+b1*X))),data=R)
summary(model)

In the result you can see how $A$ says that the limit of 3 (previously unknown) is calculated.

There is a limitation to take into account, is explained in this link, is summarized in the impossibility for all possible models to exist, the "most inside" model should be linear.

The model $\beta_0+\beta_1x+\beta_2log(x)$ could not be used, the model $\beta_0+\beta_1x$ could be used, take this into account.

First steps with Non-Linear Regression in R

Singular Gradient Error in nls with correct starting values


How to determine the complexity of an English sentence? I am working on an app to help people learn English as a second language. I have validated that sentences help in learning a language by providing extra context. I did that by conducting a small research in a classroom of 60 students.

I have mined over hundred thousand sentences from Wikipedia for various English words (Including Barrons'800 words and 1000 most common English words)

Entire data is available at https://buildmyvocab.in

In order to maintain the quality of content, I filtered out sentences which were longer than 160 characters since they might be difficult to understand.

As a next step, I want to be able to automate the process of sorting this content in the order of ease of understanding. I myself am a non-native English speaker. I want to know what features I can use to separate easy sentences from difficult ones.

Also, do you think this is possible?

Yes. There are various metrics, such as the fogg index.
Textacy in python has a nice list and implementations.

>>> ts.flesch_kincaid_grade_level
10.853709110179697
>>> ts.readability_stats
{'automated_readability_index': 12.801546064781363,
 'coleman_liau_index': 9.905629258346586,
 'flesch_kincaid_grade_level': 10.853709110179697,
 'flesch_readability_ease': 62.51222198133965,
 'gulpease_index': 55.10492845786963,
 'gunning_fog_index': 13.69506833036245,
 'lix': 45.76390294037353,
 'smog_index': 11.683781121521076,
 'wiener_sachtextformel': 5.401029023140788}


Split the data between the Training Data and Test Data using sklearn Work to do

My job is to take the data and divide it between Training and Test using 30% of the data as Test where both should have the same ratio between positive and negative.

CSV File

age,Feature 2,Feature 3,Feature 4,income,Feature 6,Feature 7,Feature 8,Feature 9,Feature 10,Feature 11,Feature 12,Feature 13,Feature 14,Feature 15,Class
77,1,0,0,3,0,1,1,1,0,1,1,1,0,1,0
35,1,0,1,4,0,0,0,1,1,0,1,1,0,1,0
79,1,0,0,2,0,1,1,0,0,0,1,1,1,0,1
61,0,1,0,1,0,1,1,1,0,0,0,0,1,0,0
62,0,0,0,2,0,1,0,0,0,1,1,0,0,0,1
63,0,1,1,0,1,1,0,0,1,0,0,1,0,0,1
29,1,1,0,1,1,0,1,1,0,0,1,1,1,1,0
39,0,1,1,5,1,1,1,0,1,0,1,1,1,1,0
51,0,1,1,6,1,1,0,1,0,0,1,1,0,1,1

Code - Training Data and Test Data

#!/usr/bin/env python3

import numpy as np
import pandas as pd
import matplotlib.pyplot as plot
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import Imputer

# Reading CVS file
dataSet = pd.read_csv('./TestDataFile_ML.csv')
X = dataSet.iloc[:,:-1].values
y = dataSet.iloc[:,15].values

df = pd.DataFrame(dataSet)
print(df)

# Missing data
imputer = Imputer(missing_values="NaN" , strategy="mean" , axis=0)

# Split the data between the Training Data and Test Data
xTrain , xTest , yTrain , yTest = train_test_split(X , y , test_size = 0.30 , random_state = 0)

# Creating linear regression object
linearReg = LinearRegression()
linearReg.fit(xTrain , yTrain)

#Now, testing model
yPrediction = linearReg.predict(xTest)

print(X.shape)
print(y.shape)
print(xTrain , yTrain)

#Plotting the training set
plot.scatter(xTrain , yTrain , color="red")
plot.plot(xTrain , linearReg.predict(xTrain) , color="blue")
plot.title("title")
plot.xlabel("x label")
plot.ylabel("y label")
plot.show()

# Test set
plot.scatter(xTest, yTest, color = 'red')
plot.plot(xTrain, linearReg.predict(xTrain), color = 'blue')
plot.title("title")
plot.xlabel("x label")
plot.ylabel("y label")
plot.show()

Problems found

The following error is showing me:
ValueError: x and y must be the same size

This is because apparently in the part of

X = dataSet.iloc [:,: - 1] .values
y = dataSet.iloc [:, 15] .values

I'm not taking the correct values. If someone can help me to correct the error I will thank you

Updates

After addingstratify = y to:

xTrain , xTest , yTrain , yTest = train_test_split(X , y , test_size = 0.30, random_state = 0 , stratify = y)

It shows me the following graph, in which I presume that it is incorrect

What you are looking for is called Stratified sampling

From this CrossValidated question, we have a short explanation

  Stratified sampling aims at splitting one data set so that each split
  are similar with respect to something. In a classification setting, it
  is often chosen to ensure that the train and test sets have
  approximately the same percentage of samples of each target class as
  the complete set.

To get a stratified split in ScikitLearn, you just need to edit this part of your code

# Split the data between the Training Data and Test Data
xTrain , xTest , yTrain , yTest = train_test_split(X , y ,
                                                  test_size = 0.30 ,
                                                  random_state = 0,
                                          ----->  stratify = y)

This will automatically split your dataset in train and test but also keep the same proportion of positives and negatives as the original dataset.

So, if you have a dataset where the positive class is 70% of the records and negative class 30%, after a stratified split, both train and test will have the same 70-30 distribution.


LSTM input and output for sentiment analysis I'm studying this LSTM network:
https://www.kaggle.com/paoloripamonti/twitter-sentiment-analysis

model = Sequential()
model.add(embedding_layer)
model.add(Dropout(0.5))
model.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(1, activation='sigmoid'))

model.summary()

I understand the input part of the embedding layer. Each word get's a unique vector that represents the meaning of the word.

The drop out will deactivate neurons.
So the input for the LSTM model is the vocabulary where each token is represented by a vector.

I understand the workflow in an LSTM model. But what exactly does it do with the input? Give it a score by learning?
And what is the output of the lstm?

Here is the summary:

Here it looks like the model takes in sequences of words, which are turned into embeddings, which are then put into the LSTM layer. The output of the LSTM layer is a 100 (because this is what is specified in the model) dimensional vector. This 100 dimensional vector is then put into a Dense layer outputting a 1-dimensional object, which I am assuming is some sort of prediction. Does this help? I feel like I may be missing the question.


Recording Data of my learning progress I'm just starting out with learning about Data Science and programming through various online course.

Was thinking a good first project would be a to create a log of what I'm learning and how long for and then work out ways that I can visualise this for use in a CV further down the road.

Was wanting to get a rough outline for how you would set a project like this up.

Was thinking of something like a simple site where I click on the learning resource I used and it asks for the amount of time I studied for and then records the data and time. Maybe more inputs for tags and percentage complete etc...

Currently learning R and Python.

So grateful for any ideas or advise you have.

Cheers

I think the first point to address is how employers will know you are being honest about your logging. Maybe a better use of your time is to not put in number of hours, but projects and code references you've made so that the output is more concrete rather than just a number of hours.  The problem is I know plenty of people who put in 1 hour and learn what a normal person would learn in 5 hours. Think about the WOW factor in this.. what would be convincing to employers?

Also, if its only you that are logging in hours and nobody else is going to see or use that program, you might as well use excel while focusing on whatever project you're passionate about.


What is the difference between an RMSE and RMSLE (logarithmic error)? RMSE vs RMSLE

Root Mean Squared Error (RMSE) and Root Mean Squared Logarithmic Error (RMSLE) both are the techniques to find out the difference between the values predicted by the machine learning model and the actual values.

  But, what is the purpose for RMSLE( "logarithmic")
  Does a high RMSE imply low RMSLE?

Can somebody explain in-detailed differences between RMSE and RMSLE? And how the metric works under the hood?

  When would one use RMSE over RMSLE?
  What are the advantages/disadvantages of using RMSE over RMSLE?

Let's start with the definitions:
$$ \text{RMSE} = \sqrt{\frac{1}{N} \sum_{i = 1}^{N} (y_i - \hat y_i)^2}, $$
$$ \text{RMSLE} = \sqrt{\frac{1}{N} \sum_{i = 1}^{N} (\log y_i - \log \hat y_i)^2}, $$
where $y_i$ is the target value for example $i$ and $\hat y_i$ is the model's prediction. Both metrics quantify the prediction error, so in general a high RMSE implies a high RMSLE as well.

RMSLE has the meaning of a relative error, while RMSE is an absolute error. Choosing one depends on the nature of your problem. Imagine that the target spans values from around 1 to around 100. Is predicting $\hat y = 1.01$ for true $y = 1$ as bad as $\hat y = 101$ for $y = 100$? Then RMSLE might be a good choice. With RMSE, on the other hand, predicting $\hat y = 2$ for $y = 1$ is as bad as $\hat y = 101$ for $y = 100$.

In general, you would often want to use the logarithm (or maybe the square root or some other power $0 < \alpha < 1$) for strictly positive targets that span a large range.


Do I use actual data or data difference to train machine learning model? I would like to predict tomorrows temperature :-).

But I'm unsure of the best approach. Do I simply drop data from the last x days, or do I try do drop data from the last x days in difference?

Last three days could look like: 21.3C, 22C, 21.9C

Where the difference is whatever previous, 0.7C, -0.1C

You would need more than 3 days, I'm just setting an data example. I think I also would need to insert season among other variables to complete the experiment. But I'm just asking for temperature here.

Have anyone checked this? While the temperature can be anything from -20 to 40 C with decimals (1, 0.x) makes up a lot of numbers! But with difference the range is maybe -5C to +5C, also with decimals (1, 0.x) but it beats the other range making the range smaller and therefor easier for ML to predict?

If difference is best, should I also do percentage difference since then I would take to account different climate has different swings?

I would use absolute temperatures. Knowing it's 37.2C, I'm more likely to say the temperature will decrease just from the context. This context would be lost if you only use the differences.

But like most data science questions, the proof is in the pudding. Try both and see what works best for you. You may even try using both absolute and difference as two feature columns.


Which loss function is the best loss function when using XGB regression with highly skewed dataset? Which loss function is the best loss function when using XGB regression with a highly skewed dataset?

The skewness of the data is very high. I used XGBoost with objective function of linear regression (but the data was transformed into the log space). It performed better than using gamma objective function. Any other suggestions?

You could try weighting the loss for each data point, such that some datapoints do not dominate the loss.


Data Entry Automation with ML I am working on a data entry task with approximately 6000 entries to go over.

The source comes in the form of a string and can look something like this:

  Air Canada B737 FFS

From this I can extract the following information:

  Company: Air Canada

  Model: B737

  Technology: FFS

For my initial plan of attack I iterated over the source strings using Regular Expression to extract as many keywords as possible, the problem is there are so many different Companies, Models and Technologies in my source, and not every source is as clean as the one above so it would take forever to write out all the possible regular expressions. Also, there is usually text within the source that isn't important, a red herring if you will.

As a result I have manually filled out 2000 of the entries myself.

My question is, do you think I could train a ML model with my manually filled dataset to do the rest of this task for me? I would consider myself to have intermediate skills in python, but what type of ML algo should I use for this task? Any help to point me in the right direction would be very much appreciated!

So, you can build a pretty good starter set of airlines from this wikipedia page. If you scraped the page, extracted the Airline column, and lowercased the text, you would be able to have some seed data to work with.

This is also a shot in the dark, but you could scrape the model names from this toy/model planes website, also generating some base set of strings you can use to automatically validate the first two fields in your output.

The trick to reducing your time here is to figure out the 80/20 (get 80% of the work done with 20% of the effort). I have no answers for how you would be able to differentiate the last field. But these are the steps I would take to reduce the effort altogether


Word2Vec embeddings with TF-IDF When you train the word2vec model (using for instance, gensim) you supply a list of words/sentences.  But there does not seem to be a way to specify weights for the words calculated for instance using TF-IDF.

Is the usual practice to multiply the word vector embeddings with the associated TF-IDF weight? Or can word2vec organically take advantage of these somehow?

Word2Vec algorithms (Skip Gram and CBOW) treat each word equally, because their goal to compute word embeddings. The distinction becomes important when one needs to work with sentences or document embeddings: not all words equally represent the meaning of a particular sentence. And here different weighting strategies are applied, TF-IDF is one of them, and, according to some papers, is pretty successful. From this question from StackOverflow:

  In this work, tweets were modeled using three types of text representation. The first one is a bag-of-words model weighted by tf-idf (term frequency - inverse document frequency) (Section 2.1.1). The second represents a sentence by averaging the word embeddings of all words (in the sentence) and the third represents a sentence by averaging the weighted word embeddings of all words, the weight of a word is given by tf-idf (Section 2.1.2).


Does Orange scale the data automatically for the linear regression with Ridge regularization I'm using the linear regression tool with the Ridge regularization. To use the Ridge regularization, I have to scale the data first. Does Orange scale the data automatically? I can't find any information about this mentioned in Orange's documentation for Ridge regularization.

In python's scikit-learn, I have to scale the data manually before using Ridge Regression. In MATLAB, the scaling in the Ridge function included. So, do I have to scale the data manually before I'm using the Ridge Regression in orange?

Thanks for your help.

Not by default, no, as shown by the normalise=False here:

  class Orange.regression.linear.RidgeRegressionLearner(alpha=1.0,
  fit_intercept=True, normalize=False, copy_X=True, max_iter=None,
  tol=0.001, solver='auto', preprocessors=None)


Scan-based operations Apache Spark Looking at the first paper on RDDs/Apache Spark, I found a statement saying that "RDDs degrade gracefully when there is not enough memory to store them, as long as they are only being used in scan-based operations"

What are scan-based operations in the context of RDDs and which of the Transformations in Spark are scan-based operations

Scan based operations are basically all the operations that require evaluating the predicate on an RDD.

In other terms, each time your create an RDD or a DataFrame in which you need to compute a predicate like performing a filter, map on a case class, per example, or even explain method will be considered as a scan based operation.

To be more clear, let's review the definition of a predicate.

A predicate or a functional predicate is a logical symbol that may be applied to an object term to produce another object term.

Functional predicates are also sometimes called mappings, but that term can have other meanings as well.

Example :

// scan based transformation
rdd.filter(!_.contains("#")) // here the predicate is !_.contains("#")

// another scan based transformation
rdd.filter(myfunc) // myfunc is a boolean function

// a third also trivial scan based transformation followed by a non scan based one.
rdd.map(myfunc2)
   .reduce(myfunc3)

If you want to understand how spark internals work, I suggest that you watch the presentation made by Databricks about the topics


Changes in the standard Heatmap plot - symmetric bar colors, show only diagonal values, and column names at x,y axis ticks I have a heatmap image (correlation between all matrix columns) and I'm straggling to preform all the changes below within the same image:

bar colors should be symmetric around zero (e.g., correlation of 1 and -1 should be with the same color)
change the correlation matrix to a diagonal matrix, since correlation values are symmetric - and show only upper matrix triangle (mask out the lower triangle )
show the correlation values in every cell of the diagonal matrix
x,y axis ticks - show the column names (instead of a serial number)

This is the code:

def generate_heatmap(X):

    """
    Pearson Correlation Heatmap Plot

    :return:
    """
    print("Start Pearson Correlation Heatmap Plot  .. ", datetime.now())

    plt.figure(figsize=(10,8))
    plt.title('Pearson Correlation of miRNAs', y=1.05, size=15)

    # Correlation matrix for heatmap
    corr = np.corrcoef(X.transpose())

    plt.imshow(corr, cmap='BuPu', interpolation='nearest')
    plt.colorbar()
    plt.show()

This is how I obtained the desired plot:

def generate_heatmap(X):

    """
    Pearson Correlation Heatmap Plot

    :return:
    """
    #from matplotlib import cm as CM
    from matplotlib.colors import LinearSegmentedColormap

    print("Start Pearson Correlation Heatmap Plot  .. ", datetime.now())

    # get column names
    cols = X.columns
    # define plot for heatmap
    fig, ax = plt.subplots(figsize=(16,16))

    # ------------------------------------------------------------
    # Correlation matrix for heatmap. the tranpose is because we want pxp matrix (rather a nxn)
    corr = np.corrcoef(X, rowvar=False)
    # show only upper matrix triangle - mask out the lower triangle of corr data
    corr = np.triu(corr, k=0)

    # ------------------------------------------------------------
    # Edit graphics of the plot
    plt.title('Pearson Correlation of ' + str(len(cols)) + ' miRNAs', y=1.05, size=15, fontsize=32)

    # bar colors shold be symetric around zero!
    colors = [(1, 0, 0), 'w', (1, 0, 0)]
    cm = LinearSegmentedColormap.from_list('heatmap', colors, N=20)

    # ------------------------------------------------------------
    # Heatmap based on corr matrix we provided
    c = plt.pcolor(corr, edgecolors='w', linewidths=2, cmap=cm, vmin=-1.0, vmax=1.0)

    # ------------------------------------------------------------
    # Editing additional graphics of the plot (if not too big)
    if len(cols) < 50:

        # set axis label names
        ax.set_xticks(np.arange(len(cols)))
        ax.set_xticklabels(labels = cols, rotation=45, fontsize=12, ha='center')

        ax.set_yticks(np.arange(len(cols)))
        ax.set_yticklabels(labels = cols, rotation=45, fontsize=12)

        # show corr values in every cell
        for (i, j), z in np.ndenumerate(corr):
            # in the symetric values, don't annotate the cell with the corr value
            if (i > j):
                continue
            else:
                # va and ha not working, we do +0.4 to overcome the centering of values
                ax.text(j+0.4, i+0.4, '{:0.2f}'.format(z), ha='center', va='bottom', fontsize=11)

    plt.colorbar(c)
    plt.show()

#generate_heatmap(miRNA_data[selected_mir_columns])


What is "SwarmPackagePy.cso.cso at 0x187cf21e340" def rosen(x):
"""The Rosenbrock function"""
return sum(100.0*(x[1:]-x[:-1]**2.0)**2.0 + (1-x[:-1])**2.0)

from SwarmPackagePy import cso

cso(1,rosen,10,1000,2,10000,pa=0.25,nest=100)

Out[41]: <SwarmPackagePy.cso.cso at 0x187cf21e340>

How can I access outputs of cso function? There aren't docs on this.

This is not really a "data science" question but more of a "(Python) programming" question. You can find more comprehensive answers at stackoverflow.
Other than that, if you go to the source code of SwarmPackagePy.cso (I was not able to find any documentation), you will see that it creates an instance (object) of the cso class and what you get when you print this object is the location or address of this object in the memory.


Shallow Neural Net for predicting numbers other then 1 or 0? I'm not gonna lie, I'm very new to neural networks but am also so interested in them and learning the way they work and what can be made from them. So in my endeavors for learning, I stumbled upon Siraj Ravel's youtube channel and furthermore, his Github where he posted this 1 neuron neural network that predicts the output given inputs.* I looked at it and his accompanying video and got the basic gist of how it works. After running it a few times making minor adjustments to say the inputs, I decided why not try to add numbers other than 1 and 0. I made a pattern with the output being the inputs added, however, I ran into an error where the output of its test was '1.'. I assumed this was because of the sigmoid function forcing it to be in between 1 and 0.

Summary: How can I change that program so I can give it an input of patterns with different numbers than 1 and 0 and it will output correctly (examples below)

*In his program inputs were like [1, 0, 1] = 1, [0, 0, 1] = 0 and [1, 0, 0] = 1 (pattern is first column of matrix is answer) and he asked the program to give output for [1, 1, 0] and it correctly outputted 1.

*My goal for it was so given "[2, 2, 2] = 6, [1, 2, 0] = 3, and [1, 1, 2] = 4 (more training included if necassary)" as input could it get 4 as output for "[2, 2, 0]?

Code for program

Well, the main problem is that the existing code and the desired codes solve different problems.

The existing code solves a classification problem, when you predict one of certain classes. And you want to solve a regression problem, so that neural net will predict an arbitrary number.

Sigmoid function is necessary when you predict classes, you need to remove it for regression. Or in other words you need not Logistic Regression, but Linear Regression.
Here is a great article with explanations on this: http://peterroelants.github.io/posts/neural_network_implementation_part01/

I have written an example of implementation: https://gist.github.com/Erlemar/6a5cfcca423ef3b5f6e890c6bef6d5ed


how to consider some miss classifications “half correct” using as base categorical_crossentropy - for a trading system I have a trading system where the model receives 9 time-series and predict :

A - strong down
B - week down
C - neutral
D - week up
E - strong up

(these classes are generated from an histogram to have a balanced training dataset ... the histogram is separated in 20% parts of examples centered in 0... accuracy of 20% is the baseline)

For each class I activate a different parametrized trading mechanism...

My model is giving acceptable results. Here is the resulting confusion matrix for val data (28.99% acc):

[[32 20  3  6  8]
 [35 19  9  7 16]
 [30  9  6 14 24]
 [21 14  9 18 29]
 [ 9 14  3 14 45]]

My question starts here :

I.e. If the model predicts B-"week down" but in reality is A-"strong down" is a miss, but in reality, it will make money...

So in this confusion matrix we can see that it happens 20 times (cell[0,1]) ... also if it is B but the model says A it will make money in 35 trades (cell[1,0])...

And also the same for the UP cases..

All together (from the confusion matrix) :
32+20+35+19  +  18+29+14+45 = 212 winning trades
21+14+9+14  + 6+8+7+16 = 95 losing trades

Assuming negative trades cancel in equal (in reality will not be equal..) positive trades, the total is = 117 winning trades.

What I want is to create a loss function based on categorical_crossentropy
 but somehow consider:

pred A real B - half miss
pred B real A - half miss
pred D real E - half miss
pred E real D - half miss

Do not penalize too much this cases. I think this will increase a bit the total number of positive trades. It will guide the learning a litle bit better (maybe not for accuracy but for a better loss that generates a better confusion matrix for profit )...

I have created a custom loss function that reduces 3% the loss for these cases:

def my_loss(y_pred, y_true):

    y_pre_indexes = K.argmax(y_pred, axis=1)
    y_test_indexes= K.argmax(y_true, axis=1)

    TN = K.tf.logical_or( K.tf.logical_or (K.tf.logical_and(K.equal(y_pre_indexes,0),K.equal(y_test_indexes,1)),
                          K.tf.logical_and(K.equal(y_pre_indexes,1),K.equal(y_test_indexes,0)))
                        ,
                          K.tf.logical_or (K.tf.logical_and(K.equal(y_pre_indexes,3),K.equal(y_test_indexes,4)),
                          K.tf.logical_and(K.equal(y_pre_indexes,4),K.equal(y_test_indexes,3))))

    pos_neg = K.cast(TN, K.floatx()) *(-0.03) + 1

    return K.categorical_crossentropy(y_pred, y_true)*pos_neg

(in the code the classes are :  0-A 1-B 3-D 4-E.   2-C is predicting neutral - ignore..)

but fixing to a fixed number of 3% to reduce loss for these cases seems a little bit hard coded.... Something better inside the categorical_crossentropy math philosophy should be better.

Any suggestions?

Thanks in advance!

I worked on a similar problem where classes were "ordered", so mis-classifications of class B as C was better than mis-classification as D.

In that project, "weighted quadratic kappa" loss function worked well.

Details :

https://en.wikipedia.org/wiki/Cohen%27s_kappa#Weighted_kappa

Example of Keras implementation :

https://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/quadratic_weighted_kappa.py


Data driven approach to define a churn user I'm trying to define a churn prediction model for an online service (betting/gambling). A lot of papers talk about churn analysis/prediction for telco companies where defining a churn user is straightforward: a churn user is a user who cancels his or her contract. I'm wondering how to define a churn user within the context of an industry where users don't cancel any contract/account?

I can envision a general approach: if a user who plays every day for a month stops playing for 2 days, this is could be flagged, but a user who only plays once a week and then has no activity for 2 days would not. What's the best approach to encoding this logic algorithmically? Is survival analysis and good approach?

Another approach would be to model "churn" (aka "diminished use of the service, including non-use") as a process and not an event.  Years ago in retention marketing this was called a "defection funnel", to mirror the "sales funnel" on the customer acquisition side (suspect -> prospect -> trial customer -> repeat customer -> loyal customer).  So a defection funnel might look like this:

  loyal/frequent customer -> disaffected customer -> infrequent customer -> non-customer

It is proper to label this a "funnel" because a subset of each stage move on to the next stage.  Not all "disaffected customers" become "infrequent customers", and so on.

There are two keys to defining a defection funnel.  First, define the behavioral characteristics for the funnel as a whole, and for each stage (category) within the funnel.  In your case, it might be a variation of the familiar RFM ("recency" "frequency" and "monetary") score.  Second, you need to identify other behavioral characteristics or patterns that show a propensity to move from one stage to another.  These might include: complaints, being victim of trolling, experiencing service interruptions, experiencing betting losses, how long they have been using the service, or maybe it is just demographics.

With these definitions in place, you are in a position to build a predictive model and also track over time the population of customers in each category.

By the way, this approach is not dependent on dividing the funnel into "stages", nor does it matter fundamentally how many stages you have.  However, defining discrete stages has many practical benefits, including communicating your model and results to other people who are not quantitative or statistically minded.


Time series prediction I am trying to predict a time serie from another one. My approach is based on a moving windows. I predict the output value of the serie from the following features: the previous value and the 6 past values of the source serie.

Is it usefull to add the previous value of the time serie ?

I feel like I don't use all the information contained in the curve to predict futures values. But I don't see how it would be possible to use all previous data to predict a value (first, the number of features would be growing trough time...).

What are the caveats of a 6 month time-window approach ?

Is there any paper about differents method of feature selection for time-series ?

Let me give you a few simple approaches in time series analysis.

The first approach consists in using previous values of your time series $Y_{t}$ as in $Y_{t} = \phi_{1}Y_{t-1} + ... + \phi_{n}Y_{t-n}$. In case you don't know, these models are called autoregressive (AR) models. This answers your first question. Of course it is useful to include the previous value of your time series. There is a whole set of models based on that idea.

The second approach is taking a window and extracting some features to describe the time series at each point in time. Then you use a conventional machine learning technique to predict future values as typically done. This is more common in a classification or regression setting but future values can be thought of as classifying future values. This technique has the advantage of dramatically reducing the number of features, although you usually lose characteristics associated with time. This addresses your second concern.

Another model that could be helpful in your case is the vector autoregressive model (VAR) (using Wikipedia's notation):

$$\left( \begin{array}{ccc}
y_{1,t} \\
y_{2,t}
\end{array}\right) = \left( \begin{array}{ccc}c_{1} \\ c_{2}\end{array}\right) + \left( \begin{array}{ccc}A_{1,1} & A_{1,2} \\ A_{2,1} & A_{2,2}\end{array}\right)\left( \begin{array}{ccc}
y_{1,t-1} \\
y_{2,t-1}
\end{array}\right) + \left( \begin{array}{ccc}
e_{1,t} \\
e_{2,t}
\end{array}\right)$$

Here you can see that $y_{1,t}$ has a contribution from its previous value $t_{1,t-1}$ but also includes the value of the other series $y_{2,t-1}$ in a linear combination. As usual, the purpose is to find the elements of $A_{i,j}$ that minimize some measure of error between observed values and estimated values.

A general suggestion: The first thing you need to do is to test the autocorrelation of your first series in order to confirm that an autoregressive approach is suitable and then test the cross correlation between both series to support the idea that using the second series to improve your predictions is appropriate.


Classification Model based on Ordered Features I am trying to build a classifier for a specific card dataset let's say cards or no cards. I am using Mobilenet trained on Imagenet dataset as my classifier and further training it on my dataset. I am able to train it, and it's performance is quite good on the dataset. Let's say my card has for different regions of interest as shown below:-

It is able to perfectly recognize the above passed image as card.

But I am able to fool my classifier if I change the ordering of the regions of card and then giving it to classifier as shown below:-

I want the classifier to learn the exact occurrences of features of cards as in First Image so that if I alter them it should be able to reject that card (like in second image).

Is there any way to achieve this?

You could try a recurrent neural network classifier.

Alternatively, and perhaps more easily, augment your dataset to contain images of cards with changed ordering of the card images, and train on the resulting dataset.


which machine learning technique can be used? I want to understand the intent of the customer using his search queries, let's say if a customer is interested in yoga pants, he can either search for yoga pants or exercise pants or workout tights etc. Is there a model that I can use to find out all the search keywords that can be related to yoga pants?

I think these are the methods that you can try out (Please feel free to add more to this list):

Highly precise with a little low recall is to use a dictionary with almost all possibilities (manual effort, but must be worth it.).
Using Word2Vec. Mikolov has already trained text data and created word vectors. Using this vector space, you can figure out which words are similar. You can try out and find a threshold above which you can say which words are similar (for example, yoga and exercise would have decent similarity.)
Train custom W2V, if you have enough data(This is an unsupervised model, so you don't need to worry about tagging the data but finding huge amounts of data relevant to the working domain.)
You can use an RNN to find the most similar words in a corpus and use it for queries. This gives a bit more flexibility than W2V.


What is "$D$ equal to $\frac{1}{2}$ everywhere" in original GAN paper In generative adverserial networks paper by Ian Goolodfellow, Y Bengio et al. the authors mention that "$D$ equal to $\frac{1}{2}$ everywhere" what does it means actually? here is the paper

$\mathbf{D}$ is the Discriminator network, but it really is being modelled as a probability distribution - the discriminator:

  estimates the probability that a sample came from the training data rather than G

where $\mathbf{G}$ is the Generator network.

In the context of a two-player min-max game, the authors say a unique solution exists in the case that the Generator perfectly learns the underlying distribution of the dataset. When this point is reached, the Discriminator's estimates will not be better than random guesses.

This means that the accuracy of the estimates will just be 50% (assuming a 50-50 split of real versus generated samples), so it doesn't know if the Generator created the image (which would correspond to an estimate of 100%) or definitely did not create that image (an estimate of 0%).

So $D = 0.5$ everywhere means the Generator has perfectly learned the data and can produce images that are indistinguishable from the real images... so the Discriminator is basically clueless and is flipping a coin.


How to measure uncertainty and prediction errors for a naive Bayesian Classifier? I have a small dataset of 30 rows and 5 columns (4 features and 1 class). The classifier is used to give the likelihood of occurrence of an incident. thus, the class variable gives the probability of occurrence.

How can I measure the uncertainty and prediction errors of my classifier?

I have read a post here about measuring the uncertainty of prediction. However, it is tacking the issue of multiclass classification!

You can check the probability returned by your model. Just be aware that probability is not uncertainty but if you calibrate your probabilities, you can have an idea of the belief of event realisation from a frequentist point of view.

For the error of prediction, with a calibrated classifier if you have a probability of 0.8 to belong to a certain class, you can expect that there is a 80% chance that your sample actually belongs to this class.


How could I improve my FB Prophet forecast? I've got 1325 days of revenue data and when plotting the components it makes 100% sense from a domain expert point of view, so the model is capturing the variations quite well (or it seems it does...). I've added the country holidays using m.add_country_holidays(country_name='GB')

However, when it comes to accuracy I'm getting the following averages:

MAPE: 0.3
MAE: 721,415

721,415 is not an acceptable error. Around 100K would be.

These are the MAE and MAPE plots:

Time-series plot:

Performance metrics (first 20):

What else can I do to improve the accuracy of this model? Thank you

Data here is bit noisy & has a lot of fluctuations. As few comment suggest apply some transformation on it. I would say get your data in some smaller range & then apply LSTM to predict it. I made time-series work with LSTM with removal of noise by eliminating outliers & it worked with nice further prediction.

RNNs tend to work better with time-series data especially bidirectional-LSTM due to their backwards learning capabilities.


Feature usage for machine learning algorithm Given a list of software installed by users as features, e.g.,

Microsoft_VC80_DebugCRT_x86_x64 1.0.0; Microsoft_VC80_DebugCRT_x86 1.0.0; ;Windows UPnP Browser 0.1.01;Adobe Acrobat Professional 10;

I want to predict whether the student will buy certain product.

Now the question is: what are the ways to turn the list of software into something learnable by a machine learning algorithm?

I would build a logistic regression with multiple independent variables. I don't think this is the only possibility, but logistic regression makes sense you are trying to model for the probability of purchase.

Obviously, your dependent variable will be whether a product is purchased, so it should be binary. Your independent variables will also be binary. You can also merge the binary variables into smaller number of categorical variables.

You should do do at least pairwise correlation (e.g. Phi coefficient). You can use it to merge highly correlated variables, such as, Microsoft Windows Word and Microsoft Windows Excel into Microsoft Office.


Feature selection before or after applying filter in Time-series forecasting I'm predicting ozone concentration based on meteorological variables and ozone value of the previous day. I applied savitzky golay filter to get rid of noise in the time-series dataset.

My question is, if I want to perform feature selection, do I do it before or after applying the filter? What is the logical order?
Because the feature importance is different before and after applying the filter.

Using XGBOOST, this is the feature importance before the filter:

And this is after the filter:

I'd really be grateful for any help or information.

I find your question confusing (this might be my fault). Let's see if I understand you correctly.

Feature selection is not applied before or after the filter. You choose to include a feature, or not. There are many ways to make a selection of features. If you use decision trees (or algorithms based on decision trees like XGBoost or random forest), then you can calculate the feature importance and use that to select features.

Your set-up to predict ozone concentration contains feature processing (the filter) and then an algorithm (boosted trees) that makes a prediction. You use a set of features, run them through a filter, you train the trees. Now you can use the filter + the trained trees to get your prediction (ozone concentration). You can also look at the feature importance of all the features that you used, make a selection, and retrain with a reduced set of features.

What makes no sense to me is to use a different set-up without the filter, to train the trees on the raw unfiltered features, and then look at the feature importance, because you have trained your trees on input data that the trees will never see in the set-up you're actually trying to use.

So I think the answer to your question is "after". :-)


How sensitive are feedforward Neural Network to weight changes? Let's consider the space of feedforward neural networks with a given structure: $L$ layers, $m$ neurons per layer, ReLu activation, input dimension $d$, output dimension $k$.

Which means I'm considering the map $F: \mathcal{W}_1 \times \mathcal{W}_2 \times \dots \times \mathcal{W}_L \times \mathbb{R}^d \to \mathbb{R}^k$, where $\mathcal{W}_i$ is the space of possible weights for layer $i$. We also assume, for simplicity, that every weight matrix has a norm upper bounded by a constant $M$.
Let's now assume that I have fixed parameters so that we obtain $v = F(W_1, \dots, W_L, x^*) \in \mathbb{R}^k$ (note that $x^*$ is fixed as well).

Now imagine that I inject some random noise $\eta \in \mathbb{R}^{m \times m} $ in a weight matrix $W_i$, where the norm of the noise is 10% of the norm of the matrix, e.g. $||\eta|| = ||W_i||/10$  . How does it affect my final output?

which means, what's the expected value of $||v - v_*||$, where $v_*$ is the output of the network obtained after the small changes in the weights described before?

Note that this has nothing to do with the learning process, it's just about the sensitivity/resistance of a neural network with respect to random noise injected in a weight matrix.

you are asking for the condition I would say.
If you skim through the formulae, the german explanation is more
detailed.
In particular, the absolute condition at $x$ is defined as  $\kappa_{\text{abs}}:= \lim \sup _{\tilde{x} \rightarrow x} \frac{||f(x) - f(\tilde{x})||}{||x - \tilde{x}||}$, which means $\kappa_{\text{abs}} \geq 0 $ is the smallest number such that
there is a $\delta >0 $ so that all $\tilde{x}$ with $||\tilde{x}-x|| < \delta $ it holds that $|| f(\tilde{x})-f(x) || \leq \kappa_\text{abs} || \tilde{x} - x ||.$
The relative condition $\kappa_{\text{rel}} \geq 0 $ at $x$ is the smallest number such that there is a $\delta > 0 $ so that all $\tilde{x}$ with $||\tilde{x}-x|| < \delta $ satisfy:
$ \frac{|| f(\tilde x)-f(x) ||}{||f(x)||} \leq \kappa_\text{rel} \frac{|| \tilde{x} - x ||}{||x||}.$
The difference is thus, that the $\kappa_{\text{rel}}$ compares the relative change of the output with the relative change of the input.
In general, let $\tilde{f}$ denote an approximation of the real function $f$, $\tilde{x}$ the input with some noise, then there are 4 categories in numerics:

condition: Analyze $||f(x)-f(\tilde{x})||$
stability: Analyze $||\tilde{f}(x)-\tilde{f}(\tilde{x})||$
consistence: Analyze $||\tilde{f}(x)-f(x)||$
convergence: Analyze $||\tilde{f}(\tilde{x})-f(x)||$.

Your question is also related to adverserial attacks. You can have a look into the literature.
Note also that $\kappa_{\text{rel}} = \frac{||Df(x)||||x||}{||f(x)||}$, so that you could compute the condition number for any given neural network.


Discovering cross category sales using transactions history (Clustering?) I have a set of sales transaction data containing more than 1 item purchased. Every item sold has a category. I would like figure out which categories are most commonly ordered together.

The data is more or less like this:

Transaction ID|Item ID | Sales Quantity|Item Category
1              Apple     1              Fruit
1              Banana    1              Fruit
1              Carrot    2              Vegetable
2              Carrot    1              Vegetable
2              Ice Cream 2              Dessert
3              Squash    2              Vegetable
3              Chocolate 2              Dessert
4              Apple     1              Fruit
4              Carrot    1              Vegetable
4              Doughnut  1              Dessert

Just eyeballing above you can see that there are a high amount of vegetable-dessert pairings on the same transaction. But now imagine that we have 250,000+ transactions in the data set and dozens of categories. I'm looking to discover cross category sales only. Not interested in Apples and Bananas (Fruit-Fruit) pairs.

I think I can teach myself how to code the analysis, but I'm just not sure what this is called or what to Google. Any thoughts?

Note that your data can be re-ordered to look like this:

Transaction ID | Items
1                {Apple, Banana, Carrot}
2                {Carrot, Ice Cream}

This kind of data set is trivial for association rule mining. A very simple and well-known algorithm of this kind is the Apriori. I'm certain there are packages for executing this algorithm in R.

For the restriction of "discover cross category sales only", you can just post-prune the generated rules, ie. let the algorithm generated inter-category sales and then remove those later, which should be trivial.


How plot GridSearch results? I trained an SVM model with GridSearch
svc = SVC()
parameters = {
  'kernel': ['linear', 'rbf'],
  'C': [0.1, 1, 10]
}

cv = GridSearchCV(svc, parameters, cv=5)
cv.fit(v_train, y_train)

print_results(cv)

Here is the result I got:
BEST PARAMS: {'C': 1, 'kernel': 'linear'}

0.912 (+/-0.037) for {'C': 0.1, 'kernel': 'linear'}
0.763 (+/-0.027) for {'C': 0.1, 'kernel': 'rbf'}
0.942 (+/-0.045) for {'C': 1, 'kernel': 'linear'}
0.903 (+/-0.044) for {'C': 1, 'kernel': 'rbf'}
0.94 (+/-0.043) for {'C': 10, 'kernel': 'linear'}
0.928 (+/-0.046) for {'C': 10, 'kernel': 'rbf'}

What is the best way to plot this result? in one plot that both contain the C parameter and kernel with their corresponding accuracy.
Thank you.

What I would suggest is to have put the results as a Data frame with
pd.DataFrame(cv.cv_results_)

And then you have the data in a dataframe which is easier to handle.
For the other question (in your comments), once you have the data in a nice dataframe is just about data visualization. How can you put it in a nice visualization?
The easier are either 2d plots or 3d or even contour plots. By googling them and python you will get nice tutorials on how to do it


Technology stack for linear regression on (not so) large dataset While attending to the Coursera's Machine Learning Course, I figured out that I could use a database from the company I work for (~50MM records) to do some linear regression experiments.

But one of the steps involved on proposing this experiment, is to define the technology stack required for this task.

From my understanding the following tasks should be covered:

Read raw data and store it on an non-production database
Transform data to a "regression friendly" format
Store the transformed data on an intermediate database
Compute the actual regression

For #1 I can take some paths, like doing a custom .NET or Java program, or even use an ETL process (this is more to copy data to somewhere else and don't mess with production database).

On #2 the funny part begins: should I consider a specialized tool for a <100MM records database? If so, what would you suggest for transforming this data into a matrix-like representation?

I believe #3 is dependant on the #4: I see lots of samples (eg.: in R, or Matlab/Octave) based on text or csv files. Are these the standard formats for these computations? Or should I read from a database

For #4, from what I could understanding using R is the way to go, right?

Finally, should I consider a multi-gig multi-processors server, or considering it's an experiment in which spending some hours of computation is not a big issue, a 4GB machine will do the job?

I am aware that this question may be considered too broad, but I really would like to hear from you about what should I consider for it, and even if I am missing something (or going to a totally wrong path).

Regarding the data, you can consider it like the house pricing in Boston: it's a 30 features (columns) dataset, used to predict the value for one of these columns.

(question originally posted on Stack Overflow)

Since no one has mentioned RAM-efficient methods yet:

Instead of loading everything into RAM, you can use online/out-of-core learning.

Python's Scikit-Learn for example has the SGDClassifier class. Set its loss function to "log" and you get logistic regression. Using the partial_fit function you can feed it small batches of data that you read straight from the database (or from some CSV file,...).

Vowpal Wabbit might also be worth a try. Its made for out-of-core learning - hardly uses any RAM and you won't find anything that's much faster.

You could also use Python's Keras library to build a neural network (or in the simplest case just logistic regression), which you can also feed with small batches of data instead of loading everything into RAM. Compared to the other two recommendations a neural network could also learn non-linear dependencies.

Besides that, try to start with fewer samples - plot the learning curves with 10k, 100k, 1M samples and see if 100M samples are even necessary to get a good score.


how to access weights of individual Neurons in the output layers in MLPs? im working on a neural network using Keras. Its an mlp(multi-layer perceptron). With  8 Neurons in the output layer. Is there a way I can access weights and biases of individual neurons of the output layer for every iteration?

I'm guessing you want something like this:

model.layers[-1].get_weights()


Information about LTSM RNN backpropagation algorithm I am attempting to make a LTSM RNN in python from scratch and I have completed the code for forward pass but I am struggling to find a clear outline of the equations I need to calculate to get the gradients using back-propagation. Is there any straightforward resource that I can learn these equations from and how to implement them(as my programming skills are limited)?
Thanks for any help,

The notation for my code is that; hu is the update gate, hf is the forget gate and ho is the output gate.

def forward(inp,target):
    loss = 0
    c_temp,c,x,a,y,prob = {},{},{},{},{},{}
    c_old = {}
    c[-1] = np.zeros((hidden_size,1))
    a[-1] = np.zeros((hidden_size,1))
    for t in range(len(inp)):
        x[t] = np.zeros((vocab_size,1))
        x[t][inp[t]] = 1
        X = np.concatenate((x[t],a[t-1]))
        c_temp[t] = tanh(wc @ X  + bc)
        hf[t] = sigmoid( wf @ X + bf)
        hu[t] = sigmoid( wu @ X + bu)
        ho[t] = sigmoid(wo @ X + bo)

        c[t] = hu*c_temp[t] + hf * c[t-1]
        a[t] = ho * tanh(c[t])

        y[t] = wy @ a[t]  + by
        prob[t] = softmax(y[t])
        loss += loss(prob[t],target[t])

Here is a good tutorial on LSTM from scratch with forward and backward pass.


GAN vs DCGAN difference I am trying to understand the key difference between GAN and DCGAN.

I know that DCGAN uses a convolutional network. But:

What data is better to push into GAN and what data fits better to DCGAN?
Does DCGAN work better with small data dimensions?

A Generative Adversarial Network (GAN) takes the idea of using a generator model to generate fake examples and discrimator model that tries to decide if the image it receives is a fake (i.e. from the generator) or a real sample. This was originally shown with relatively simple fully connected networks.

A Deep Convolution GAN (DCGAN) does something very similar, but specifically focusses on using Deep Convolutional networks in place of those fully-connected networks. Conv nets in general find areas of correlation within an image, that is, they look for spatial correlations. This means a DCGAN would likely be more fitting for image/video data, whereas the general idea of a GAN can be applied to wider domains, as the model specifics are left open to be addressed by individual model architectures.

The linked paper that proposed DCGANs specifically raises the topic of unsupervised-learning, and essentially wanted to marry the (at the time) recent success of conv nets with the new idea of GANs.

I also couldn't find any direct comparisons of when to use them, but there are plenty of articles that explain both models. This is a good place to start - after reading that, you could probably decide for yourself.

Regarding dimensions - I don't think the dimensions of your data would dictate which of the two variants to go for, other than of course influencing things that we always have to consider, such as training time, model complexity, capacity to learn and so on.


AttributeError: 'numpy.ndarray' object has no attribute 'predict' I have trained and saved a model :

import numpy as np

# load the dataset
dataset = np.loadtxt("modiftrain.csv", delimiter=";")

# split into input (X) and output (Y) variables
X_train = dataset[:,0:5]
Y_train = dataset[:,5]

from sklearn.naive_bayes import GaussianNB

# create Gaussian Naive Bayes model object and train it with the data
nb_model = GaussianNB()

nb_model.fit(X_train, Y_train.ravel())

# predict values using the training data
nb_predict_train = nb_model.predict(X_train)

# import the performance metrics library
from sklearn import metrics

# Accuracy
print("Accuracy: {0:.4f}".format(metrics.accuracy_score(Y_train, nb_predict_train)))
print()

# import the lib to load / Save the model
from sklearn.externals import joblib

# Save the model
joblib.dump(nb_predict_train, "trained-model.pkl")

Then, i'm loading the model and try to make predictions on a new dataset :

# import the lib to load / Save the model
from sklearn.externals import joblib

import numpy as np

# Load the model
nb_predict_train = joblib.load("trained-model.pkl")

# load the test dataset
df_predict = np.loadtxt("modiftest.csv", delimiter=";")

X_train = df_predict

nb_predict_train.predict(X_train)

print(X_train)

Here comes the error :

  File "predict01.py", line 14, in <module>
    nb_predict_train.predict(X_train)
AttributeError: 'numpy.ndarray' object has no attribute 'predict'

You don't want to pickle the predictions but rather the fit.

Change joblib.dump(nb_predict_train, "trained-model.pkl") to joblib.dump(nb_model, "trained-model.pkl")


Differerent ways to detect the appropriate number of topics I implemet the LDA topic modeling in R.

One essential parameter is the selection of the number of topics

Which of the following ways could it be the most suitable:

1. mallet
2. stm
3. or this way https://cran.r-project.org/web/packages/ldatuning/vignettes/topics.html

There's no "most suitable" way, but there might be one which works better with your data. The only way to know that would be to try all of them.

In case choosing the number of topics is an issue, you might be interested in using the non-parametric extension of LDA for topic modeling, which doesn't require you to specify the number of topics: this is called Hierarchical Dirichlet Processes, see for instance this introduction.


Ball Tree and Pseudometrics The docs for sklearn.neighbors.DistanceMetric state that

  in order to be used within the BallTree, the distance must be a true metric

(i.e. be non-negative, 0 only if objects are equal, symmetric, and satisfy triangle inequality). My question is, how strict are those requirements? More specifically, would ball tree work for pseudometric (may be 0 for non-equal objects but satisfies other conditions)?

I think a pseudometric should be fine.

The BallTree's ability to limit the number of distances to compute relies heavily on the triangle inequality, and (perhaps to a lesser extent) symmetry.  And nonnegativity certainly seems critical.  At the very least, the code in sklearn will probably in places tacitly assume symmetry and nonnegativity.

In a pseudometric, the relation "distance 0" is an equivalence relation, and the distance naturally extends to the quotient space, where it is an honest metric.  So I think for BallTrees, it really shouldn't matter; any ball that contains a given point will also automatically contain all its "equivalent" points.


Scala vs Java if you're NOT going to use Spark? I'm facing some indecision when choosing how to allocate my scarce learning time for the next few months between Scala and Java.

I would like help objectively understanding the practical tradeoffs.

The reason I am interested in Java is that I think some of my production, frequently refreshed, forecasts and analyses at work would run much faster in Java (compared to R or Python) and by becoming more proficient in Java I would enable myself to work on interesting side projects, such as non-Data-Science apps I'd like to develop. Currently I've taken a couple of Java courses, but I need much more education and practice to master it.

The reason I started considering learning Scala is very similar -- I figured that the statistical/ML orientation would make it good for my work as a Data Scientist, and since it is based on Java I would be getting practice at work that helps me with my side-interest in Java, even though there are a few major differences such as functional vs. imperative and traits vs. interfaces.

It seems like a lot of the advantages of Scala revolve around integration with Spark. I am thinking that this should be the tipping point for my decision, because my team is not currently using Spark and I don't have a good enough reason to request it. However, I thought I should ask here so that I don't waste too much time if Scala is still a better choice.

For the purposes of this question please ignore alternatives such as Python, R, Julia, etc (I've eliminated those from consideration for other reasons, such as already being sufficiently familiar with them for my use cases).

This is a bit off topic for this SE, or maybe opinion-based, but, I work in this field and I'd recommend Scala.

No I would not characterize Scala as a "stats-oriented" Java. I'd describe it as what you get if you asked 3 people to design "Java 11" and then used all of their ideas at once.

Java 8 remains great, but Scala fully embraces just about all the good ideas from languages that you'd want, like type safety and closures and functional paradigms, and a more elaborate types/generics system, down to more syntactic sugar conveniences like case classes and lazy vals. Plus you get to run everything in the same JVM and interoperate with any Java libraries.

The price is complexity: understanding all of Scala is a lot more difficult than all of Java. And some bits of Scala maybe were a bridge too far. And, the tooling ecosystem isn't great IMHO in comparison to the standard Java tools. But there again you can for example use Maven instead of SBT. But you can mostly avoid the parts of Scala that are complex if you don't need them. Designing Scala libraries takes a lot of skill and know-how; just developing run-of-the-mill code in Scala does not.

From a productivity perspective, once you're used to Scala, you'll be more productive. If you're an experienced Java dev, I actually think you'll appreciate Scala. I have not liked any other JVM language, which tend to be sort of one-issue languages that change lots of stuff for marginal gains.

Certainly, for analytics, the existence of Spark argues for Scala. You'll use it eventually.


How to scale a variable when not knowing the maximum I have a dataset with different features where some of them are not categorical, so they need to be scaled or normalized (especially the target).

However, normalizing between 0-1 for instance means that the variable maximum value will be equal to one, and the mean to 0.

Now if I receive a new example never seen before, and this example has a value higher than the max of the training examples, how should this value be normalized ?

EDIT

As an example. If my maximum value is 150, it will be scaled to 1.0. Now if I receive a new example, with a value equal to 320, how should it be scaled ?

If your model works in production, you should not retransform your scaler, you should transform new example like 150 is still the maximum value. (It will give you higher than 1, so its a bit problematic but possible solution is below)
However you can still label those example as outlier.

Possible solution for that case: If you have high number of outliers/leverages, you should consider tree ensembles and/or regularized models.

If your predictor is not on production, just add those examples to your train set and fit again since your sample in the first training would be different than the reality.


Row-wise Jacobian with pytorch Suppose I have $f:\mathbb{R}^{d_i}\to\mathbb{R}^{d_o}$.
Let $X \in \mathbb{R}^{n \times d_i}$ and I apply $f$ to each row of $X$, obtaining $Y = f(X) \in \mathbb{R}^{n \times d_o}$.
I would like to compute a tensor $Z$ which is defined as

$$Z_{i,j,k} = \frac{\partial Y_{i,j}}{\partial X_{i,k}}$$

using pytorch. How should I define v to achieve that?

def get_row_wise_jacobian(X, f):
     n = X.shape[0]
     d_i = X.shape[1]

     x = X.clone().detach().requires_grad_(True)

     y = f(x)
     d_o = y.shape[1]

     v = ?
     y.backward(v)

     Z = x.grad
     return Z

If f(x) is a deep network, you could use the differential network implementation of this repository. This differential network adds a few computations to the feed-forward graph to compute the analytical Jacobian w.r.t. to the network input. In contrast to the autograd implementation, the computational overhead is minimal and the Jacobian is computed using forward differentiation. Furthermore, the examples of the Repository show that you want to use softplus activation to get smoother Jacobians.