Number of parameters in Resnet-50 I'm using Keras, and I am struggling to know how many parameters Resnet-50 has. Keras documentation says around 25M, while if I use model.param_count() when loading a Resnet-50 model, it says 234M.
Which one is correct? I'm confused.

model.summary prints this:

Total params: 234,355,586
Trainable params: 210,767,874
Non-trainable params: 23,587,712

from keras.applications.resnet50 import ResNet50

resnet_model = ResNet50(weights='imagenet')

#resnet_model.count_params()
resnet_model.summary()

  Total params: 25,636,712
  Trainable params: 25,583,592
  Non-trainable params: 53,120

Check your code once to be sure that it is ResNet50


When should I use StandardScaler and when MinMaxScaler? I have a feature vector with One-Hot-Encoded features and with continous features.

How can I decide now, which data I shall scale with StandardScaler and which data scale with MinMaxScaler? I think I do not have to scale the one-hot-encoded anyway because they are already between 0 and 1.

(I use afterwards a MLPClassifier)

StandardScaler and MinMaxScaler are more common when dealing with
continuous numerical data.

One possible preprocessing approach for OneHotEncoding scaling is "soft-binarizing" the dummy variables by converting softb(0) = 0.1, softb(1) = 0.9. From my experience with feedforward Neural Networks this was found to be quite useful, so I expect it to be also benefitial for your MLPClassifier.
StandardScaler is useful for the features that follow a Normal distribution. This is clearly illustrated in the image below (source).

MinMaxScaler may be used when the upper and lower boundaries are well known from domain knowledge (e.g. pixel intensities that go from 0 to 255 in the RGB color range).


What's the best way to tune the regularization parameter in neural nets I'm tuning the regularization parameter of a neural net (L2 regularization) using a grid. Starting with values 0.0005, 0.005, 0.05, 0.5, 5. Then if 0.005 brigs the best validation result, I continue using a grid like this: 0.0015, 0.0025, 0.0035, 0.0045, 0.0055, 0.0065, 0.0075, 0.0085, 0.0095. And it continues..

I would like to know if there's a more intelligent way to tune this parameter.

Wikipedia lists some well-known approaches to hyper-parameter searches.

The brute-force scan/search, or a grid search across multiple parameters, is still a very common and workable approach. As is random search, just trying some variations of parameters automatically and picking the best result from cross validation.

A guided search by intuitive feel of what might best work is also still quite common approach, despite being characterised as "optimisation by graduate descent" (a pun based on the fact that a senior researcher will hand off the tuning work to their smart junior researchers who make educated guesses on hyperparameter values) - for regularisation parameters you can look at training curves and difference between training loss vs cv loss to help decide where to look. You generally want to increase regularisation if there is a large difference and decrease it if the values become close but not very good. What counts and as "large difference" and "not very good" is subjective though, and typically you only get a feel for what works after a few random attempts and immersing yourself in the problem so that you recognise good vs bad behaviour of a model.

There are some claimed "smart" schemes that apply Bayesian optimisation or similar, and even some automated services or paid tools that attempt to make these available for tuning models automatically. However, I don't see these used much in practice.

Unless you are trying to win a Kaggle competition, you quickly hit diminishing returns attempting to fine-tune your parameter. I've rarely found tuning the second significant digit of a meta-param like L2 loss to be worth much attention. Maybe with the exception of testing between e.g. 1.0 and 1.5 - often useful to think in terms of geometric progression, not linear, even when fine-tuning.

You also have to be careful that you are not simply tuning to your cross-validation set, with no meaningful or justifiable improvement for test cases or in production. You can mitigate this in part by using k-fold cross validation (which can reduce error on metrics estimates effectively by using more data to estimate them), although of course that increases time/CPU costs.


Getting the leave-one-out error on least square regression to fit polynomials I need to implement least square regression to fit polynomials of degree 1-27. I then need to get the leave-one-out error (kfold cross validation where k = n). After doing a lot of research it seems the best way to get the LOO error is to use sklearn cross_val_score(). My problem is I do not know how or if it is possible to use with regression models.

n = len(x)
p, res, _, _, _  = numpy.polyfit(x,y,1,full=True)

cv = cross_val_score(?, X, y, scoring=mse, cv=n)

I cannot figure out what the estimator would be or how to make it in cross_val_score. Being new to python and these topics make this twice as challenging.

numpy and scikit-learn are quite separate open-source projects. As such, they are compatible in some ways but not in others. All scikit-learn models are variants on the fundamental estimator object (see https://scikit-learn.org/stable/tutorial/statistical_inference/settings.html). The cross_val_score function expects a scikit-learn-style estimator object and there's no easy way to create such an object from the output of numpy.polyfit.

I'd suggest working entirely in scikit-learn, creating a regression estimator that you can cross-validate with cross_val_score. This SO post shows you how to build that polynomial regression estimator.


Create price matrix from tidy data without for loop I have a dataframe in the following format

     symbol     name        date   close      market
0    BTC  Bitcoin  2013-04-28  134.21  1500520000
1    BTC  Bitcoin  2013-04-29  144.54  1491160000
2    BTC  Bitcoin  2013-04-30  139.00  1597780000
3    BTC  Bitcoin  2013-05-01  116.99  1542820000
4    BTC  Bitcoin  2013-05-02  105.21  1292190000
...
          symbol    name        date     close  market
752109    W3C  W3Coin  2018-04-07  0.000277       0
752110    W3C  W3Coin  2018-04-08  0.000286       0
752111    W3C  W3Coin  2018-04-09  0.000270       0
752112    W3C  W3Coin  2018-04-10  0.000286       0
752113    W3C  W3Coin  2018-04-11  0.000300       0

I want to create a matrix with prices and a matrix with marketcaps. Because I'm interested in re-creating the S&P500 index. I have done this with a for loop below. Question: How do I do this without a for loop?

edit: Note that some symbols have different number of dates, for example BTC might have 1000 days but W3Coin only 50 days. I think that is what the error message is telling me.

import data:

df = pd.read_csv('data-coins-scrape/CryptoData.csv', parse_dates=True)
cols = ['symbol', 'name', 'date', 'close', 'market']
df = df[cols]
symbols = df.symbol.unique()
nr_coins = df.symbol.nunique()

for loop to get prices and makretcaps

prices = []
marketscaps = []
shapes = []
for s in symbols:
    # slice out one coin
    dfc = df.loc[df.symbol == s]
    shapes.append(dfc.shape)
    prices.append(dfc['close'])
    marketcaps.append(df['market'])
del dfc

check the output

len(prices), len(marketcaps)
# (1531, 3065) is the output

type(prices[0]), prices[0].shape, prices[0].name
# output: (pandas.core.series.Series, (1810,), 'close')

df.loc[df.symbol == 'BTC', 'close'].head() == prices[0].head()
# returns True

You can do the following:

df1 = df.set_index(['name', 'date'])
prices = df1['close'].unstack()
marketcaps = df1['marketcaps'].unstack()


Percentile as a threshold for Anomaly Detection? I'm following this article about Unsupervised Anomaly Detection Algorithms. In this article, a threshold value is calculated using the scipy score percentile method to determine whether the point is an outlier or not. What is the connection between percentile value and threshold and how can we decide if a point is an outlier or not using a threshold value?

You have a distribution of data (multidimensional or not it does not matter). And anomalies will be scattered towards corners of this dataset. Hence low percentile will indicate that certain points are anomalies in the data---cornered in. Threshold is, in this context, just a cutt-off value when you claim something is anomaly.


Display images after augmentation in Keras How can I display all images after augmentation?

How can I get the number of the trained data after augmentation?

Thank you

Depending on the kind of data set you are using you can use .flow (if you have data as numpy arrays) or .flow_from_directory (if you have images in file system) to run through the data generator and save the output using save_to_dir argument.


Does anyone could help me to understand what is the autoencoders? Does anyone could help me to understand what the autoencoders means?

What we expect is that the outputs are equal to the inputs, then why we need to do that? It doesn't make any sense to me.

I find some interpretation that it learned how to reconstruct the input data, does that mean that we could just pick some of the pixels from the origin picture and then reconstruct the whole original picture? If so, it still makes no sense to me, because the reconstructing part of the model is from the hidden layer to the output layer, we cannot just put the selected data into the hidden layer, cuz the input data of the hidden layer are combinations of the whole raw data from the input layer.

Thanks in advance.

Autoencoders are a neural network solution to the problem of dimensionality reduction.

The point of dimensionality reduction is to find a lower-dimensional representation of your data. For example, if your data includes people's height, weight, trouser leg measurement and shoe size, we'd expect there to be some underlying size dimension which would capture much of the variance of these variables. If you're familiar with Principal Component Analysis (PCA), this is another example of a dimensionality reduction technique.

Autoencoders attempt to capture a lower-dimensional representation of their data by having a hidden "bottleneck" layer which is much smaller than the dimensionality of the data. The idea is to train a neural network which throws away as much of its dimensionality as possible and can still reconstruct the original data.

Once you have an autoencoder which performs well, by observing the activations at the bottleneck layer, it's possible to see how an individual example scores in each of the reduced dimensions. This may allow us to begin to make sense of what each of the dimensions represents. One can then use these activations to score new examples on this set of reduced dimensions.


Semantic Annotation in text with curlie.org I came across curlie.org (previously known as the dmoz taxonomy) and I'm interested to see how I could best start tagging a given text, with concepts from that taxonomy:

Are there any tools out there that do semantic annotation based on a taxonomy (I couldn't find any)
How would one go about making such a semantic annotation process

I know this question might be a large to answer in a short reply, but any pointers are greatly appreciated.

Thanks in advance!

So, I'm still not sure about what you consider to be an annotation on curlie.org. What I understood, you'll correct me if I'm wrong, is that you would like to annotate some text, more specifically you would like to annotate/identify concepts, and you would like to follow the structure on curlie.

From what I see, curlie doesn't contain any annotation, it's more like a website ontology, which has at the bottom real url and as hypernyms domain-specific topics and then at the top level broad topics. So the only thing you might consider an annotation, at document level, i.e. a single label for a whole website, is the domain-specific topic.

If you want to create an ontology similar to curlie, Protege could be a good choice. Also, if you're interested in retrieving some general taxonomy information for a word, you probably want to look at some lexical databased like WordNet or ConceptNet.

If this is not what you want and you're interested in making your own annotations at word level, e.g. "I bought an iPhone[object]" then probably the taxonomy of curlie is too generic and you will definitely need to narrow down the scope of your annotations. Take a look at some literature of NLP tasks like Named entity recognition, it's plenty of papers. Also, if you will go down this road (creating your own annotations) a good tool is Prodigy. Consider that even at word level, annotations can defined at several levels for each document. For example, in opinion mining usually people are interested in annotating not only a chunk of words representing an opinion, but also other aspects like the target of the opinion, e.g. "[This [GPS]$_{target}$ really sucks]$_{opinion}$, it never works!".
But in my experience people stop at max 3 layers of annotations, and it truly depends on the purpose for which the annotations are made in the first place.


Data duplication optimization I am working in python3 cleaning data. I have a large number of midi files scraped from a variety of sources using beautiful soup.  Many of the files may be duplicate musical pieces. I can change the the keys of the midis so that they are the same and change the instrument to piano (they are monophonic files).

So, then it should be possible to check if the song content (in terms of the midi encoding) is similar. It is particularly pressing as some of the files only have numbers as names on them. So, to be clear, it is the content of the files that I have to check for duplication. Also, I am not looking for exact matches I am just looking for percent similarity.

My current approach is to use SequenceMatcher from Difflib. I am checking if a buffer of 600  on   each seperate file has a SequenceMatcher ratio >.9 Then, I am flagging them so I can compare the duplicates by listening to them.

This approach works in test scenarios with songs in midi format. However, it is extremely slow. So, I am wondering if there is anyone who has faced this issue before. Can anyone offer any insight on optimizing this algorithm for comparing the files or paralleling the code to gain speed or any other approach aside from hashing (I am not looking for exact file matches). Any help would be appreciated.

Here is my code:

import sys
import os
import hashlib
from difflib import SequenceMatcher

def similar(a, b):
    return SequenceMatcher(None, a, b).ratio()

count=0
for dirName, subdirs, fileList in os.walk('path to my directory of files'):
    print('Scanning %s...' % dirName)
    for filename in fileList:
        path = os.path.join(dirName, filename)
        in_file = open(path, 'rb')  # Provide a path to disk or ISO image
        data = in_file.read(600)
        in_file.close()
        for filename2 in fileList:
            path = os.path.join(dirName, filename2)
            in_file2 = open(path, 'rb')
            data2 = in_file2.read(600)
            in_file2.close()
            if filename==filename2:
                pass
            else:
                s=similar(data,data2)
                if s>0.9:
                    print(filename +filename2+" "+str(s))

`

If you are looking for only for identical copies for files, then you can calculate a hash like md5sum on each file and compare that, which is much faster than analysing the content.

If you’re looking for a similarity metric then I think you should be able to use a faster algorithm than the one that SequenceMatcher uses, which finds the longest exactly matching sequence, which takes quadratic time on average. I suggest you look at other similarity metrics, such as Levenshtein or Hamming distance, which will be faster to compute.


Combining data sets without using row.name I start with a data.frame (or a data_frame) containing my dependent Y variable for analysis, my independent X variables, and some "Z" variables -- extra columns that I don't need for my modeling exercise.

What I would like to do is:

Create an analysis data set without the Z variables;
Break this data set into random training and test sets;
Find my best model;
Predict on both the training and test sets using this model;
Recombine the training and test sets by rows; and finally
Recombine these data with the Z variables, by column.

It's the last step, of course, that presents the problem -- how do I make sure that the rows in the recombined training and test sets match the rows in the original data set? We might try to use the row.names variable from the original set, but I agree with Hadley that this is an error-prone kludge (my words, not his) -- why have a special column that's treated differently from all other data columns?

One alternative is to create an ID column that uniquely identifies each row, and then keep this column around when dividing into the train and test sets (but excluding it from all modeling formulas, of course). This seems clumsy as well, and would make all my formulas harder to read.

This must be a solved problem -- could people tell me how they deal with this? Especially using the plyr/dplyr/tidyr package framework?


Why does the logistic regression cost function need to be the negative of log? I am going through this course. The professor is talking about the logistic regression cost function:

$$
P(y|x) = \hat y^y (1- \hat y)^{(1-y)}
$$

Taking log on both sides provides:

$$
\begin{align}
log[P(y|x)] &= log[\hat y^y (1- \hat y)^{(1-y)}]\\
&= - L(\hat y,y)
\end{align}
$$

Why does the logistic regression cost function need to be the negative of log?

Normally we want to maximize the likelihood (consequently the log likelihood). This is the reason why we call this method maximum likelihood estimation. We want to determine the parameters in such a way that the likelihood (or log likelihood) is maximized.

When we think about the loss function we want to have something that is bounded by 0 from below and is unbounded for positive values. Our goal is to minimize the cost function. Hence, we take the negative of the log likelihood and use it as our cost function.

It is important to note that this is just a convention. You could also take the log likelihood and maximize it, but in this case, we would not be able to interpret it as a cost function.


How to fix "Expected sequence or array-like" I am trying to get the accuracy of the model and I am getting this error

  TypeError: Expected sequence or array-like, got

Here's my code sample.

%matplotlib inline
import matplotlib.pyplot as plt
import pandas as pd

dataframe = pd.read_csv("WA_Fn-UseC_-HR-Employee-Attrition.csv")

dataframe.head(2)

# spare input and target variables
inputs = dataframe.drop('PerformanceRating', axis='columns')

target = dataframe['PerformanceRating']

from sklearn.preprocessing import LabelEncoder

MaritalStatus_ = LabelEncoder()
JobRole_ = LabelEncoder()
Gender_ = LabelEncoder()
EducationField_ = LabelEncoder()
Department_ = LabelEncoder()
BusinessTravel_ = LabelEncoder()
Attrition_ = LabelEncoder()
OverTime_ = LabelEncoder()
Over18_ = LabelEncoder()

inputs['MaritalStatus_'] = MaritalStatus_.fit_transform(inputs['MaritalStatus'])
inputs['JobRole_'] = JobRole_.fit_transform(inputs['JobRole'])
inputs['Gender_'] = Gender_.fit_transform(inputs['Gender'])
inputs['EducationField_'] = EducationField_.fit_transform(inputs['EducationField'])
inputs['Department_'] = Department_.fit_transform(inputs['Department'])
inputs['BusinessTravel_'] = BusinessTravel_.fit_transform(inputs['BusinessTravel'])
inputs['Attrition_'] = Attrition_.fit_transform(inputs['Attrition'])
inputs['OverTime_'] = OverTime_.fit_transform(inputs['OverTime'])
inputs['Over18_'] = Over18_.fit_transform(inputs['Over18'])

inputs.drop(['MaritalStatus', 'JobRole', 'Attrition' , 'OverTime' ,
                'Gender', 'EducationField', 'Department', 'BusinessTravel', 'Over18'], axis=1, inplace=True)

inputs.rename(columns={'MaritalStatus_':'MaritalStatus',
                       'JobRole_' : 'JobRole',
                       'Gender_' : 'Gender',
                       'EducationField_' : 'EducationField',
                       'Department_' : 'Department',
                       'BusinessTravel_' : 'BusinessTravel',
                       'Attrition_' : 'Attrition',
                       'OverTime_' : 'OverTime',
                       'Over18_' : 'Over18'}, inplace=True)

inputs.head(2)

from sklearn import preprocessing

scaler = preprocessing.StandardScaler()
scaled_df = scaler.fit_transform(inputs)
scaled_df = pd.DataFrame(scaled_df, columns=inputs.columns)
scaled_df.head()

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
import numpy as np

inputs = scaled_df
X_train, X_test, y_train, y_test = train_test_split(inputs, target, test_size=0.2)

model = RandomForestClassifier(ccp_alpha=0.01)

# train
pred = model.fit(X_train, y_train)

from sklearn.metrics import accuracy_score
accuracy_score(X_test,pred) #--------------> this is where I get the error

Can someone help me?

# train
pred = model.fit(X_train, y_train)

Here you are juts fitting the model, not making any predictions.

accuracy_score(X_test,pred)

Here you are supposed to pass y_test, and y_predict, since its the output you are comparing not the input data.

Try this-

pred = model.predict(X_test)
accuracy_score(X_test,pred)


Is it possible to get worse model after optimization? I am trying recently to optimize models but for some reason, whenever I try to run the optimization the model score in the end is worse than before, so I believe I do something wrong.
in order to optimize my model I define param grid and than fit with the train data and then according to the results run again with nre parameters, e.g-
#ROUND 1
param_grid={
    'max_depth': [3,4,5],
    'learning_rate':[0.1,0.01,0.05],
    'gamma': [0,0.25,1.0],
    'reg_lambda':[0,1.0,10.0],
    'scale_pos_weight':[1,3,5]
}

grid_search = GridSearchCV(estimator = clf_xgb, param_grid = param_grid,
                          cv = 3, n_jobs = -1, verbose = 2)
grid_search.fit(X_train,y_train)
grid_search.best_params_

>>>.....

(and now based on the result changing the params...)
after this step I choose the best hyperparameters and run the model;
clf_xgb=xgb.XGBClassifier(seed=42,
                         objective='binary:logistic',
                         gamma=0,
                         learn_rate=0.7,
                         max_depth=6,
                         reg_lambda=0.8,
                         scale_pos_weight=1,
                         subsample=0.9,
                         cilsample_bytree=0.5)

clf_xgb.fit(X_train,
           y_train,
           verbose=True,
           early_stopping_rounds=10,
           eval_metric='aucpr',
           eval_set=[(X_test,y_test)])

The problem is that when I check the model score
clf_xgb.score(X_test,y_test)

I always get lower score than what I got before the optimization which makes me suspect that I'm missing something in the way doing it/basic principle in this process.
Is it possible that after running the optimization my score won't get better (and even worse?) ? Where is my mistake? Are there other parameters that could influence or improve my model?

Is it possible that after running the optimization my score won't get better (and even worse?) ?

Yes, theoretically, by pure luck, it is possible that your initial guess, before optimization of hyper-parameters, provides better results than the best of parameter combination found in the parameters grid. However, assuming you have enough data and your parameter grid is wide enough it is rather unlikely that the tuning of hyper-parameters would not be able to find better results. Such behavior rather indicates that something is wrong with your approach or your data.
If understand correctly, the choice of the best parameters is based on the cv results on training data, while in your final run the performance is assessed based on test dataset. If the distribution of training and test data differ significantly it could lead to the situation when the parameters providing the best results on the training data perform poorly on test data.

Where is my mistake?

As already mentioned by others, the parameters you are testing after the tuning were not included in the parameter grid. In this case it is incorrect to talk about the model performance "after running the optimization".
I suggest the following in order to investigate and fix the problem

Instead of using the hard-coded parameters in the XGBClassifier  call, use the optimal parameters found by tuning process, i.e. grid_search.best_params_. Furthermore, if you think that subsample and cilsample_bytree (a typo?) are relevant parameters include them in the parameters grid.
Increase the cv parameter to e.g. 5-10, the results with cv = 3 might be very unstable. You can assess the stability of your current results by using different random seeds and repeating the entire exercise.
Make sure that you use the consistent parameters in tuning process and in the final evaluation, or just include these parameters in the parameters grid if possible. In particular, check early_stopping_rounds and eval_metric.

Are there other parameters that could influence or improve my model?

From your code it is unclear how many rounds you use. Either increase n_estimators or include it in the parameters grid.
Given that you use AUCPR you might need to explicitly set the parameter maximize=True, otherwise in your final run you could minimize the AUCPR, which could explain poor results.


First steps with Python and scikit-learn I believe I have a simple if not trivial question. I have a background in statistics and I tend to use Stata and R quite a bit. I am interested in learning Python. I used it for a while now and recently came into contact with scikit-learn.

I am trying to reproduce a simple example.

from sklearn import tree
features = [[140, 1],[130, 1], [150, 0], [170, 0]]
labels = [0 , 0 , 1 , 1]
clf = tree.DecisionTreeClassifier()
clf = clf.fit(features, labels)
print clf.predict([[150, 0]])

As you can see, the tiny script tries to predict  - by the means of a decision tree - wether a object with the properties [150, 0] is likely a type 1 or 0.

I run the script and get following error:

File "clf_decision_tree.py", line 6
    print clf.predict([[150, 0]])
            ^
SyntaxError: invalid syntax

I realy don't get what is wrong... Can you help me out? Best /R

PS: I not sure if Cross Validated or Stackoverflow are better places to ask. Let me know. Thanks.

In python 3 the print function must have parenthesis, so print(clf.predict([[150, 0]])) will work


How is hinge loss related to primal form / dual form of SVM I'm learning SVM and many classic tutorials talk about the formulation of SVM problem as a convex optimization problem: i.e. We have the objective function with slack variables and subject to constraints. Most tutorials go through the derivation from this primal problem formulation to the classic formulation (using Lagrange multipliers, get the dual form, etc...).
As I followed the steps, they make sense eventually after some time of learning.

But then an important concept for SVM is the hinge loss. If I'm not mistaken, the hinge loss formula is completely separate from all the steps I described above. I can't find where the hinge loss comes into play when going through the tutorials that derive the SVM problem formulation.

Now, I only know SVM as a classic convex optimization / linear programming problem with its objective function and slack variables that is subject to constraints. How is that related to hinge loss??

Hinge loss for sample point $i$: $$l( y_i, z_i) = \max(0, 1-y_iz_i)$$

Let $z_i=w^Tx_i+b$.

We want to minimize

$$\min \frac1n \sum_{i=1}^nl(y_i, w^Tx_i+b)+\|w\|^2$$

which can be written as

$$\min \frac1n \sum_{i=1}^n\max(0,1-y_i (w^Tx_i+b))+\|w\|^2$$

which can be written as

$$\min \frac1n \sum_{i=1}^n \zeta_i + \|w\|^2$$

subject to $$\zeta_i \ge 0$$

$$\zeta_i \ge 1-y_i (w^Tx_i+b)$$

The constraint comes from hinge loss. It is a reformulation of a minimax optimization problem.


Clustering not producing even clusters I'm using k-means clustering to processes running on machines.

Dataset sample :

machine name, process
m1,java
m2,tomcat
m1,word
m3,excel

Build a matrix of associated counts :

   java,tomcat,word,excel
m1,1,0,1,0
m2,0,1,0,0
m3,0,0,0,1

I then run k-means against this dataset (have tried Euclidean and Manhattan distance functions)
The dataset is extremely sparse which I think is causing the generated clusters to not make much sense as many machines get grouped into the same cluster(as they are very similar)

How to achieve clusters where each cluster contains approx equal number of points ? Or perhaps this is not possible due to the sparseness of the data and instead I should try to cluster on a different attributes of dataset ?

Cluster analysis is not supposed to produce paritions of equal size. It is meant to discover structure in the data.

If the majority of objects is highly similar, then this majority is supposed to be in the majority cluster.

Consider all your data is identical. Any clustering algorithm producing more than one cluster has failed, in my opinion...

So you may be using the wrong class of algorithms for your problem.


How to include class features to linear SVM I am planning to do a simple classification with a linear SVM. One feature I have is another classification of some sort done previously.
Can I just use this class feature as a 1-hot encoded array? So, e.g. for 3 different classes, I'd have 3 binary features being 0 or 1?

The problem I see is that this feature is not linear but binary. Will this pose a problem? And if yes, how can I somehow transform a binary feature into a linear one?

The quick answer is yes, you can use a liner SVM in presence of an encoded categorical variable.

Short explanation: The linearity of the model has nothing to do with the features of model, and actually linear feature doesn't mean anything. The linearity refers to the model, i.e. the equation that links the target to the features.

The equation

$$y = b*x + q$$

means that y is linear with respect to x, not that the variable x is linear. x is x thats it.

What you should check when dealing with linear models is if your data is linearly separable, or in other words if you can separate the different classes with a straight line. If that is not the case you are in trouble and should probably think of changing the model.


Logistics Demand Forecasting with 20k Different Time Series I'm trying to tackle a very challenging problem and I would appreciate your help.
My organization has a lot of different items which can be demanded by our clients. Those items can also be returned back to our warehouses. Our goal is to predict how many instances of each item our clients (which are actual warehouses) will demand next year.
As for now, my organization is predicting the demand using a weighted average, based on the quarterly demands of the last 7 years. This gives us a prediction for the demand for the next quarter. Then, they multiply this prediction by 4 to get the next year's prediction. This is my baseline to beat.
Our data is irregularly sampled, meaning that demand can occur any day. This also means that we have to resample the data ourselves (we tried monthly and quarterly).
Each demand has a few features:

Item Id
Date of Demand
Receiving Warehouse - our organization has a lot of warehouses spread out across the country, this is the warehouse that received the item.
Sender Warehouse - the warehouse that sent out the item.
Type of Demand
Quantity - can be either positive (if the client consumed the item from the warehouse) or negative (if the client returned the item back to the warehouse)

Again, for clarification: the goal is to predict the Total Quantity of the next year for each item.
Obviously, this is a time-series forecasting situation. The problem is that we have over 20K different time series (because each item’s history is a distinct time series).
What we have done so far:

Simple statistic models (exponential smoothing, Holt-Trend Methods, etc.)
Classical machine learning models (linear regression, decision trees, etc.)
Ensemble methods (random forest, boosting, etc.)
Simple deep learning models – neural networks with a few linear layers and an RNN layer.
Complex state of the art neural networks for time series forecasting (we tried Google's Temporal Fusion Transformer)
Calculating the regular weighted average (as the company does now) but only after removing anomaly demands (based on a DBSCAN algorithm).

The main problem is that none of the above methods can beat the regular weighted average for all the time series we have – meaning that for many of them the new proposed model preformed even worse than the weighted average. Also, we couldn’t find any pattern that allowed us to match each time series to a "winning" method.
What we are looking for is a method which will perform better for all the different time series.
Any help will be highly appreciated.
Thanks in advance.

First of all, there is no guarantee that you can beat baseline with ML, this is something you should take into account.
You also might find a solution for some particular Type Of Demand due to some inner behaviour, but this might not generalize well for the rest of the cases.
I would also try to use Facebook`s Prophet
From the authors:
Prophet is a procedure for forecasting time series data based on an additive model where non-linear trends are fit with yearly, weekly, and daily seasonality, plus holiday effects. It works best with time series that have strong seasonal effects and several seasons of historical data. Prophet is robust to missing data and shifts in the trend, and typically handles outliers well.
It might work well out of the box, without any major tweaks. You can also run in in parallel (because the process of fitting of model is kinda automatic), which is helpful when you have this much time series.
Here is a link on how to implement it efficiently (but note that original docs are quite good as-well)
https://towardsdatascience.com/implementing-facebook-prophet-efficiently-c241305405a3
And a post regarding parallel execution
https://medium.com/spikelab/forecasting-multiples-time-series-using-prophet-in-parallel-2515abd1a245
I would also spend more time analyzing results: can you check the error between different types of demand, search for the particular cases when prophet beats baseline and try to understand why its so and so on.


Text topic classification in tensorflow I want to create a CNN in tensorflow that does the following:
Classify a recipe headline and find out the topic. For instance super yummy cheesy cake should result in cheese cake and so on.

I thought for going with tensorflow, but need some help in getting things started.

My strategy is like that:

Normalize the headlines so cheesy becomes cheese and cheesecake becomes cheese cake for instance and so on.
Having a dataset like:

super yummy cheesecake | cheese cake
summer strawberry cake | strawberry cake

Train the model to learn what matters for the topic and what is just additional information.

The way, the dataset is modeled, I have no static lables, as I understand. This makes things complicated, right?

As this is my first AI experiment with tensorflow, I don't really know if this will work out, or if I should go with another strategy, therefore I need your help.

To me it looks like not tensorflow task at all. At least not at the first place.

"Normalize headlines" task (lemmatization). Spacy does nice job here and it has great documentation. Here is an example, have a look at the "lemma" property.
Use the food2vec as a database of topic names.
Parse sentence via spacy and find the phrase in the food2vec. Parsing should be done not word-by-word, but by phrase: first look up 3-words phrase in the dictionary; if not found - 2-word dictionary; than 1 word.

This should be enough to solve your task.


how to use predictions on a single value? I am comfortable using Machine learning on my train data and test data and validate it.
But the question here is if I want to predict a single variable how do I do it?
Let's suppose I have done feature selection using lasso or applied PCA  or done feature engineering like transformation or scaling but when I want to utilize it for new data, maybe single data I am getting confused. Should I have to follow all the steps mentioned above even for the single row of data or how do I proceed with this. Please suggest.

Building a model pipeline is the way to go once you are done experimenting with your model. Following is a toy example -
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

#Generating random dataset
X, y = make_classification(random_state=0)

#Splitting test and tran sets
X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                    random_state=0)

#Defining a pipeline with standardscaling transformer and then svc estimator
pipe = Pipeline([('scaler', StandardScaler()), ('svc', SVC())])

# The pipeline can be used as any other estimator
# and avoids leaking the test set into the train set
pipe.fit(X_train, y_train)

#During prediction on a test or on a fresh set of data, just use the pipe object to run the whole pipeline and get predictions.
pipe.score(X_test, y_test)

You can easily create a pipeline in Sklearn for almost every possible transform available in Sklearn. Check the user guide for more details.
Another example with PCA -
from sklearn.pipeline import Pipeline
from sklearn.svm import SVC
from sklearn.decomposition import PCA
estimators = [('reduce_dim', PCA()), ('clf', SVC())]
pipe = Pipeline(estimators)
pipe

Pipeline(steps=[('reduce_dim', PCA()), ('clf', SVC())])
#Now you can use the fit method of the pipeline object to perform feature engineering and train model, while the scoring method for predictions

For Sklearn Pipelines, you can have either a single or multiple transformation paths (which are combined together with a FeatureUnion) and then it has to end with an estimator (only a single estimator is allowed)

While all this is powerful enough for most purposes, sometimes, there are custom transformations that you will need for your data, and those transformations will not be a part of the Sklearn API. In that case you can actually use the base classes of sklearn to defined your own transformer which can easily become a part of the pipeline that you define. More information here.


How is the cross-product transformation defined for binary features? I am reading the paper on Wide & Deep learning and for the wide component, it states that one of the most important transformations is the cross-product transformation. This is defined as follows:

$$\phi_{k}(\mathbf{x})=\prod_{i=1}^{d} x_{i}^{c_{k i}} \quad c_{k i} \in\{0,1\}$$

Then, the authors argue that for binary features, a cross-product transformation (e.g., “AND(gender=female, language=en)”) is 1 if and only if the constituent features (“gender=female” and “language=en”) are all 1, and 0 otherwise.

But how does this relate to the given formula? And how can I think of the cross-product transformation in general?

Let's do this in the opposite order of how you asked.  i.e. first:

How can I think of the cross-product transformation in general?

For me a cross-product comes from linear algebra, and it is a different transformation.  Therefore let's start by not confusing both.  The cross-product in the equation above is a transformation alright but is not related to what most of us call a cross product.

With that out of the way let's look at the formula.  Since both the index on the product and components of $\vec{x}$ and $\vec{c}$ are the same (they're all $i$), we can be rather confident that $d$ is the number of dimensions/features we are working with.  Therefore to make things a little easier to see let's work with $d = 4$:

$$
\phi_{k}(\vec{x})=\prod_{i=1}^{4} x_{i}^{c_{k i}} \quad c_{k i} \in\{0,1\}
$$

Now we can write an example vector to which we apply the transformation, say:

$$\vec{x} = [7, 5, -0.5, 0.2]$$

And an example vector which parametrizes the transformation:

$$\vec{c} = [0, 1, 1, 0]$$

And for a start let's evaluate the transformation:

$$\phi_k(\vec{x}) = 7^0 \cdot 5^1 \cdot -0.5^1 \cdot 0.2^0 = 5 \cdot -0.5 = -2.5$$

And that's it.  But now comes the important part, let's describe what this transformation just did: We took a vector in 4 dimensions, took it's 2nd and 3rd components and multiplied them together (whilst we ignored the 1st and 4th components).  In other words the transformation $\phi_k$ uses its parametrization $\vec{c}$ to decide which components of its argument to product together.

We can therefore use different parametrizations to achieve different transformations of the vector $\vec{x}$.

What this may be useful for? If you are confident that all components in the vector are $> 1$ (or all are $< 1$) then you have the combined magnitude of the selected dimensions.  Although keep reading below since this is really useful to binary features.

How does this relate to the given formula?

i.e. how the binary features can only be 1 if all (relevant) features are present.  I believe that this relates to the definition of $0^0$ and $0^1$.  Let's take a different vector:

$$\vec{x} = [0, 1, 0, 1]$$

Let's say that these are non-exclusive binary features: [male?, dog?, white?, fed?].  So we have a female dog which is not white and is currently well fed.

Now let's take the parametric vector to be:

$$\vec{c} = [0, 1, 1, 0]$$

And perform the transformation:

$$\phi_k(\vec{x}) = 0^0 \cdot 1^1 \cdot 0^1 \cdot 1^0 = 1 \cdot 1 \cdot 0 \cdot 1 = 0$$

A small note is now in order: $0^0 = 1$ is not true in all fields of mathematics, but since it is accepted in most fields and makes sense here we will use it here (and so I believe that the authors of the paper do use this assumption as such).

Wait!  So had we had the dog been white we would have had a $1$ as the result.  In other words, had $\vec{x}$ been $[0, 1, 1, 1]$ we would have gotten $1$ as the result of the transformation because the third term would be $1^1$.  Try it out!

This means that $\phi_k$ with $\vec{c} = [0, 1, 1, 0]$ will return $1$ (true) for all $\vec{x}$ that represents a white dog, does not matter if the dog is male or female or whether it is fed.  And will return $0$ for anything that is either not a dog or not white.

In other words, the parametric $\vec{c}$ defines the features that matter (the ones in the vector $\vec{c}$), and such a parametrized transformation $\phi_k$ return true if all the features that matter are true in the argument of the transformation ($\vec{x}$).

The big difference in the binary features is the fact that $0^1 = 0$, which binds the entire product to $0$.  Whilst $0^0 = 1$ and $1^1$, which results in a dimension/feature whose value does not matter for our transformation.

P.S. I prefer physics notation for vectors, a component of a vector is $x$ but a full vector is $\vec{x}$ instead of $\mathbf{x}$.  Please bear with the notation.


How to handle weekdays in a NN? I want to test if using additional information of weekdays would improve my NN. Therefore, I just converted the weekdays numerically such as

Monday -> 0
Tuesday -> 1
...
Sunday -> 6

but my NN fails totally with that alongside to other 16 variables. Without it's ok. Now I wonder if I maybe have too few data or if the conversion does not make sense at all but I guess I do not need something like one-hot-encoder, or do I?

What you have is a cyclical feature, i.e. a feature which takes values which repeat cyclically in time. With that trivial encoding, for any algorithm, Sunday will be one day from Saturday, but 6 from Monday. It should be one day, also.

The proper way to handle cyclic features is to encode them with a sine and a cosine variable. So instead of a feature weekday, you should have weekday_sin and weekday_cos, which are calculated as:

data['weekday_sin'] = np.sin(2 * np.pi * data['weekday']/7)
data['weekday_cos'] = np.cos(2 * np.pi * data['weekday']/7)

Although I can't assure you that this will work in your case, it is definitely a conceptual error you're dealing with.

For more in-depth explanation with some plots which make it a lot easier to understand, you can read this: https://www.kaggle.com/avanwyk/encoding-cyclical-features-for-deep-learning

Hope this helps!


Help making a custom categorical loss function in Keras I am a bit new to machine learning, and I'm trying to get the basics working towards a bigger project using a very simple encoder-decoder model. It looks like this:

embedding_dim = 300
lstm_layer_size_1 = 300
lstm_layer_size_2 = 300

model = Sequential()
model.add(Embedding(self.max_input_vocab, embedding_dim,
    input_length=self.max_input_length, mask_zero=True))
model.add(LSTM(lstm_layer_size_1)) # encoder
model.add(RepeatVector(self.max_output_length))
model.add(LSTM(lstm_layer_size_2, return_sequences=True)) # decoder
model.add(TimeDistributed(Dense(self.max_output_vocab, activation='softmax')))

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['acc'])

It takes in a sequence of words encoded as integers, with 0 padding up to max_input_length. And outputs a one-hot-encoded version of the output for words up to max_output_length.

For example, with a max ouput length of 115, and an expected output of length 20, the network should predict 20 integers in the range max_output_vocab, followed by 95 predicted 0's.

My problem:

I've been running into the issue that the network trains way too much off of the zero tokens in the output, as many of the target sequences have output lengths far below the max output length. The network ends up learning it can get the most accuracy by just predicting almost all 0's for most of the output.

I want to try to make a custom loss function that won't train on any output that comes after the first 0 token, but I'm not sure how I would go about doing this properly.

I know it will look similar to the keras.backend categorical_crossentropy, but would it be as simple as continuing to use a version of that function, but only feeding it the portion of the output sequence I want (everything before the first 0 token in the expected output)?

The issue is very easy to solve assumming you still want to use crossentropy as your loss.

Tell your model to use temporal sample weights. You can do it like this by model.compile(sample_weight_mode='temporal', **other_params)
Generate your sample weights, I think you are smart enough to write your own implementation. The idea is as you said apply weights of 1 if it is supposed to be counted and apply 0 if it is not supposed to be counted. For example you have a sequence [3,5,1,-3,2,0,0,0,0] then your sample weights will be [1,1,1,1,1,0,0,0,0].
Supply the sample weights during fitting. Simply use model.fit(X,y,sample_weights=sample_weights, **other_fit_params).

Done. Now loss is only counted on non-zero entries of your output.


Dicsrete values as taget variable I have discrete values in the target variable(Exactly 13 different values in total) . When I am giving that as input to Random forest Classifier ,it gives error that input as continuous. And if I give it to regressor it is predicting a value between the discrete values. How can I treat this problem

I suspect the problem is that your discrete values are non-integer floats.  The classifier's fit method runs type_of_target, which returns (in part):

  'continuous': y is an array-like of floats that are not all
            integers, and is 1d or a column vector.

(source)

(Tracing the source code wasn't straightforward; a traceback of the error from you would have been helpful.  For reference here, RandomForestClassifier inherits from BaseForest, whose fit method calls _validate_y_class_weight which (in the ForestClassifier child class) calls check_classification_targets which calls type_of_target.)


How to deal with TypeError: ufunc 'isnan' not supported for the input types I have dealt with all the Nan values in the features dataframe, then why I am still getting this error?

sns.heatmap(features, annot=True, annot_kws={"size": 7})
sns.plt.show()

TypeError                                 Traceback (most recent call last)
    <ipython-input-11-534a699b432d> in <module>()
    ----> 1 sns.heatmap(features, annot=True, annot_kws={"size": 7})
          2 sns.plt.show()

    c:\users\jetjo\appdata\local\programs\python\python36\lib\site-packages\seaborn\matrix.py in heatmap(data, vmin, vmax, cmap, center, robust, annot, fmt, annot_kws, linewidths, linecolor, cbar, cbar_kws, cbar_ax, square, xticklabels, yticklabels, mask, ax, **kwargs)
        515     plotter = _HeatMapper(data, vmin, vmax, cmap, center, robust, annot, fmt,
        516                           annot_kws, cbar, cbar_kws, xticklabels,
    --> 517                           yticklabels, mask)
        518
        519     # Add the pcolormesh kwargs here

    c:\users\jetjo\appdata\local\programs\python\python36\lib\site-packages\seaborn\matrix.py in __init__(self, data, vmin, vmax, cmap, center, robust, annot, fmt, annot_kws, cbar, cbar_kws, xticklabels, yticklabels, mask)
        166         # Determine good default values for the colormapping
        167         self._determine_cmap_params(plot_data, vmin, vmax,
    --> 168                                     cmap, center, robust)
        169
        170         # Sort out the annotations

    c:\users\jetjo\appdata\local\programs\python\python36\lib\site-packages\seaborn\matrix.py in _determine_cmap_params(self, plot_data, vmin, vmax, cmap, center, robust)
        203                                cmap, center, robust):
        204         """Use some heuristics to set good defaults for colorbar and range."""
    --> 205         calc_data = plot_data.data[~np.isnan(plot_data.data)]
        206         if vmin is None:
        207             vmin = np.percentile(calc_data, 2) if robust else calc_data.min()

    TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

There may be column of type object in your dataFrame. Remove it or convert it into float,int etc. if possible

From Documentation of Seaborn:

data : rectangular dataset

2D dataset that can be coerced into an ndarray. If a Pandas DataFrame is provided, the index/column information will be used to label the columns and rows.

so you can try

sns.heatmap(features.drop(['columnName01_OfTypeObject','columnName02_OfTypeObject'],axis=1), annot=True, annot_kws={"size": 7})

This will drop your columns temporarily.


Identifying whether training dataset is imbalanced in CART I'm new to machine learning domain. I want to build a Classification tree for Binary classification of data.

I have a training data set of 269 records out of which 56 records belong to 'yes' class and 213 records to 'no' class.

Is this data imbalanced for building CART model? Do I need to undersample 'no' class records? Also from Gini index, Chi-Square and Information Gain, which algorithm is best for node splitting?

P.S.:- I can't increase the size of the dataset further.

Yes, your data is not balanced.

Before applying any kind of modeling it is suggestible to balance the data, you can use packages like SMOTE,ROSE to overcome the imbalance in your data.

For your next question, which index to use: It is subjective to your data. Please go through these links Link-1, Link-2 for better understanding but suggestible to try both Entropy and Gini Impurity. There is a good chance to get similar trees and similar outcome.

Gain Ratio works better to reduce bias towards highly branching features like time, salary etc. (Information about the features of the dataset isn’t mentioned)

Do let me know if you have any additional questions.

Finally, how big is your test dataset?


List of keywords as features I'm new to machine learning, being this the first time I'm involved in a project in the area. I have a dataset of news articles and have extracted the keywords present on the news title such as ['china factory activity shrinks', 'first time', '2 years']. Note, the list size varies.

I would like to use this data in my features, but I don't know how and what is the best way to extract features based on a list of keywords.

Since the keywords cannot be converted to a categorical value, I think I can't use OneHotEnconder, but I'm not 100% certain. Is bag-of-words or TF-IDF viable possiblity? How can I encode this keywods to numerical values maintaining some meaningful information to the model?

Sure, of course we can use TF-IDF or bag of words. The easiest way is to build a separate TFIDF transformer for every group of keywords and then combine them together using FeatureUnion. Just make sure the custom tokenizer only filters out the keywords in each group that you're looking at.

from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.feature_extraction.text import TfidfVectorizer

corpus = [
    "china factory activity shrinks some text here",
    "first time some text here",
    "2 years some text here",
    "combining first time and 2 years here.",
    "adding china factory activity first but doesn't match exactly"
]

# custom tokenizers

keywords1 = lambda x: [x for x in x.split() if x in ["china", "factory", "activity", "shrinks"]]
keywords2 = lambda x: [x for x in x.split() if x in ["first", "time"]]
keywords3 = lambda x: [x for x in x.split() if x in ["2", "years"]]

combined_features = FeatureUnion([
    ('kwd1', TfidfVectorizer(tokenizer=keywords1)),
    ('kwd2', TfidfVectorizer(tokenizer=keywords2)),
    ('kwd3', TfidfVectorizer(tokenizer=keywords3))
])

pipeline = Pipeline([('bow', combined_features)])
output_corpus = pipeline.fit_transform(corpus)

For the deep-learning approach, I would recommend a similar kind of pattern so that the embedding learned from each group would have a different semantic meaning from another group. You'll just need to ensure that the tokenizer only considers the keywords from each group when it is processed. If you're using a pre-build word embedding; you'll just have a filter before it goes into each embedding so that it doesn't "leak" into another topic.

Example using Keras:

from keras import layers
from keras.models import Model
from keras.preprocessing.sequence import pad_sequences

def custom_sequence_tokenizer(txt, vocab):
    vocab_mapper = dict([(word, idx+1) for idx, word in enumerate(vocab)])
    text_split = [vocab_mapper.get(x, 0) for x in txt.split() if x in vocab]
    return text_split

keywords1 = pad_sequences([custom_sequence_tokenizer(x, ["china", "factory", "activity", "shrinks"]) for x in corpus], padding='post', maxlen=10)
keywords2 = pad_sequences([custom_sequence_tokenizer(x, ["first", "time"]) for x in corpus], padding='post', maxlen=10)
keywords3 = pad_sequences([custom_sequence_tokenizer(x, ["2", "years"]) for x in corpus], padding='post', maxlen=10)

input_count1 = layers.Input(shape=(10,), name='in1')
input_count2 = layers.Input(shape=(10,), name='in2')
input_count3 = layers.Input(shape=(10,), name='in3')

# input size is reflection of the vocab size
embed1 = layers.Embedding(5, 8)(input_count1)
embed2 = layers.Embedding(3, 8)(input_count2)
embed3 = layers.Embedding(3, 8)(input_count3)

combine = layers.Concatenate()([embed1, embed2, embed3])

model = Model(inputs=[input_count1, input_count2, input_count3], outputs=combine)
output = model.predict({
    'in1': keywords1,
    'in2': keywords2,
    'in3': keywords3,
})


Forecasting time series: Method Selection Im new to forecasting time series and Im looking for some advice on selecting the best method based on the analaysis of the graph.

I have the following data and based on the little knowledge I have, Im assuming it is a stationary time series because of the shape of the data and because of the result I got when performing a Dicker Fulley Test

Test Statistic                -6.560544e+00
p-value                        8.402824e-09
#Lags Used                     2.100000e+01
Number of Observations Used    1.164000e+03
Critical Value (5%)           -2.864026e+00
Critical Value (1%)           -3.435980e+00
Critical Value (10%)          -2.568094e+00
dtype: float64

That made me choose Arima with a p and q values of 1. However the result Im getting are pretty awful. Can anyone guide on how should I choose the adequeate method for forecasting and  if case ARIMA is a good choice, how should I tweak parameters to improve my results? The end goal is to predict the next month of data, so it might not be necessary to use all of the data.

You can try two different approaches:

1) Kalman filter, the method is battle-tested and has proven useful in many areas.

Resources:

Understanding the Basis of the Kalman Filter
kalman filter in pictures

2) Recurrent Neural Networks, the LSTM and GRU architectures are particularly interesting for time series predictions.

Resources:

RNN effectiveness
Understanding LSTMs

To do regression and predict future data points, you would need to build a training dataset consisting of a sequence of events. Let's say a value $x$ for every timestamp $t$.

Your data seems to have 1 dimension, so both the network input layer and the output layer would consist of 1 unit. You would then train your model to predict $(x_{t+1})$ given $(x_{t})$.

Let $M$ be our trained model and let's say you want to forecast a data point at time $k$ and you know the current value at time $t$.

$M(x_{t}) = (x_{t+1})$

$t = t+1$

$M(x_{t}) = (x_{t+1})$

$...$ increment $t$ and keep predicting until $t+1 = k-1$

$M(x_{t+1}) = (x_{k})$

Put in pictures this corresponds to:

(picture from Udacity lecture about Deep Learning)


EagerTensor object does not support item assignment I'm trying to assign a new Value to a TF-Array. Here's my Code:

import tensorflow as tf
x = tf.zeros(shape=[5],dtype=tf.float32)
x[1]=0

The error message:

'tensorflow.python.framework.ops.EagerTensor' object does not support item assignment

What am I doing wrong?

You can't do this in Tensorflow.

For reference see: https://github.com/tensorflow/tensorflow/issues/33131


Detect if my ANN model is overfitted I've been trying the kaggle dataset of Credit card fraud detection Dataset . I've used ANN using keras and tensorflow. You can find the code in the screenshot. The only problem is im getting accuracy to be around 99.9 % , so i think it's surely a case of some false hopes or over fitting. Can you please tell whats wrong with it? And even my test set gave a result of 99.93% accuracy.

This dataset presents transactions that occurred in two days, where we have 492 frauds out of 284,807 transactions. The dataset is highly unbalanced, the positive class (frauds) account for 0.172% of all transactions.

I guess your model learnt nothing at all ;)

You should consider some form of resampling and using metrics that can handle imbalance.
This might be a good starting point. You can find bunch of similar threads here as well.


GAN: Discriminator converges, generator learns almost nothing In my GAN, the discriminator loss goes down steadily, while the generator loss oscillates / does not converge.

I suspect this is due to the vanishing gradient problem. Theory: as the discriminator loss doesn't start out big to begin with, the generator will never be backpropagated small gradients into, and therefore it doesn't learn to generate better images.

What are some remedies to resolve this and make the generator actually work?

This is one of the bigger problems with GANs, they are usually hard to train.

This is an active area of research and it does not seem like this problem has been solved. Without additional information or code, it is hard to help, but usually different loss functions are one way to mitigate the problem.


recommendations for self-study in ML/Deep Learning/the underlying math I am a beginner in ML, though I have completed Andrew Ng's Machine Learning course on Coursera and am soon to complete his Deep Learning specialization, also on Coursera. I'm looking for recommendations for things to study after that.

They need to be on the easier side since I am still learning. Also in addition to resources on ML or Deep Learning, I'm also interested in any resources on the underlying math. I have a book in probability and another book in calculus, though both are fairly dense. Anything lighter might be a plus.

Are there any resources you would recommend? I'm particularly interested in other courses on Coursera since that's what my company currently sponsors, though I'm interested in whatever suggestions you have to offer.

If you're looking for an introduction to mathematics for data science, take a look at the Coursera course Data Science Math Skills.


Forecasting Consumption for Multiple Products for Multiple Regions Came across a very interesting Real-World Time Series Forecast Problem.

Can you please help me understand the right track to resolve the below Time Series problem:

Input Data Sample:

and we want to predict the consumption of Product at each location by type for next 6 months.

I have tried doing it by building a simple ARIMA model to get monthly values and then multiplied by the factors calculated for consumption by Product, Type etc.

But can this problem be solved by any other Time Series Technique?
Can we use SARIMAX? If yes, then how can we deal with exogenous regressors being Categorical Variables?

Few observations in data:

Tea might have been consumed by a country in 2017 have now been replaced by Tea2 (Tea type 2) for next years, so how does time series maintain the continuity in data.

Any leads and solution will be highly appreciated.

Models

Xgboost can be used for Time Series. Here is a tutorial on Kaggle.
If Deep Learning is an option (the data set
has at least several thousand samples), you can try a RNN
(Recurrent Neural Network), with LSTM or GRU cells.

Features
You can transform categorical variables into indicator variables with the get_dummies function in pandas.


Pattern Recognition on Financial Market Which machine learning or deep learning model(has to be supervised learning) will be best suited for recognizing patterns in financial markets ?

What I mean by pattern recognition in financial market :
Following Image shows how a sample pattern (i.e. Head and shoulder) looks like:

Image 1:

And Following Image shows how it actually forms in real chart events:

Image 2:

What I'm trying to do is:
Any pattern similar to Image 1 can be defined as Head and Shoulder Pattern but in a Chart (Price Chart) it will not form as clearly as Image 1. Image 2 is the sample of Head and Shoulder Pattern form in Chart (Price Chart).
As it seems in Image 2, it can not be identified as Head and Shoulder Pattern by normal algorithms or analysis (Because there are a lot of highers and low forming a lot of structure, which can easily mislead into a lot of shoulders or head or any other structures).
I'm expecting to train machine to recognize the Head and Shoulder Pattern when similar (as Image 2) pattern is formed.

Thank you for your time.

Let me know if I'm taking it to wrong way.
I only have beginners knowledge on Machine Learning.

You should look at a classifier based on Dynamic Time Warping (DTW) distance. DTW is a method that calculates an optimal match between two given sequences (e.g. time series). It has been used in a supervised learning setup, in particular it has been reported to achieved state of the art results, when used in a nearest neighbour classifier.


Dealing with a dataset where a subset of points live in a higher dimensional space I have a classification problem where I am dealing with economic data in a high dimensional (~400) space which includes dates, addresses, salaries, and a number of other variables. Most of the variables are made up of discrete categories and so these end up being encoded into many dimensions of binary variables.

There is a subset of the data where observations are made in two separate locations instead of just one. As a result, many of the dimensions are repeated for this subset. Most of the repeated data results in extra dimensions of binary class variables which would just be zero for the compliment set (i.e. only one location) of observations. However, one dimension is a salary where there are real values for the 2-location subset but Nans for the remaining data. To illustrate:

Obs | pay_1 | loc_1_A | loc_1_B| ... | job_1_A | job_1_B | ... | pay_2 | loc_2_A | ...
======================================================================================
1   | 50k   |   1     |   0    |     |   1     |   0     |     |   40k |   1     |
2   | 40k   |   0     |   1    |     |   1     |   0     |     |   45k |   1     |
3   | 70k   |   1     |   0    |     |   0     |   1     |     |  NaN  |   0     |
4   | 50k   |   1     |   0    |     |   1     !   0     |     |  NaN  |   0     |

Question:

How to deal with data that has this systematically disjoint set of dimensions in which it lives?

I believe for many of the variables in the higher-dimensional subset it is no problem, since they will simply equal zero for any of the data points not in this subset. But since it is a classification problem, it seems pertinent for the sake of numerical stability to standardise the salary data ($pay_2$) to have values similar to a standard gaussian. Thus, I could not set the NaNs to zero for the lower-dimensional observations as zero would be a meaningful value.

I can think of a couple of approaches, one being to have a kind of non-linear "switch" variable that turns the higher dimensions on and off:

$$y = A * pay_1 + B * \text{loc_1_A} + ... + S * ( M * pay_2 + N * \text{loc_2_A} + ... )$$

though I don't see how I can implement this in a lot of standard methods.

Another option could be to just copy the $pay_1$ values into the $pay_2$ column for those rows where there is no existing $pay_2$ value. This would seem to me like it is imposing a lot of structure onto the data that is difficult to justify. Maybe if I add another binary "switch" variable on top of this, it can separate the classes in my data space?

I know this is probably a very problem/domain-specific issue but I would like to know if there are any standard treatments for data which has this kind of structure, as I'm sure it is a pretty common scenario. Any suggestions of ways to think about this or similar problems in other fields to look at?

Why don’t you just put the data with pay_2 40k and 45k into two additional rows?

Additionally, you could add an indicator variable loc_1_2 and set it to 1 or 2 depending on the location.

In general, dummy encoding might be dangerous. Instead you could convert your addresses or locations into physical coordinates (latitude and longitude) or just distance from, let’s say, centre of the city or center of the country (depending on your data).

Edit

Ok, I see now that putting the data with pay_2 into additional rows will not work.

  But since it is a classification problem, it seems pertinent for the sake of numerical stability to standardise the salary data (pay2) to have values similar to a standard gaussian.

For your information: not all classification algorithms need standardised data, e.g. you may want to consider random forest or GBMs. If you stick to SVMs, regularised logistic regression or similar then I agree, it’d make sense to standardise.

  Thus, I could not set the NaNs to zero for the lower-dimensional observations as zero would be a meaningful value.

The standard approach you asked about, would depend on the properties of your classifier. I’m aware of the following options:

If you classifier can handle NaNs (NAs), e.g. R version of xgboost, then you can just use the data as you show in your table above.
If you have to standardise the data, then you can use mean-imputation. The average and standard deviation that are used for the standardisation should be estimated on non-NaN data, of course, (pay_2 column excl NaNs). In general, in order to prevent data leakage you should not use the data from holdout data subset (or “test”, “validation” or whatever you call it) for calculating the mean and standard deviation.
If your classifier doesn’t require standardisation, you can put some numerical sentinel values instead of NaNs, e.g. -1, 0, or -9999.

If you pick option 2 or 3, you’ll need to add a binary predictor indicating whether pay2 column was modified or not (NaN or not).

Re dummy encoding:
You haven’t asked about it explicitly and it’s difficult to suggest anything concrete without knowing the cardinality of your nominal variables, so I just put here the link to my answer about impact coding


How to use two different datasets as train and test sets? Recently I started reading more about NLP and following tutorials in Python in order to learn more about the subject. The problem that I've encountered, now that I'm trying to make my own classification algorithm (the text sends a positive/negative message) regards the training and the testing datasets. In all the examples that I've found, only one dataset is used, a dataset that is later split into training/testing. I have two datasets, and my approach involved putting together, in the same corpus, all the texts in the two datasets (after preprocessing) and after, splitting the corpus into a test set and a training set.

datasetTrain = pd.read_csv('train.tsv', delimiter = '\t', quoting = 3)
datasetTrain['PN'].value_counts()

datasetTest = pd.read_csv('test.tsv', delimiter = '\t', quoting = 3)
datasetTest['PN'].value_counts()

corpus = []
y = []

# some preprocessing
    y.append(posNeg)
    corpus.append(text)

from sklearn.feature_extraction.text import TfidfVectorizer
transf = TfidfVectorizer(stop_words = stopwords, ngram_range = (1,1), min_df = 5, max_df = 0.65)
X = transf.fit_transform(corpus).toarray()

# Splitting the dataset into the Training set and Test set
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.11, random_state = 0)

The reason why I've done this is because I'm working with the Bag of Words model and if I'm creating from the beginning X_train and X_test (y_train, y_test respectively) and not using the splitting function, I get an error when running the classification algorithm:

X_train = transf.fit_transform(corpustrain).toarray()
X_test = transf.fit_transform(corpustest).toarray()

...

classifier.fit(X_train, y_train)
y_pred = classifier.predict(X_test)

ValueError: Number of features of the model must match the input. Model n_features is 2770 and input n_features is 585

I'm kind of new at this and I was wondering if anyone could please guide me in the right direction?

You may want to use a pipeline to do this operation. Specifically, you do NOT want to train the TFIDFVectorizer the entire corpus- doing so gives your model hints about what features may be in the test set that don't exist in the training set- a concept frequently referred to as "leakage" or "data snooping".

The correct pattern is:

transf = transf.fit(X_train)
X_train = transf.transform(X_train)
X_test = transf.transform(X_test)

Using a pipeline, you would fuse the TFIDFVectorizer with your model into a single object that does the transformation and prediction in a single step. It's easier to maintain a solid methodology within that pattern.

In the example code, you're both fitting and transforming in the same step fit_transform, which is creating different features each time and is the source of your error.


GausianNB: Could not convert string to float: 'Thu Apr 16 23:58:58 2015' I'm beginner in python so please bare with me. I'm trying to solve one machine learning problem using GaussianNB. I've certain fields which are not in proper date format, so I converted it into UNIX format. For example column state_changed_at has value in csv as 1449619185. I'm converting it into proper date format.

Now the problem is, when I'm selecting those date features to train my model, it gives me an error:

  Could not convert string to float: 'Thu Apr 16 23:58:58 2015'

import pandas as pd
import numpy as np
from sklearn import metrics
from sklearn.naive_bayes import BernoulliNB
from sklearn.naive_bayes import MultinomialNB
import time
from sklearn.naive_bayes import GaussianNB

train = pd.read_csv("datasets/train2.csv")
test = pd.read_csv("datasets/test.csv")
train.head()

import time

#   state_changed_at,deadline,created_at,launched_at are date time fields
# and I'm converting it into unix format
unix_cols = ['deadline','state_changed_at','launched_at','created_at']

for x in unix_cols:
    train[x] = train[x].apply(lambda k: time.ctime(k))
    test[x] = test[x].apply(lambda k: time.ctime(k))

#   state_changed_at,deadline,created_at,launched_at are date time fields.
cols_to_use = ['keywords_len' ,'keywords_count','state_changed_at','deadline','created_at','launched_at']

target = train['final_status']

# data for modeling
k_train = train[cols_to_use]
k_test = test[cols_to_use]

gnb = GaussianNB()

model = MultinomialNB()
model.fit(k_train, target)  # this lines gives me error saying: could not convert string to float: 'Thu Apr 16 23:58:58 2015'

expected = target
predicted = model.predict(k_test)
print(model.score(k_test, predicted, sample_weight=None))

Any help would be really appreciated.
Thank you

To cast that column of your data-frame as type float try:

k_train = train['cols_to_use'].astype(float)
target = train['final_status'].astype(float)

More documentation can be found here or you can cast them when loading the csv file sep=',' maybe usful assuming that your data is separated by a , in your CSV file

train = pd.read_csv("datasets/train2.csv",
                    dtype={'cols_to_use': float , 'final_status': float})

Note that converting unix TimeStamp to readable date is very simple with the datetime library example:

import datetime

datetime.datetime.fromtimestamp(1526972723).strftime('%Y-%m-%d %H:%M:%S')

I hope it's helpful.


How to encode high cardinality categorical data? I have a dataset of 1600 rows and 28 columns. Only one column is partially complete with 1300 records. The rest is NaN. I did a value count of this columns and it has 84 different categories that are nominal. What is the best way to impute this column.
I need to convert these in numbers impute it and then convert back.
I understand that One-Hot encoding does not work in this case because of the high cardinality.
What is the best way to approach this problem?

You may want to look into Target Encoding as an example:
https://contrib.scikit-learn.org/category_encoders/targetencoder.html
Another post from the forums:
Problem with converting string to dummy variables


Is it possible to customize the activation function in scikit-learn's MLPClassifier? Scikit-learn lists these as the implemented activation functions for it's multi-layer perceptron classifier:

‘identity’, no-op activation, useful to implement linear bottleneck, returns f(x) = x
‘logistic’, the logistic sigmoid function, returns f(x) = 1 / (1 + exp(-x)).
‘tanh’, the hyperbolic tan function, returns f(x) = tanh(x).
‘relu’, the rectified linear unit function, returns f(x) = max(0, x)

Does Anyone know if it is possible to implement a custom activation function? If not, can someone point me to a library where this is possible?

Thanks!

Although @Emre's comment that sklearn wasn't specifically made for the task is correct, nothing stops you from introducing another activation function by implementing it in sklearn/neural_network/_base.py. You should be able to find the file in your installation.

To this end, you would define your single-parameter function and add it in the ACTVATIONS list. Refer to the existing implementations for some guidance how to do it.


Trying to understand encoder-decoder sequential models in Keras? My understanding is that for some types of seq2seq models, you train an encoder and a decoder, and then you set aside the encoder and use only the decoder for the prediction step. For example this seq2seq time series prediction model from Uber:

Now I am trying to implement a to version of this in Keras.

This is the Keras code for a vanilla LSTM:

# define model
model = Sequential()
model.add(LSTM(50, activation='relu', input_shape=(n_steps, n_features)))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')
# train model
model.fit(X, y, epochs=200, verbose=0)
# predict
x_input = array([70, 80, 90])
x_input = x_input.reshape((1, n_steps, n_features))
yhat = model.predict(x_input, verbose=0)

This is the Keras code for a stacked LSTM model:

# define model
model = Sequential()
model.add(LSTM(50, activation='relu', return_sequences=True, input_shape=(n_steps, n_features)))
model.add(LSTM(50, activation='relu'))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')
# train model
model.fit(X, y, epochs=200, verbose=0)
# predict
x_input = array([70, 80, 90])
x_input = x_input.reshape((1, n_steps, n_features))
yhat = model.predict(x_input, verbose=0)
print(yhat)

And this is the Keras code for an encoder-decoder model:

# define model
model = Sequential()
model.add(LSTM(100, activation='relu', input_shape=(n_steps_in, n_features)))
model.add(RepeatVector(n_steps_out))
model.add(LSTM(100, activation='relu', return_sequences=True))
model.add(TimeDistributed(Dense(1)))
model.compile(optimizer='adam', loss='mse')
# train model
model.fit(X, y, epochs=100, verbose=0)
# predict
x_input = array([70, 80, 90])
x_input = x_input.reshape((1, n_steps_in, n_features))
yhat = model.predict(x_input, verbose=0)

The problem is, I don't see much difference between the encoder-decoder code and the vanilla and stacked LSTM. In particular, I don't see how we are using only the decoder in the predict step, and which variable or method in Keras corresponds to the embedding that we would be using as an input for predicting new time series?

How can I implement the code for a model similar to the one in the illustration using Keras?

First of all, I personally think that the code you provided is unnecessary since you are looking for architectural comparison between models. It would have been easier (and less bulky) to include the graphs of your vanilla and stacked LSTM networks for simplicity. You can do that by keras.utils.plot_model() as explained here.

Answering your questions:

  I don't see how we are using only the decoder in the predict step...

Uber's model does not use the decoder component of the autoencoder. It uses a separate sequential LSTM model, judging by the graph provided.

  ...which variable or method in Keras corresponds to the embedding that
  we would be using as an input for predicting new time series?

The input to the forecasting model is simply the latent space of the autoencoder. In particular, the latent space constitutes a condensed representation of the past observations, so that you don't need to include the complete timeseries because it might be very long and include redundant information, repetition, noise, etc. Instead, you compress it into something similar to a "zip" file, that has smaller size but still includes all the essential information. Then, you train the "forecasting" network to predict future values based on the "zipped" past information instead of training it on the original timeseries.

  How can I implement the code for a model similar to the one in the
  illustration using Keras?

Check the 4th model of this autoencoder example. First, train the autoencoder. Only after it is properly trained and optimized, train a simple LSTM forecasting model, like the one in the code you provided.

The key is to use the latent space of the autoencoder to train the forecasting model, which in the current example is the output of the 1st LSTM layer. To do that, you need to first encode your complete dataset into a latent representation using the already trained encoder, and then use these encodings to train the forecasting model in the second step.


Rank feature selection over multiple datasets Through backward elimination I get a ranking of features over multiple datasets. For example in the dataset 1 I have the following ranking, the feature in the top being the most important:

feat. 1
feat. 2
feat. 3
feat. 4.

...

, whereas for dataset 2 I have for example the following ranking:

feat. 3
feat. 1
feat. 2
feat. 4.

I want to filter out those features which end up in the top of the ranking the most (incorporating that finishing in the top is better than finishing in the 3rd place). Which kind of ranking metric can I use for this problem?

An easy one to try would be average ranks, where you take the mean of the ranks for each feature. For your example,

$
\begin{array}{cc}
  \textbf{Feature} & \textbf{Avg. Rank} \\
  1 & 1.5 \\
  3 & 2 \\
  2 & 2.5 \\
  4 & 4 \\
\end{array}
$

You could also weight the ranks by the size of the dataset, if the datasets that you are testing on are not the same size.


Item-to-Item recommendation using DNN I am new to DNN still learning, have a need to build item-to-item content based recommendation using DNN. For example, say I have a column of strings where each row represents a document I need to compute the cosine similarity of this column and recommend similar documents.
id   document

1     "hi this document is about science"

2     "hi this document is about wars"
3     "This document is about peace"

now need to train based on document and column and recommend all the documents similar to a given document. I have solved this problem by using cosine similarity in ML now wanted to achieve the same in DNN (Deep Neural Nets). I have spent a lot of time on the net they all talk about movie rating where userId, rating, title is used. The problem I only have one column document how can I achieve this in DNN?

To be able to compare strings / Words or documents the data needs to be converted to a format the computer understand, vectors.
Google has a nice guide on Universal Sentence Encoder for sentence similarity that you can follow which explains how to generate a vector from the neural network that they have already trained.
What they do and what you can try is to use cosine similarity to compare the vectors. Compare one sentence vector with all other vectors and find the closest.
I am adding Annoy here, because I find it quite intuitive to use and you can get the n closest items.


How to encode ordinal data before applying linear regression in STATA? I have a data set that has student performance marks (continuous and dependent variable), Teacher Qualification (Ordinal and independent variable containing categories: Masters, Bachelors, High School). I want to apply the regression analysis to check the impact of teacher qualification on student's marks.
How can I encode ordinal data before applying linear regression?

I think the best way is to dummy-encode teacher qualification. So each level of qualification enters the regression with a separate intercept term. Note that dummy-encoding always works against a contrast level. So when "Master degree" is the base-level, you will see the effect of "Bachelor" compared to "Master" etc.
You can dummy-encode in Stata by using the i. prefix, e.g. summarize i.size. In a regression you would use reg y i.x.
See the Stata docs for details.


What is an efficient clustering approach for grouping multivariate timeseries data into optimally persistent states? I have a length 200000 time series of six dimensional data describing the evolution of a chaotic dynamical system. I am exploring ways to partition the state space of the data into minimally interacting regions; I want the least transitions from cluster to another in my time series. Other than not being interested in the trivial case of only 1 cluster I have no restrictions on cluster number though I expect it to be small (K<6 say).
The purpose of this is to try and identify quasi-stable basins of attraction in the chaotic system.

I have tried Hidden Markov Models with both Gaussian and Gaussian Mixture emissions, but they do not well capture the very non-gaussian structure of my data in phase space. I have also tried using DBSCAN with some success but since it didnt treat data as a time series, the parameters had to be very carefully tuned to get good results, which left me dissatisfied.

Do any other approaches leap to mind?

You can trivially modify DBSCAN (see GDBSCAN) to take time into account rather than dropping the precious time information.

I.e. neighbors are those points that are within a time difference of dt, and that have a minimum similarity.

Apart from that, classic "change in mean" analysis techniques for time series come to my mind. Maybe also changepoint detection.


Univariate Outlier Detection Let's say I have a dataset with the following format:

customerid
product
orders_in_last7days
orders_in_last6days
orders_in_last5days
orders_in_last4days
orders_in_last3days
orders_in_last2days
orders_in_last1days
orders_currentday

This dataset could have multiple customers and some customers could place $n$ numbers of orders on different days. How can I flag customers that have unusual number of purchases on the current day, by looking at the distribution of orders on the previous day for that specific customer?

The definition of an outlier per si is already quite dubious, but we can define them as those values that surpass the limit of +- 1.5 * IQR. In this case, either the standard deviation method or the Tukey method are valid options. We just need to try and see which gives better results.

# Tukey Method

n = 2 #In this case, we considered outliers as rows that have at least two outlied numerical values. The optimal value for this parameter can be later determined through the cross-validation
indexes = []

for col in df.columns[0:14]:
    Q1 = np.percentile(df[col], 25)
    Q3 = np.percentile(df[col],75)
    IQR = Q3 - Q1

    limit = 1.5 * IQR

    list_outliers = df[(df[col] < Q1 - limit) | (df[col] > Q3 + limit )].index # Determine a list of indices of outliers for feature col

    indexes.extend(list_outliers) # append the found outlier indices for col to the list of outlier indices

indexes = Counter(indexes)
multiple_outliers = list( k for k, v in indexes.items() if v > n )

Once you detect the outliers, you can either remove them or replace them with max/min limit values. In the first case, you just need to do this:

df.drop(multiple_outliers, axis = 0)

df = df.drop(multiple_outliers, axis = 0).reset_index(drop=True)

But for the second case, you should do this:

#Setting the min/max to outliers using standard deviation
for col in df.columns[0:14]:
    factor = 3 #The optimal value for this parameter can be later determined though the cross-validation
    upper_lim = df[col].mean () + df[col].std () * factor
    lower_lim = df[col].mean () - df[col].std () * factor

    df = df[(df[col] < upper_lim) & (df[col] > lower_lim)]

Finally, you can also use Isolation Forest or LocalOutlierFactor (more appropriate for Anomaly/Fraud Detection Problems).


options available for visualizing a matrix type data frame in R I have a data frame looking like the following:
p1-p5 are different products

A-C: price bands
1-3: size bands
e.g. A1: small store selling the expensive that product
     D3: big store selling cheap that product
values are the numbers of stores for each category A1-C2 per product

type    p1  p2  p3  p4  p5
  A1    158 48  18  6   62
  A2    343 151 58  49  55
  B1    400 181 200 29  863
  B2    398 093 448 215 795
  C1    809 217 154 15  422
  C2    877 549 340 46  576

What would it be the best way to visualize that information and which packages are available for that. I have found gridplot but unfortunately I havent managed to try it yet as I have issues with the Rstudio version.

Examples are more than welcome!

I've got that so far from a function found online

I'm expecting your data in this form (use melt to transform the data):

   type product value
1   A1      p1   158
2   A2      p1   343
3   A3      p1   247
4   B1      p1   400
5   B2      p1   398
6   B3      p1   408

You may easily (using lattice package) draw a barchart diagram.

Personally, I'd switch to heatmap only after this diagram is no more feasible due to large number of levels in the product or channel factors.

library(lattice)
barchart(product~ value | type,
data =  dfp,
scales=list(x=list(rot=90, cex= 1 ),y=list(cex=1)),par.strip.text=list(cex=1), origin = 0,
ylab="Product", xlab="Volume",
main= "Matrix Visualization")

Of course you may also swap the dimensions by changing the formula. Use the parameter layout to align the barchart based on the number of levels.

barchart(type~ value | product,
data =  dfp,
scales=list(x=list(rot=90, cex= 1 ),y=list(cex=1)),par.strip.text=list(cex=1), origin = 0,  layout= c(5,1),
ylab="Channel", xlab="Volume",
main= "Matrix Visualization")


Obtaining column values for a dataframe from another dataframe based on a common column variable I have 2 dataframes:
df1 = pd.DataFrame({"animal": [Cat, Dog, Rat, Bull, Dog, Bull, Bull, Dog, Cat, Rat, Dog], "lifeSpan": [2, 4, 6, 0, 4, 0, 0, 4, 2, 6, 4]})
df2 = pd.Dataframe({"animal":[Bull, Cat, Rat, Dog]})
The "LifeSpan" value is unique to each i in df1["animal"]. I need a third dataframe that mentions the "lifeSpan" value in a separate column against each j in df2["animal"] based on the 'lifespan' values in df1.

Just use pandas.merge to combine the two dataframes:
import pandas as pd

df1 = pd.DataFrame({"animal": ["Cat", "Dog", "Rat", "Bull", "Dog", "Bull", "Bull", "Dog", "Cat", "Rat", "Dog"], "lifeSpan": [2, 4, 6, 0, 4, 0, 0, 4, 2, 6, 4]})
df2 = pd.DataFrame({"animal":["Bull", "Cat", "Rat", "Dog"]})

df3 = df2.merge(df1.drop_duplicates(), on="animal")

This returns the following dataframe:

animal
lifeSpan

Bull
0

Cat
2

Rat
6

Dog
4


How to deal with Optional Input I'm from the vision world and only worked with pixels from 0-255, ignoring any side effects. My current problem is different, in the way that I cannot rely on the input data.

What my problem is:

I have a number of inputs. Each input is categorical (for now) and optional. For example I have a number of user features, {male, female, [..not given]}, {single, relationship, … , [..not given]}, ..

What I want:

X optional Input-Features mapped to Y Output-Features, with uniform output vales across the features.

I tried kernel pca, different kinds of matric factorization for guessing missing inputs and simple autoencoder networks. By sight, the last two yield "ok" results. Any advice here?

It sounds like you essentially have a sparse input problem, similar to doing something like a recommendation system. Imagine trying to recommend a film to somebody based on films they have already watched and rated. There would be many many films they have not seen, so you (in a sense) have missing data. This was the case in the Netflix Prize challenge.

Facorisation methods such Single Value Decomposition are common approaches in these cases. There is a nice summary of such methods used in the Netflix Prize.

Using a neural network Auto-Enconder will be reliant on relatively large numbers of samples, which is something to bear in mind when assessing performance.

Just one other idea that might be worth trying out, depending on how many features you have and the general sparsity, could be to replace the missing values with fixed values on a feature-level. This could be as simple as replacing the missing values e.g. in "relationship status" to <RELATIONSHIP>, and then in general "feature X" to just be <X>. This is a kind of trick that is often used in NLP (natural language processing) to allow filler-words to encode some useful information.

In your case, this might at least help any model distinguish between missing values across features. I have not tested something like this myself, so unfortunately I cannot cite references nor point to results.


How to Standardize/Normalize three sets of test data of different lengths I have three sets of data that were sampled at the same interval (20hz). The data is from a data logger that measured vehicle speed at a specific time.

In each of the tests, the Car starts at the same starting point A, and is manually drive to the end point B. This was performed 3 times. The point at which the vehicle comes to a stop is different each time since it is manually driven but that is intended.

My intention is to gather enough data sets from this test route so if another person drives the test route, we can compare how he performed.

I would ultimately like to average the data sets from my run and call it the optimal path and add an error band, so if someone else runs the test route, I can compare how they did to it, like distance from end of route, and max speed through a turn.

Initially I thought I could plot the data vs distance and take the average that way but the problem I am running into is, log A is 400 samples, log B is 420 samples, log C is 440 samples. So they arent exactly 1:1. Any idea how to skin this cat?

Here is one idea.

First plot the time vs distance. These lines would have different slopes between the measurement periods and you would essentially estimate the speed between measurements. Here is a graph with one measurement window:

Since they're racing on the same track, the total distance driven should be approximately equal. You might need to stretch or shrink the height of lines to make sure they travel the same distance. (This might not be a good assumption for your case.)
Now, go from the y-axis and for regular distance intervals, look up the slope of the graph.
Then plot the observed estimated speed for each distance period. You'll end up with speed measurements at regular distances which will now have the same length of samples.

This way you'll have a fairly good approximation of the speed at which racers traveled at specific parts of the track. I.e. you can compare cornering speed etc.


Supervised learning for a turn-based game? So I have 4GB of turn-by-turn data for many games of a particular strategy game. It appears that most people interested in using ML to build an AI for turn-based games use reinforcement learning to build a model on the fly.

Since I already have really good data, can I use supervised learning to solve this task?

EDIT: I was considering using regression to assign a score to a given action based on its likelihood of eventually resulting in a win; is this the right way to think about it?

Maybe the correct way of addressing this is by making sub optimizations of every step, even though it could be done by regression, I would suggest decision trees.

You have and advantage: A game is made of discrete steps, so in every moment you can "stop" and decide the best move based on your (possibly comprehensive) history of moves.

Supervised learning vs reinforcement learning for a simple self driving rc car


Questions on "Active Archive" In the big data world there is lot of talk about implementing an "active archive". I see cloudera talk about it a lot.

The value prop is that you move the low value (and less used) data from EDW to Hadoop and then save on expensive EDW storage by using Hadoop.

So in my company say we keep 10 years of data on EDW and say we don't use anything below 2 years very actively. So I move 8 years of data from EDW to Hadoop.

I can setup an Impala (or equivalent) product to query the 8 years of data as well.

But the problem is how do you order, sort a query which requires some data form EDW and some data from Impala?

because this kind of grouping, sorting ordering etc will have to be done in memory and the apps which queried the EDW were not written for operations and don't have the capacity as well to sort, group and process so much of data.

So how are people implementing the Active Archive?

I think the idea is that you have all data in HDFS, and query it with Impala, and also keep some small amount of data in your data warehouse. That is, keep all 10 years in Hadoop, and also 2 years in an EDW. The cost of also having the 2 years in Hadoop is small.


How to deal with possible data leakage in time series data? I have historical consumer data who have taken out a loan at some point in time. The task is to predict if a consumer will default when requesting a loan.

My issue is that for some customer in the data set, historical transactions are only available after the loan was issued. I believe using data after the loan event for prediction will cause data leakage.

This is a subtle leakage because it does not involve using information not available at prediction time. My concern is more about behavioral change when the customer is indebted that create a shift in the underlying distribution.

To test my hypothesis I was wondering if comparing whether the two samples before and after the loan is issued come from the same distribution will be a good approach.

These are my questions:

Is there really data leakage in the scenario I described
If yes, can I test it in any way?
Can a two samples test provide an answer? Which one? Note that the sample is composed of multivariate data
Can I do testing using any machine learning approach? I was thinking of using a Mixture Model to test for instance.

Any suggestion on how to best deal with this problem other than what I suggested will be appreciated.

Thanks

There is no need for sample tests.

A customer may have received many loans 1 to n - 1. To predict the default rate of nth request at time t(n), you are allowed to use any information up until t(n). When a user has no transaction history before t(1) system cannot predict the default rate for her; except maybe based on her age, income, etc. However, for the next loan request at t(2) system can use the transactions between t(1) and t(2), but still cannot use any transaction that happened after t(2). For any particular prediction at t(n), events happened after t(n) must never be used.

Regarding "it does not involve using information not available at prediction time", it does involve using information not available at prediction time t(n), since system is trying to utilize transactions that occur after t(n).


Difference between from nltk import word_tokenize and from nltk.tokenize import word_tokenize? What is the difference between the word_tokenize, one imported directly from nltk and the other being imported from tokenize package of nltk?

So basically
from nltk import word_tokenize
and
from nltk.tokenize import word_tokenize
from nltk import word_tokenize is basing to the same tokenizeAPI as both are the same with the usage and I don't see any difference in the results which they give.