grid search - optimal weighting of classifiers I am using three different of the shelf classifiers. It's a three class classification task.
I want to calculate the optimal weights (c1weight, c2weight, c3weight) for each classifier (real task more classifiers and also weights for each class).
Maybe simple grid search approach or sklearn ensemble classifier could do that.
vc = VotingClassifier(estimators=[('gbc',GradientBoostingClassifier()),
                   ('rf',RandomForestClassifier()),('svc',SVC(probability=True))],
                   voting='soft',n_jobs=-1)

params = {'weights':[[1,2,3],[2,1,3],[3,2,1]]}
grid_Search = GridSearchCV(param_grid = params, estimator=vc)
grid_Search.fit(X_new,y)
print(grid_Search.best_Score_)

I don't understand how to implement this for the following code.
def get_classification(text, c1weight, c2weight, c3weight):

  prediction1 = classifier1.predict(text)
  if prediction1 = 1:
    class1 =+ 1 * c1weight
  elif prediction1 = 2:
    class2 =+ 1  * c1weight
  else:
    class3 =+ 1  * c1weight

  prediction2 = classifier2.predict(text)
  if prediction2 = 1:
    class1 =+ 1 * c2weight
  elif prediction2 = 2:
    class2 =+ 1  * c2weight
  else:
    class3 =+ 1  * c2weight

  prediction3 = classifier3.predict(text)
  if prediction3 = 1:
    class1 =+ 1 * c3weight
  elif prediction3 = 2:
    class2 =+ 1  * c3weight
  else:
    class3 =+ 1  * c3weight

  if class1 > class2 and class1 > class3:
    return ("class1",class1)
  elif class2 > class1 and class2 > class3:
    return ("class2",class2)
  else:
    return("class3",class3)

c1weight = 0.5
c2weight = 0.7
c3weight = 0.4

for i, row in df_raw.iterrows():
    classification = get_classification(df_raw.at[i, 'text'],c1weight, c2weight, c3weight)
    df_raw[i,'classification'] = classification

score = get_accuracy(df_raw['classification'],df_raw['label'])

SOLVED: This sample code helped me to understand it
def your_function(number):
    print(number)

from sklearn.model_selection import ParameterGrid
param_grid = {'param1': [1, 2, 3]}

grid = ParameterGrid(param_grid)

for params in grid:
    your_function(params['param1'])

GridSearch finds those optimals weights for you.

You can access these weights through the attribute best_params_ of the GridSearch object, which will return all the optimal parameters (including the weights):

optimal_weights = grid_Search.best_params_


Is this enough information to fully define a neural network? Background:

A recent paper describes using machine learning to predict the outcome of chemical reactions. In the supplementary information (see pages S77-S78), the neural network used is described.

I would like to try implementing this network as a first project for learning neural networks and TensorFlow. However, I don't know enough to tell whether the description in the supplementary information is complete.

Question:

Does the supplementary information (see pages S77-S78 only) provide enough information to implement the network? Or did the author leave a lot out (perhaps on purpose)?

Description of the data for non-chemists:

The paper considers one specific type of chemical reaction. For this reaction, you have to combine five types of species: molecule 1, molecule 2, a catalyst, a base, and a solvent to dissolve everything. For each species, there are multiple options. You could use a water-based or an alcohol-based solvent, for example.

There are thousands of possible combinations for this chemical reaction. We have data that pairs each combination with the reaction yield, a number between 0 and 1. We have a bunch of vectors, basically, and we want to predict the yields of new combinations.

Reference:

The recent paper.

Yes.

The neural network mentioned in this paper is simple and straightforward: it is a fully connected network with 2 hidden layers.
 The authors specified the network structure, the number of neurons in each layer, and the activation functions in each layer and the final loss function.

Given those information, the network is fully specified. For instance, you can create such a network using

sklearn.neural_network.MLPRegressor(hidden_layer_sizes=(50,7), activation=‘logistic’)


Finding a proper method for an online driving style classification using acceleration data I am using a smartphone in my car to gather acceleration data (both longitudinal and lateral). Now, I want to classify my data in real-time based on the acceleration force applied through accelerating, breaking and maybe turning.

Is there a proper method to do this kind of online classification?
And also how does one determine whether a "harsh/aggressive" or just "normal" driving style?

Supervised vs. Unsupervised Learning

You will first need to decide whether to approach this as a supervised or unsupervised learning problem.

For supervised learning, you would need to take some portion of the data and hand score it as harsh/aggressive and normal.  I suggest you not go this route to start.  But if you do, I suggest using a support vector classifier for your initial testing.

I suggest approaching this as an unsupervised learning problem. Assuming you tend to be a normal/safe driver you can view this behavior as normal and harsh/aggressive driving as an anomaly or an outlier.  This is known as outlier detection.

Feature Engineering

Feature engineering will play a significant roll in this problem. Despite your attention to the longitudinal and lateral axes of the car, the phone will have arbitrary orientation within the car i.e. it will just be in your pocket.  Thus its probably best to factor out the direction somewhat.

First, I would find the length of the acceleration vector (magnitude of acceleration without regard to its directional orientation).  Since you have three orthogonal axes, you can just use:

$$||\mathbf{a}||=\mathbf{a}_x^2+\mathbf{a}_y^2+\mathbf{a}_z^2=a$$.

Additionally, you can also integrate the acceleration vector over time and find its magnitude to find the vehicle speed:

$$speed=||\mathbf{v}||=\mathbf{v}_x^2+\mathbf{v}_y^2+\mathbf{v}_z^2=v$$

where $$\mathbf{v}(t)=\int_{start-of-trip}^t\mathbf{a}(t)$$

The speed is very useful in determining the difference between different modes of acceleration e.g. driving around in a circle will yield constant acceleration, but also constant speed vs. holding down the gas pedal and driving straight, which could yield constant acceleration with changing speed.

An important aspect of this method lies in the fact that integrals of your data will tend to damp noise, where as derivatives (differences) will tend to amplify noise.  Don't take derivatives of your data.

Outlier Detection

At this point you can normalize the speed and acceleration-magnitude and find the mean and standard deviation of both values.  When your normalized speed and acceleration-magnitude lies outside of some z-value, say two standard deviations, then you can classify your driving as dangerous.

Don't forget to think of speed and acceleration as orthogonal values and form a circular function out of the respective z-values for proper outlier detection.  You can read about 2D or nD outlier detection and also here

Remember that the equation for a circle is:

$$r^2=a^2+v^2$$

We can therefore solve for $r$ and set it equal to $2$ standard deviations.

As an aside, note that since variance is just the square of the standard deviation, you could alternatively think of this in terms of variance.

Now for two standard deviations, the driving is aggressive if:

$$\sqrt{\big(\frac{v-\mu_v}{\sigma_v}\big)^2+\big(\frac{a-\mu_a}{\sigma_a}\big)^2} \gt2$$

The online learning component simply requires that you continuously update the means and standard deviations.  Though, I suggest that this be done in some sort of batch mode rather than continuously.

Hope this helps!


Understanding Training and Test Loss Plots I have attached a figure that contains 6 subplots below. Each shows training and test loss over multiple epochs. Just by looking at each graph, how can I see which one is the best? Which ones are overfitting or underfitting. Which ones are getting worse with each epoch?

Assuming that the train and validation sets in the curves under comparison are the same, the best curve is probably the one with the lowest validation loss value.

Numbering your figures from left to right and from top to bottom, I would say the best one is #5 (second row, second column).

Now, let's break down what is going on in each plot:

Very high values, seemingly random, no decrease whatsoever in either train or validation losses: the model is not learning; probably there's something wrong with either the model or the optimization process, or maybe some hyperparameter value is terribly wrong.
Descending values for both training and validation losses, with validation loss having a gap with the training one, and both stabilized (i.e. neither of them will probably go any lower -if in doubt about this, leave them more training time-): training seems Ok, but there is room for improvement if you regularize your model so that you get your training curve upper and the validation one lower.
Initially, both curves descend, then validation starts going up around step 800: overfitting. You should try regularizing the model and, if that's not effective, use early stopping to use the model that performs best on the validation data. You may also try with some hyperparameter tunning, or having a learning rate schedule that makes it smaller with time.
Both curves are descending and it seems they will keep doing that for a while: training is not finished, leave it training more time.
Both curves descend, despite the initial plateau, and reach a low point, with no gap between training and validation curves: you can probably improve the model weight initialization. Anyway, this plot seems the best, as the validation curve reaches the lowest value and there is no overfitting.
Both curves go up: there is something wrong, probably in how you define your loss function optimization process.

I don't see any clear case of underfitting among your plots. In an underfitting scenario, we would see that the model learns something but both the training and validation losses stabilize at too high values. This would suggest a lack of capacity of the model, preventing it to properly capture the data distribution with respect to the labels.


Machine Learning Perceptron Algorithm I'm studying Machine Learning using Sebastian Raschka's book.

Wonder if someone could please help me to confirm if I have the following steps correct if I apply Perceptron Algorithm to Iris dataset as shown in the table below.
What do I set $\eta$ to?
What do I set $θ$ to? I randomly set it to 2.

Perceptron rule - summarized by the following steps:

Initialize the weights to 0 or small random numbers.
For each training sample $x(i)$ perform the following steps:

Compute the output value $\hat{y}$ .
Update the weights.

\begin{array}{c|c|c|}
 & \text{sepal_length, $X_1$ } & \text{sepal_width, $X_2$} & \text{petal_length, $X_3$} & \text{petal_width, $X_4$} & \text{species, $Y$}\\ \hline
\text{Row 0} & 5 .1 & 3.5 &1.4 &0.2 & setosa \\ \hline
\text{Row 1} & 4.9 & 3.0 & 1.4 & 0.2 & setosa \\\hline
\text{Row 2} & 4.7 & 3.0 & 1.4 & 0.2 & setosa \\\hline
\text{Row 3} & 4.6 & 3.1 & 1.5 & 0.2 & setosa \\\hline
\text{Row 4} & 5.0 & 3.6 & 1.4 & 0.2 & setosa \\\hline
\end{array}

set: $θ = 2$

set: $virginica = -1,$ $\>$ $setosa = 1$

let's say if $z \ge  θ,$ $\>$ $θ(z) = 1$,  $\>$ $θ(z) = -1$ if otherwise

Initialize weight vector: $W$ = [ $w_1$ = 0.1, $w_2$ = 0.2, $w_3$ = 0.3, $w_4$ = 0.4]

For observation row 0:

$z = w_1*x_1 + w_2*x_2 + w_3*x_3 + w_4*x_4$
$= (0.1* 5.1) + (0.2*3.5) + (0.3*1.4) + (0.4*0.2)$
$= 1.71$

Since $z = 1.71 \lt θ = 2$, $\>$ then $θ(z) = -1$

As for row 0, the perceptron algorithm incorrectly predict $virginica = -1,$, hence the weights would be adjusted.

$w_j:= w_j + \Delta w_j$

$\Delta w_j = \eta(y^{(i)} - \hat{y}^{(i)}) x_j^{(i)}$

Hence,

$\Delta w_1 = \eta(1-(-1))5.1 = 10.2\eta$

$\Delta w_2 = \eta(1-(-1))3.5 = 7.0\eta$

$\Delta w_3 = \eta(1-(-1))1.4 = 2.8\eta$

$\Delta w_3 = \eta(1-(-1))0.2= 0.4\eta$

Update weight vector: $W$ = [ $w_1 = 0.1 + 10.2\eta$, $w_2 = 0.2 + 7.0\eta$, $w_3 = 0.3 + 2.8\eta$, $w_4 = 0.4 +0.4\eta$]

Using the updated weight to Row 1 and repeat the process above.

OK WHAT I understood from your understanding is below:-

1.What do I set η to?
Answer:**It is learning rate. As usual everyone uses **η=0.01. Learning rate is crucial because if you set bigger values when updating weights the it will reach to a local minima and never reach to global minima. Now if you set it low then it will gradually went to global minima. This iris dataset problem is so trivial that it takes no time to reach global minima but for a complex dataset you will need another factor such as decay rate. After a while, when training you will find the accuracy isn't increasing then the neural net suppose that it reached global minima now if you set decay rate to learning rate like 0.01* decay rate then learning rate will be much less and then it will TRULY converges to global minima as the gradient will be minimized in a small amount.

2.Z is the label for your dataset.You should have used one hot encoding. I think rather than writting everything here i would suggest following links.
Please go to there and then pick up a paper and pen.do the math.Bang.

How Neural Network learns

And obviously check 3Blue1Brown youtube channel's playlist on Neural Net.

Welcome to Deep Learning.Best of luck.


depmixS4 - How to fit HMM usgin multiple sequences of observation Is there any R package that supports fitting an HMM using multiple sequences of observations? to the best of my knowledge depmixS4 does not support this feature

No, depmixS4 supports multiple external variables to be included to forecast underlying time series.   In this case transition matrix is a function of all the other external variables. It is given in depmix S4 vignette also.-

a useful material for start -
https://machinelearningstories.blogspot.com/2017/02/hidden-markov-model-session-1.html &
http://machinelearningstories.blogspot.in/2017_03_01_archive.html

R Code snippets-

Required library

library(depmixS4)

data loading-

physician_prescrition_data <-c(12,16,45,45,56,67,78,98,120,124,156)

model execution-

HMM_model <- depmixS4::depmix(physician_prescrition_data~1, nstates = 2,ntimes=length(physician_prescrition_data))

model fitting

HMM_fm <- fit(HMM_model)

Transition probabilties-

HMM_fm@transition

posterior states-

posterior(HMM_fm)
plot(ts(posterior(HMM_fm)[,1]))

Emission probabilties-

HMM_fm@response


The effect of imbalanced distribution of data I read on Google's ML website if I have classification dataset with a ratio of 90% for one classification and 10% of the data for another classification.

In that case, should I use the exact same percentage of data for each classification?
i.e. deleting around 80% of the dataset to make it 10% for each classification.

The reason is that Google said that the ML model will learn and then it is more likely to have a classification of the 90% and that won't provide good predictions. (i.e) The predictions might be biased towards a single label/feature.

My dataset is 90% to 10% but that is indeed the actual ratio and it is more likely to have the classification of the 90%

Shall I delete 80% of my data or keep it as is and let the ML learn that it is indeed more likely to have a classification of the 90%?

The best way forward here depends highly on the real life question you try to answer.

Let's say you want to make a medical diagnosis:

'Sick with exotic Illness X' or 'Not sick with exotic illness X' in this case you might want to catch all instances of being sick as a warning sign and could live with 'false positives'.

Conversely your algorithm will be used to predict 'customers likely to cancel soon', in this case it would not be a good idea to proactively talk to 'false positives' i.e. customers who did not plan to cancel about why they might be dissatisfied.

In either cases your training set and indeed reality might be severely unbalanced but the cost and consequences of this varies.

In the first case I would recommend using balancing methods (like the aforementioned Under-/Oversampling, etc.) to improve recognition of the minority class while in the second case that might be unnecessary.

In any case I would practically go on to do the following:

Include balancing/sampling in your beauty contest of algorithms and parameters and check the impact on the accuracy of predicting the test set (which is left unbalanced as in the original).

This will simply show you whether the inherent bias of the training set is problematic for your real world case (i.e. produces models that never identify the minority class) or not.


Pros and cons of pandas or R for longitudinal data? Note: I believe this question is not off-topic because it meets all of the criteria for subjective questions that are allowed. I would be happy to rephrase or clarify if others disagree

I'm about to begin a two-year project which will predominantly involve longitudinal panel data.
I've found numerous questions and answers (Pros and Cons of Python and R for Data Science), and blog posts (https://www.quora.com/Which-is-better-for-data-analysis-R-or-Python | http://www.kdnuggets.com/2015/05/r-vs-python-data-science.html), about the relative merits of R and Python+pandas for data science but nothing that discusses longitudinal data.

My question is therefore:

Which environment (Python+pandas or R) would you use for longitudinal data analysis, and why?

For example, I would love to see answers that:

Tell me how you have used one or both environments to solve a particular problem with longitudinal data
Which environment or package(s) you found easier to work with, and why
If you used a notebook such as beaker to work with multiple environments simultaneously.
If so, which environment did you use for which step(s) in the data analysis pipeline, and why?
If so, did this confer advantages over just using one language: would you use such an approach again?

I am not asking which one is best (we all know that such questions are never constructive).

I am more familiar with one environment than the other, but I'm not averse to learning new skills (and both share similar syntax anyway), so I'm not going to say which one: I want answers based on your experience, not my abilities.

Let me answer this from the point of data in general. I understand you want the answer for longitudinal data but the reason I don't want to specifically answer that is because pretty much all these frameworks treat data in similar ways.

What you should be looking for?

1. How proficient are you in coding?

If you code well enough and you can tackle most problems by coding the I suggest Python + Pandas. The reason is simple since Python is a programming language you can use the language outside just the normal ecosystem of running your task. So if you want to integrate other components such as maybe a web server, store in a database etc. Python will be most useful.
            where as if you are not comfortable with coding then I suggest R is the way to go since it is much more easy to learn but the disadvantage comes when you are trying to do things other than just perform your daily data analysis. It might be limiting

2. In terms of efficiency

Now pretty much all these tools work pretty fast and the differences are hardly noticeable. Usually, you would have a better edge with R in just the way computations are performed. It is a bit more optimized and it utilizes CPU cores to perform a tad bit faster computation than python does. However these days with the heavy intensive RAM etc it is not really that big of a difference. Also, most libraries for heavy matrix calculations are present in both these toolkits. So not really much difference.

In my opinion

I would go for Python just because of the diverse uses it has. Also has a great supporting environment for libraries etc.


Symlink or Rename files with spaces? Premise:
I often get files from colleagues that I need to work on. Often times, these files have spaces in the names. Working with these files at the command line or in scripts can be tedious.

Possible solutions:
With the rename program (on nix systems), I can easily rename these files, eg:

$ rename --sanitize --lower-case *

I recently found that rename can actually just create a link to the original file, leaving the original filename unchanged:

$ rename --sanitize --lower-case --symlink *

Question:
What are the (potential) pros/cons of these two methods?

For example, it seems that creating a bunch of symlinks has the drawback of making my filesystem 'messier', but renaming the files has the draw back of trying to match up the files that I'm using with my colleagues (whether I'm redistributing my code or just communicating "I did [analysis] on [file_x.csv]").

Additional info:

Generally I'm the only person actively working on these files, but it is important that my work be archived so that other people can refer back to it or re-analyze the data in any way of their choosing. I work in an academic setting, so, in principle, the raw data and my methods should be archived indefinitely.

I understand your pain!

Can you not simply use tools to read the filenames and filter those as required, thereby letting your development language do all the escaping etc. for you?

In Python, for example, that'd mean using something like: os.walk() or os.listdir().


Finding out which values lead Random tree to a decision I have a dataset of machines that produce plastic parts. A camera evaluates whether a plastic part was produced correctly or not (binary classification). I'm trying to figure out which factors influence a part being wrongly produced. E.g. I have different temperature values of the machine parts during the production.

I'm using a Random Forest to classify the data. The test dataset is being recognized quite well. The next step is to figure out which values lead to a wrongly produced part (e.g. when temperature > 150K: Part is broken). I've searched the internet but I couldn't find any information about this.

At the moment I'm trying a brute force method where I simply generate a test dataset where I go through different value ranges. But so far everything is classified as wrongly produced part.

Are there other methods I can use to get the values?

Thank you!

If your dataset was separable into a series of neat decisions, then a classification and regression tree (CART), would give you the type of solution you're looking for. The price you pay for random forest is that by reducing the variance through generating many random trees, you're also reducing the interpretability of the model significantly. Solutions that can give you local solutions are LIME or you can calculate SHAP values or feature importances, but that gives you importances in the context of the model and may not be useful for the type of decision you're looking to make.


How Does Cross-Entropy Work With Softmax Activation Function? I found online that the derivative of a cross-entropy activation function with a softmax activation is (output - expected), which had me very confused. If for example, the expected value is 1, and since the output is always less than 1 (softmax), doesn't it mean that the derivative is negative, and that if we train our weights with this gradient, the weights will decrease, and so the output will decrease in the next iteration, thus increasing the error? How does it work?

As you mentioned, (output-expected)
see, this value will be less than 0 (negative as output<1 and expected=1)
Now w_new = w_old - learning_rate*derivative
now based on derivative, w_new can be greater or lesser than w_old.

I guess when you are assuming with negative derivative, weights decrease is not correct.
Also even if weights decrease the output will decrease is also not correct.


Performing Binary classification using binary dataset Can a dataset with only  binary digits 0's and 1's perform good classification such as decision tree and naive Bayes

Yes. Many data sets are one-hot encoded into only 0s and 1s because some ML algorithms work best that way.

http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html


Why do we pick random features in random forest I understand that random forest is a stylized version of bagging of trees. We choose randomly data points as well as random features for constructing random forest.

But if we use just plain version of bagging by choosing only data points randomly then we have trees which have trained on more number of features unlike the random forest in the stylized version. Since learning with more features every individual tree has more information about the data points and so are more 'intelligent' in some sense than the individual trees in the random forest.

So why does the random forest using stylized version of bagging performs better than the random forest using plain implementation of bagging?

I understand that the random forest using the stylized version gives a model lower variance but since each of its trees is trained on some of the features shouldn't it make the model a bit  high biased?

The idea of random forests is basically to build many decision trees (or other weak learners) that are decorrelated, so that their average is less prone to overfitting (reducing the variance). One way is subsampling of the training set. The reason why subsampling features can further decorrelate trees is, that if there are few dominating features, these features will be selected in many trees even for different subsamples, making the trees in the forest similar (correlated) again.

The lower the number of sampled features, the higher the decorrelation effect. On the other hand, the bias of a random forest is the same as the bias of any of the sampled trees (see for example Elements of Statistical Learning), but the randomization of random forests restrict the model, so that the bias is usually higher than a fully-grown (unpruned) tree. You are correct in that you can expect a higher bias if you sample fewer features. So, "feature bagging" really gives you a classical trade-off in bias and variance.


Modeling maps using trip data I am working on a problem where I need to identify the most likely used routes for a number of trips. The trips have their own starting and ending locations, along with distances covered and durations of these trips. I am not sure how to use the underlying road maps. Is indirectly modeling the route segments from these attributes a good idea? Any suggestion on how could I implement and test it? Some alternatives?

It depends on the application and the Data.Could you provide more details about the data of the trips? Do you have GPS-positions for the trips?

There are Libraries such as Barefoot which support you in map-matching, meaning that it maps your positions to the underlying map, in most cases OpenStreetMap. This library uses Hidden Markov Models for online and offline Map Matching. Maybe this could help you as well, when implementing your own model.


Can natural language generation algorithms generate valid words too? Natural Language Generation (NLG) is the natural language
  processing task of generating natural language from a machine
  representation system such as a knowledge base or a logical form. —
  Wikipedia

Is NLG about building meaningful sentences, reports, etc.? Can NLG build valid dictionary words as well? For example, without consulting/reading from an English (or any language) dictionary, can an algorithm generate such words?

Outside of context of NLG (thus not a direct answer to your whole question, but an answer to your question's title): Generating words from a character-level model has been done using RNNs exposed to large corpora of text, such as Wikipedia content, and trained to predict text character-by-character.

Used to generate content, the model is normally fed a few starting characters and asked to predict the next one. A choice is made from its most-likely predictions and fed back to it to continue the sequence.

Here is a blog showing some examples trained on some Shakespear and Wikipedia.

Such a network can and does generate nonsense words, although they are often fitting and might read like e.g. a noun or verb as you could expect depending on context. The sentence structure and grammar can come out sort of right, but the semantic content is usually complete gibberish.


How can I group texts with similar content together? I need to find a solution to group a corpus of texts according to document similarity. Premising I have no experience in ML - only a few readings - I'd like to know if calculating the tf-idf on each text is the right approach. I've read something about calculating and compare the cosine similarity on tf-idf, but I don't know how to further process the results.

Saying "almost equal" means "the order of the words matters", it' not just the same words in different order.

I was thinking about using MLlib from Apache Spark to do the job (or at least use a Scala library).

Can anybody head me to the right direction, or even better add a link to a tutorial to this page?

Well, after further googling I found the solution: MinHash or SimHash will do the job and I also found a tool implementing MinHash written in Scala on GitHub right at this link


How to get the number of steps until a certain accuracy in keras? I want to see how many steps does it take for my model to reach a certain accuracy.Say 90 percent on cifar10.How can I get this info from the keras model ?
EDIT:
accuracy in each epoch is accessible in history object fit() returns,but im looking for accuracy in each step
Solution:
I made a callback object that keeps loss in each step
import pickle
from tensorflow.keras.callbacks import Callback

class LossHistory(Callback):
    def __init__(self,path='',name=''):
        super(Callback, self).__init__()
        self.path = path
        self.name=name
        self.accuracy = []
        self.losses=[]

    def on_batch_end(self, batch, logs={}):
        self.accuracy.append(logs.get('accuracy'))
        self.losses.append(logs.get('loss'))

        history_={}
        history_['accuracy']=self.accuracy
        history_['loss']=self.losses

        with open(self.path+self.name+'_history.pkl', 'wb') as f:
            pickle.dump(history_,f)

When you train the model using the .fit() call, it actually returns an object called history. Train the model using history = model.fit(x_train, y_train...) instead and then once training is complete, you can access the history.history dictionary to see this kind of info.

Refer to https://machinelearningmastery.com/display-deep-learning-model-training-history-in-keras/ for more information


Question mark on Correlation Matrix with RapidMiner I'm using RapidMiner to evaluate correlation between attribute in my dataset. The problem is that a lot of values appear with '?'.

Someone can help me?

This is a sample of data

All your data is non-numeric, so there is no straightforward method to compute a correlation value. It appears that the software does compute a correlation value between 2-valued fields (breast and irradiat).

You will have to work on your data if you want to compute the full correlation matrix:

Make fields with numeric values numeric. For instance, the 6th field seems to be numeric data encoded as strings (make sure those are not numbered categories that cannot be ordered, otherwise the correlation will mean nothing)
Give numeric values to ordered categories. For instance, for the age field, you could certainly use a centered numeric value for each category (45 instead of 40-49, 55 instead of 50-59, etc.), or just a rank.
Ignore rows with no data. For instance, in the 5th field, you have binary results (yes/no) and appear to have a few question marks. Removing the lines with question marks will make the field 2-valued, something that appears to allow the software to compute the correlation.

For the rest of the fields, there is no magic trick, unfortunately. You may want to one-hot-encode them, but interpreting the results will be more complicated. There are also ways to test categorical fields independence, but that's a different measure: see this post.


Is there any reason not to use classification instead of regression when looking for ranges? In the case I want to predict only ranges from a continuous value, is there any reason to use regression instead of classification ? Could it depend on the type of model I am using (neural network, decision tree, bayesian, ...) ?
Example
Let say I have a dataset with images. Each image has one human on it and is labeled with his/her height. Now I am only interested in predicting height ranges, for instance these four classes [ A, B, C, D ] = [ <150, 150-170, 170-190, >190 ] (in cm). Is there any reason why one of the two following approaches should lead to better performances ?

case 1: using regression - First create and fit a model that predicts the exact height from an image, then simply gives its associated height range.
case 2: using classification - First label all the images with the wanted ranges (=classes), then create and fit a classifier to predict this height range.

Note: I am wondering if there is a general answer to this question, not only to this example
EDIT
As @n1tk pointed out, in the post Performance of CNN based deep models with number of classes, the question is answered if we think about increasing the number of classes. In my question, I am wondering about regression vs classification. So try to fit a continuous value vs ranges from this value.

The general answer is how the model will be used deal. Either way may be optimal for the case.
For example - If the model groups applicants into good credit risk and bad credit risk, that might be fine to say model score > x = good risk and model score <= x = bad risk. But maybe there will be differential action based on the model score - like giving a different interest rate or a bigger loan.
In the original example, in regression actual = 191, predicted = 189 you can calculate the loss.
In classification, if actual = 191 and P(>190)=0.35, P(170-190)=0.40, P(150-170)=0.25, then you just know the wrong class. Is that enough for the model usage?
There is also the assumption that a "closer" class will be chosen but that might not be true, e.g actual=191, P(>190)=0.25, P(170-190)=0.25, P(150-170)=0.5. The regression could come up with 160 also but you can measure that loss if the model usage requires it. Many classification algorithms do not know if classes are "close" - Confusion matrix. "How close I am to the diagonal?". Is there such metric?
You can also look at Ordinal Regression https://en.wikipedia.org/wiki/Ordinal_regression. In this case there is an implicit ranking in the "class".
Choose based on how the model will be used. Always important to know the usage and the problem being solved, then work backwards to the model.
Hope that helps.


How do I convert an L2 norm to a probability? I am using the dot product as a way to measure the similarly of two facial-model vectors extracted by a ML algorithm (OpenFace in fact).

I would like to convert the L2 norm to a probability U[0,1] in order to compare with other solutions from other providers that map directly to probabilities but in a blackbox fashion.

By probability I mean the probability that the two vectors represent the same person. So an identical image would give a probability of 1; images of the same person would give a high probability; and orthogonal images (different people, or some suitable toy case) would in the limit have such a probability of zero.

I'd like a uniform distribution for comparison with other providers, but this boils down to knowing what the underlying probability distribution of the feature vectors is.

How do I do this?

Rayleigh distribution CDF
Cosine similarity
(1) is the same as (2)
Draw from a simulated distribution
Arbitrarily scale to U[0,1]
Train into the comparison probability space directly (skipping the feature-vector step)
None of the above
Something else?

Some useful links are:

Cosine similarity versus dot product as distance metrics

http://blog.christianperone.com/2013/09/machine-learning-cosine-similarity-for-vector-space-models-part-iii

http://houseofjeff.com/2015/05/10/using-probability-distribution-and-cosine-similarity-to-automatically-detect-data-contents - this one is almost there?

I guess you either need to train directly into probability space, or train another model on top of the feature vectors to learn the probability of two vectors belonging to the same person.

All the other metrics you could use on top of the feature vectors are a bit arbitrary, as they were not considered by the cost function used when training the original model.


Is it possible to predict groups vs groups in machine learning? I have a problem that I think machine learning can solve. I'm new to it however and don't know where to start so any pointers to appropriate solving methods/guides etc for the problem are appreciated.
I've searched a bit and couldn't find any questions similar to this, but most likely because I dont know exactly what to search for:

Lets say we have a bunch of fruits: a,b,c,d,e,f which all have two factors associated with it, a price and a quality. Every day in a store the customers can choose between two bundles(g1,g2), each consisting of two fruits with varying price and quality(p,q). The 4 fruits every day must be unique, so g1 cannot have a fruit in g2.
The customers then buy one bundle each. They value the price and quality, but are probably also influenced by outside factors such as what fruit it is and what it is combined with.

Is it possible to create a training set from this? I came up with two ways to represent this:
The simplest way i could think of: let a fruit x be represented as x:{x_p,x_q}, a group as g1(a,b), and a training element would then be defined as day(g1,g2) which we will use to train our machine.
The second way was to let the fruits contain a present flag to mark if they are in a group or not, x:{x_p,x_q,present} and a group would then become g1(a,b,c,d,e,f) and a day still day(g1,g2)

Is it possible to do it the first way without data for several fruits, or is it just implementation specific?
How would one train for the mirroring case of day(g1(a,b),g2(c,d)) vs day(g1(c,d),g2(a,b)), by just creating the mirrored training element or is something that the machine could figure out on its own?
While this is a minimum example, what would be different if we were to cover more fruits(5/bundle) from a larger fruit pool(20)?
Can we use it to predict which bundle is more likely to be picked even if that group/fruit combo hasn't been evaluated in the set?

Another way I thought of was training on the groups, but I think that would result in a flat score for the groups and most likely miss relationships between the groups, such as g1 is preferred over g2 which is preferred over g3 which is preferred over g1? Is this true or would a machine be able to understand that too?

I was looking a bit at the javascript libraries brain and mind and thought that it might be possible with one of them, but I dont know what type of algorithm/solving method I'm looking for. Any suggestions for libraries to solve this in, preferrably python/php/java/javascript are appreciated!

Also if someone could add suitable tags to the question would be great!

EDIT: same as with the group mirroring, how do we distinguish the groups, do we need to sort the fruits alphabetically or can a machine handle it too? (apple, banana) should be identical to (banana,apple)

I think, you need to look carefully through popular ranking algorithms. I would suggest to start from Elo, at a glance, it could help you to solve the problem. I worked only with "classical" "1-vs.-1" version. But here are some thoughts on your case.

Consider every bundle as a team of two players; every customer's choice as a duel. The main issue here would be the "teams randomness", as you have no constant team compositions. But you can try to solve it, using average ratings of team members. With the large number of duels and "teams" you will get enough data to predict the outcome for brand new mixtures of fruits.


Is using unsupervised learning to setup supervised classification reasonable? I've got a biological dataset describing genes. The overall idea is that there are thousands of these genes to sort through, so if ML can rank them I can then know which should be going into the lab for functional research first. Currently, I make labels for supervised classification of these genes based on their known biology (so for example some genes interact with drugs related to a disease so I label them as 'most likely to cause the disease' and this goes down until I have a final 4th label of 'unlikely to cause the disease').  The way I make these labels seems impossible to not be biased, since I'm making all the decisions, so I'm wondering if I can compare my decisions with seeing how an unsupervised model would group the data (e.g. I've got 4 labels but if the model finds 5 groups then it shows how far off I am potentially?).

Would it even also be possible to use unsupervised learning to create the labels by itself or would this too be unreliable as you can't know why it's grouping certain genes together? Or would doing this step  alone actually make the supervised step redundant anyway?

Is using unsupervised learning to setup supervised classification reasonable?

Absolutely. This is a common strategy in ML. As you said yourself, using information coming from the data itself has the benefit of being less biased.

Would it even also be possible to use unsupervised learning to create the labels?

Technically yes. Though, some clustering techniques require you to specify the number of clusters, which isn't helpful. As you said, if you can cluster data points in a satisfactory manner, you don't need supervised learning anymore. Also, indeed, if your scenario requires you to have an understanding of what differentiates the clusters, you may not be lucky depending on which clusters come out. They are not always interpretable.

What I would suggest would be to turn your classification problem into a regression problem. 1.0 could be most likely to cause the disease, 0.0, least likely. This way, you don't have to worry about how many labels you need in the first place.


Why imbalanced data-set will bias the prediction model towards the more common class? As we know, an imbalanced data-set has a disadvantage of training a model for deep learning. However, I don't know how to explain it with mathematics?

Let us say you have a skewed class (which is linearly inseparable, otherwise the prediction is bound to be correct always) like this:

Image Credit: Learning from Imbalanced Classes

Now you can clearly see, if you move the boundary downwards you will be miss-classifying much more blue dots than red dots. And we know the cost function of a ML algorithm roughly tries to reduce the miss-classification (by converting it into a real valued loss optimisation problem, instead of discrete mis-classification reduction).

Think of it as democracy, each individual point tells the classifier what to do. The classifier will always listen to majority points. In this case the classifier will be miss-classifying much more blue points rather than red points if moved downwards. So the bias is set to high, so the line moves upwards.

Mathematically speaking: A classifier will always try to minimise its loss function, which roughly corresponds to making correct classifications (making correct classifications is discrete, optimising loss is not). So, it will position the boundary as such to make minimum number of mistakes disregarding class type. In general if a class has more points in the data-set, a classification boundary will tend to to make more miss-classifications of that class compared to class with less points. Hence optimising the class with higher points becomes more of a priority for th classifier, and it sets its bias accordingly.


0.1 accuracy on MNIST fashion dataset following official Tensorflow/Keras tutorial My goal is to classify products pictures into categories such as dress, sandals, etc.

I am using the MNIST fashion dataset, following this official tutorial word-per-word: https://www.tensorflow.org/tutorials/keras/basic_classification so my code is 100% identical to what can be read there:

# TensorFlow and tf.keras
import tensorflow as tf
from tensorflow import keras

# Helper libraries
import numpy as np
import matplotlib.pyplot as plt

print(tf.__version__)

fashion_mnist = keras.datasets.fashion_mnist

(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()

class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
               'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']
model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),
    keras.layers.Dense(128, activation=tf.nn.relu),
    keras.layers.Dense(10, activation=tf.nn.softmax)
])

model.compile(optimizer=tf.train.AdamOptimizer(),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(train_images, train_labels, epochs=5)

test_loss, test_acc = model.evaluate(test_images, test_labels)

print('Test accuracy:', test_acc)

Problem: The resulting accuracy is always around 0.1, much lower than the tutorial's example output of 0.876.

I am obviously doing something wrong, but I can't figure out what. How to improve accuracy to something reasonable?

My output:

$ python classify-products.py
1.10.1
Epoch 1/5
2018-09-18 13:33:46.971437: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
60000/60000 [==============================] - 3s 47us/step - loss: 13.0161 - acc: 0.1924
Epoch 2/5
60000/60000 [==============================] - 3s 46us/step - loss: 12.8998 - acc: 0.1997
Epoch 3/5
60000/60000 [==============================] - 3s 46us/step - loss: 13.3386 - acc: 0.1724
Epoch 4/5
60000/60000 [==============================] - 3s 47us/step - loss: 12.9031 - acc: 0.1995
Epoch 5/5
60000/60000 [==============================] - 3s 47us/step - loss: 13.6666 - acc: 0.1521
10000/10000 [==============================] - 0s 26us/step
('Test accuracy:', 0.1005)

Switching to 20 epochs does not improve accuracy.

You haven't normalized your image dataset such as setting the pixel values between 0-1 which could help classifier converge faster.

Please do it by doing the operation below.

train_images = train_images / 255.0

test_images = test_images / 255.0

It seems you are using 50% of data for training as well as testing.
try to use the data in the ratio of 7:3 for training and testing.
Increase the epochs hope it will work.


How to prevent model from recognizing false Classes Let's say that I have a model that can recognize Cats and Dogs. However, when I use a picture of a Cup or Human it generates a random prediction at 0.70 confidence. Should I use sigmoid instead of softmax?

Or is there any fix for preventing massive confidence on new class object?

I haven't actually played around with this problem, but here are a few resources:

AI.se question - Why not teach to a NN not only what is true, but
also what is not true?
How to Make a Neural Network say "Don't Know"? (maybe try
requesting fulltext through ResearchGate )
blog post - Making Your Neural Network Say “I Don’t Know” — Bayesian
NNs using Pyro and PyTorch

Edit: I just found the following questions here on DS.se:

Multi Class + Negative Class Image Classification Strategies
Fine-tuning a CNN for recognizing two classes, but also being able to tell if none of them is present in an image


Discarding non-english words in column I have some non-english words/sentences in my data.
I tokenized my text and tried using nltk.corpus.words.words() but its not really helpful as it also removes the brand names, company names, like NLTK etc. I need some solid solution for the purpose.
Here's what I tried:
def removeNonEnglishWordsFunct(x):
    words = set(nltk.corpus.words.words())
    filteredSentence = " ".join(w for w in nltk.wordpunct_tokenize(x) \
                                if w.lower() in words or not w.isalpha())
    return filteredSentence

string = "NLTK testing man Apple Confiz Burj Al Arab Copacabana Palace Ｗは比較的新しくてきれいなのですが Ｓｈｅｒａｔｏｎ hotelは時々 ＮＹらしい小さくて清潔感のない部屋"

res = removeNonEnglishWordsFunct(string)
Output: testing man Apple Al Palace

Expected output: NLTK testing man Apple Confiz Burj Al Arab Copacabana Palace Ｓｈｅｒａｔｏｎ hotel
```

To Tokenise, clean up symbols (i.e. Normalise), etc. just use one of the widely used NLP libraries, they should be able to do most of the work for you.
Examples include:

NTLK
Spacy
SparkNLP
.. and many more. Perhaps look up some articles comparing their strengths and weaknesses on Google to decide what's best with your project.

As for the detecting English words, that might be slightly trickier, but you can find answers to this already from a bit of Googling. E.g. https://intellipaat.com/community/5638/removing-non-english-words-from-text-using-python
Might also be worth posting some code, output examples and what you're intending to do down the line (e.g. training a neural network?) so that other's can provide further help.
All the best,
Kelvin


Exporting Correlation Matrix (from function) I'm interested in exporting a correlation matrix to csv. I've tried using the to.csv functionality, but my matrix is obtained from a function and I get an "object has no attribute 'to_csv'" error. I appreciate any guidance. Code is below.

import numpy as np
from matplotlib import pyplot as plt
from matplotlib import cm as cm

def correlation_matrix(df):

    fig = plt.figure()
    ax1 = fig.add_subplot(111)
    cmap = cm.get_cmap('jet', 30)
    cax = ax1.imshow(df.corr(), interpolation="nearest", cmap=cmap)
    ax1.grid(True)
    plt.title('Feature Correlation')
    labels = None
    # Add colorbar, and make sure to specify tick locations to match desired ticklabels.
    cbar = fig.colorbar(cax, ticks=[-1.1, -0.8, -0.6, -0.4, -0.2, 0, 0.2, 0.4, 0.6,.8,1])
    plt.show()

correlation_matrix(pd.DataFrame(np.c_[Xs,Y]))

Don't use the matrix obtained from your function rather,

It looks like correlation is a DataFrame too, so you can simply use to_csv:

correlation_df(within your function itself).to_csv("C:\destinationfolder\file.csv")


Learning rate of perception I don't understand the following statement:

  The choice of learning rate m does not matter (for Perceptron) because
  it just changes the scaling of w (weights). The site with this statement that wikipedia cites.

The update rule of perceptron is $w \pm x$ for mistaken cases. If the update to the weight vector is addition or subtraction, how can it only scale the weights?

After some study, I figured out the answer and want to share with people if someone also finds it helpful. The loss function of Perceptron is hinge loss or

$J(w) = max(0, -yw^Tx)$.

Adding a constant to the loss function does not change the function value, as it does not change the sign of the decision. In other words

$J_2(w) = max(0, -\alpha yw^Tx) = J(w)$.

If we do gradient descent using $J_2$, we have

$\partial(J_2)/\partial(w) = 0$,  if $J_2 = 0$;

$\partial(J_2)/\partial(w) = -\alpha yx$,  otherwise.

So the update function of gradient descent is

$w_{new} = w_{old} \pm \alpha x$.

As long as $\alpha > 0$, it does not change Perceptron decision in any step. This is why for Perceptron, you only need to set learning rate to be 1.

Specifically answer the question, when people say "the learning rate only scales $w$", they are referring to $J_2(w) = max(0, -\alpha yw^Tx)$ rather than $w_{new} = w_{old} \pm \alpha x$.

A related question I found very helpful is  Normalizing the final weights vector in the upper bound on the Perceptron's convergence


Is there any good practice to cluster 3D data array? So I'm not sure what word fits best to describe this data, probably "dimension" would be wrong since it may be used for flat samples with 3 features;

but by 3D data I mean some structure in a form of [samples, timesteps, features]. And there are 2 features in each timestamp.

It looks like [ [ [1,2], [3,4] ], [ [5,6], [7,8] ] ], like an LSTM input.
[1,2] is a timestep and [[1,2],[3,4]] is a sample.

So one way is to just flatten out timesteps and make them into a 1D array. However is there any better way that would somehow utilize the information conducted by "grouping" of features inside a timestamp?

Also how do I properly describe this data structure?

Given that allmpst all clustering algorithms assume data is unordered, reshaping the data into some n*p format ist indeed appropriate. If you want to take positions into account, you'll have to encode them as additional features (which can prove to be tricky because of scaling and feature weighting).

But don't treat clustering as a black box. You may have some particular goal in mind, and adequately preparing the data is a must for clustering. Consider k-means: it searches a least-squares appropximation. It's you job to prepare the data in a way that least-squares on these features is useful.


How to plot/visualize clusters in scikit-learn (sklearn)? I have done some clustering and I would like to visualize the results.

Here is the function I have written to plot my clusters:

import sklearn
from sklearn.cluster import DBSCAN
from sklearn import metrics
from sklearn.preprocessing import StandardScaler

from sklearn.cluster import DBSCAN
from sklearn import metrics
from sklearn.datasets.samples_generator import make_blobs
from sklearn.preprocessing import StandardScaler

def plot_cluster(cluster, sample_matrix):
    '''Input:  "cluster", which is an object from DBSCAN,
       e.g. dbscan_object = DBSCAN(3.0,4)
"sample_matrix" which is a data matrix:
X = [
    [0,5,1,2],
    [0,4,1,3],
    [0,5,1,3],
    [0,5,0,2],
    [5,5,5,5],
    ]
        Output: Plots the clusters nicely.
    '''
    import matplotlib.pyplot as plt
    import numpy as np

    f = lambda row: [float(x) for x in row]

    sample_matrix = map(f,sample_matrix)
    print sample_matrix
    sample_matrix = StandardScaler().fit_transform(sample_matrix)

    core_samples_mask = np.zeros_like(cluster.labels_, dtype=bool)
    core_samples_mask[cluster.core_sample_indices_] = True
    labels = cluster.labels_

    # Black removed and is used for noise instead.
    unique_labels = set(labels)
    colors = plt.cm.Spectral(np.linspace(0, 1, len(unique_labels)))
    for k, col in zip(unique_labels, colors):
        if k == -1:
            # Black used for noise.
            col = 'k'

        class_member_mask = (labels == k)  # generator comprehension
        # X is your data matrix
        X = np.array(sample_matrix)

        xy = X[class_member_mask & core_samples_mask]

        plt.plot(xy[:, 0], xy[:, 1], 'o', markerfacecolor=col,
                 markeredgecolor='k', markersize=14)

        xy = X[class_member_mask & ~core_samples_mask]
        plt.plot(xy[:, 0], xy[:, 1], 'o', markerfacecolor=col,
                 markeredgecolor='k', markersize=6)

    plt.ylim([0,10])
    plt.xlim([0,10])
#        plt.title('Estimated number of clusters: %d' % n_clusters_)
    plt.savefig('cluster.png')

The function above is copied almost verbatim from the scikit-learn demo here.

Yet, when I try it on the following:

dbscan_object = DBSCAN(3.0,4)

X = [
    [0,5,1,2],
    [0,4,1,3],
    [0,5,1,3],
    [0,5,0,2],
    [5,5,5,5],
    ]

result = dbscan_object.fit(X)
print result.labels_

print 'plotting '
plot_cluster(result, X)

...It produces a single point.  What is the best way to plot clusters in python?

When I run the code you posted, I get three points on my plot:

The "point" at (0, 4) corresponds to X[1] and the "point" at (0, 5) is actually three points, corresponding to X[0], X[2], and X[3]. The point at (5, 5) is the last point in your X array. The data at (0, 4) and (0, 5) belong to one cluster, and the point at (5, 5) is considered noise (plotted in black).

The issue here seems to be that you're trying to run the DBSCAN algorithm on a dataset containing 5 points, with at least 4 points required per cluster (the second argument to the DBSCAN constructor). In the sklearn example, the clustering algorithm is run on a dataset containing 750 points with three distinct centers. Try creating a larger X dataset and running this code again.

You might also want to remove the plt.ylim([0,10]) and plt.xlim([0,10]) lines from the code; they're making it a bit difficult to see the points on the edge of the plot! If you omit the ylim and xlim then matplotlib will automatically determine the plot limits.


What is the best ML/DL model to choose to calculate mobile network utilization increase Let’s imagine the following scenario:

The marketing department decides to do the promotion next month and would like to give to every single mobile customer extra 10GB of data.

Due to simplicity, I have the following 5 features:

A: Amount of data volume downloaded by all customers per base station/transmitter (MB)

B:  Number of connected customers to the base station (#)

C: Total duration of the connected customers to the base station (seconds)

D: Radio resource utilization (%)

E: Average throughput per customer (kbps)

Considering the marketing requirements, I can work out that the average increase of data volume per base station is 10% (Feature A).

The question is: What impact is this promotion going to have on the mobile network (every single base station), mainly on Radio resource utilization (Feature D) and the average throughput per customer (Feature E).

I have tons of data for each base station from low to high traffic scenario, but I cannot train the model with 10% extra data volume traffic increase as this situation never happened in the network. It is still going to be "the peak data volume per Base station so far" + extra 10%.

How can I train the model, if target label is unknown (peak data volume + extra 10% never happened on the particular base station and I cannot take the data/stast from a different base station as the traffic pattern is different)?
It would be enough to point me into the direction, I can find more info and to study it further. Thanks.

I do not see why having a percentage greater than 100% might be a problem to you? You might normalize it by dividing by 100, so 100% might become 1. That feature will be your label.
I suggest you try several regression algorithms, with different settings, and choose the best model i.e. the model which provides the best result. You have a bunch of regressors in scikit learn, you may also try XGBoostRegressor or CatBoostRegressor.


Kernel with complex vectors I have a question regarding my machine learning lecture where we had to decide whether $$K(x,y)=x_1y_1-x_2y_2$$ is a valid kernel (e.g. for a SVM). My intuition would say that it is a valid kernel since we can display it with: $$\Phi(x)=(x_1, ix_2)\implies K(x,y)=\Phi(x)\Phi(y)$$ with $i$ being the imaginary number. Is that right?

Kernels are considered valid if they are positive-definite, so we must have
$$
\sum_{i=1}^n \sum_{j=1}^n c_i c_j K(x_i, x_j) \ge 0
$$
for all $n \in \mathbb{N}$ and $c_1, \dots, c_n \in \mathbb{R}$ and (in your case) all $x^1, \dots, x^n \in \mathbb{R}^2.$

Letting $n=1$ and $c_1 = 1$ and $x^1 = (0, 1)$ shows that $K$ is not positive-definite, since
$$
K\big((0,1), (0,1)\big) = 0\times 0 - 1\times 1 = -1.
$$

The problem with your approach is that you define $\Phi$ on $\mathbb{C}$ but write $K$ as the dot product for real-valued vectors.
The feature map argument for positive definite vectors only works if the product is an inner product on $\Phi$'s codomain.
The dot product for real-valued vectors is not an inner product on $\mathbb{C}^2$ (because $i\cdot i = -1 < 0$).
You would have to use the generalized dot product $x \cdot y = \sum_i x_i \overline{y_i}$, or another inner product on $\mathbb{C}^2$.


How do I know how to construct the layers of my CNN I've done a CNN project with Keras and OpenCV, and I've got roughly 65% accuracy. And now I have to present this work in my University, but I'm afraid if the teachers ask me for how do I knew how to construct the right layers to my CNN.

In fact in my development I took a look in others projects to see how the other people do, and I know that you have to have a number of neurons equivalent to the number and size of your image. But I know that has a mathematic behind this.

So how could I tell if they ask me it?

There isn't a formula behind it - layer sizes are selected with a lot of trial and error (often automated by cross-validation). But saying "I've used an architecture from a network from a similar problem, or at least from the same domain" is a legitimate answer.


Classifier design for website screenshots I'm working on a project that requires determining if the page representing a hosted file on a third party platform (such as rapidgator or nitroflare) is still up or not.

For example, here is a file that is still up.

Here is a page representing a file that is no longer up.

I have identified several hundred file hosting services and each of them have their own way of displaying up vs down pages.

Is it possible to design/train a model that would classify screenshots of these hosted file pages as up vs down? If so what approach is recommended? Does using a method other than image classification make more sense here?

Yes, instead of image classification you should look to scrape the websites themselves to get to file links or down messages directly.

You could still use an ML to determine down messages with minimal user input but it will be much faster and easier than image recognition.

Image recognition will fail you here because to start you have a population of 100 or so which is a small sample and you'd need to hand label at least 50 to train any kind of viable classifier at which point you might ad well just hand label them all.


Time Series pattern recognition and classification problem I have some labeled sensor data.

Now, I would like to know how to extract features from time series using DFT, DWT, and HAAR transforms.

I know that the transformations above transform a signal to frequency or time frequency domain, but how can I get average frequency for this non stationary signal?

Thanks.

I do not fully understand what you mean by "average frequency". But this may help to think about it.

You can calculate the power spectrum of the signal using DFT.
From the power spectrum, you will be able to get the average power of all the frequencies.
Use this value of average power to map the frequency which it closely corresponds to.

This will give a sort of the average frequency you are looking for from the average value of power.


Opensource tools for help in mining stream of leader board scores Consider a stream containing tuples (user, new_score) representing users' scores in an online game. The stream could have 100-1,000 new elements per second. The game has 200K to 300K unique players.

I would like to have some standing queries like:

Which players posted more than x scores in a sliding window of one hour
Which players gained x% score in a sliding window of one hour

My question is which open source tools can I employ to jumpstart this project? I am considering Esper at the moment.

Note: I have just completed reading "Mining Data Streams" (chapter 4 of Mining of Massive Datasets) and I am quite new to mining data streams.

This isn't a full solution, but you may want to look into OrientDB as part of your stack. Orient is a Graph-Document database server written entirely in Java.

In graph databases, relationships are considered first class citizens and therefore traversing those relationships can be done pretty quickly. Orient is also a document database which would allow you the kind of schema-free architecture it sounds like you would need. The real reason I suggest Orient, however, is because of its extensiblity. It supports streaming via sockets, and the entire database can be embedded into another application. Finally, it can be scaled efficiently and/or can work entirely through memory. So, with some Java expertise, you can actually run your preset queries against the database in memory.

We are doing something similar. In creating an app/site for social science research collaboration, we found ourselves with immensely complex data models. We ended up writing several of the queries using the Gremlin Traversal Language (a subset of Groovy, which is, of course, Java at its heart), and then exposing those queries through the binary connection server of the OrientDB. So, the client opens a TCP socket, sends a short binary message, and the query is executing in Java directly against the in-memory database.

OrientDB also supports writing function queries in Javascript, and you can use Node.js to interact directly with an Orient instance.

For something of this size, I would want to use Orient in conjunction with Hadoop or something like that. You can also use Orient in conjunction with esper.

Consider:
An introduction to orient: http://www.sitepoint.com/a-look-at-orientdb-the-graph-document-nosql/

Complex, real-time queries: http://www.gft-blog.com/business-trends/leveraging-real-time-scoring-through-bigdata-to-detect-insurance-fraud/

A discussion about streaming options with java and orient: https://github.com/orientechnologies/orientdb/issues/1227


Does resizing the image size, reduce the quality of the images in CNN? I am doing a project for cancer recognition. My data set has hunderds of images but not of equal size. I wanna resize them to the size of the smallest image. But I am wondering do you think using the cv2.resize will effect the quality of images and so the result of  my work? If yes what I should do. If no please let me know.
Width disturbuation

higth disturbuation

Of course, making them smaller will cause loss of information. Since before you had N x M data points for each image to describe the content of the image, but after resizing you will have n x m (Where n and N denote the number of rows in an image, m and M denotes the number of columns in an image. Also, n <= N and/or m <= M). That is, a small number of pixels will result in less information.
Open CV's resize() function squeezes the image instead of cropping, meaning it preserves aspect ratio, original content but it displays objects or parts in the image with fewer pixels. To achieve it, Open CV uses the interpolation technique. (read more here)
Two solutions you can try:

If your smallest image's size does not differ much from the size of the majority of your image, then resize them to the smallest. Because in that case, if you have a good model, your performance will not be affected severely.

If your smallest image's size differs much from the majority, or in general you have a group of small images and big images which sizes differ a lot, then choose a threshold of minimum size to which you will resize the images and drop the images that have a size less than your threshold.

In both cases, you will lose data or information, it is inevitable. The best practice is to test various possibilities to see which works well. Look at the distribution of the sizes of your images, make your decision accordingly.
If the size of the smallest image is big enough, and the smallest image and the biggest image do not differ a lot in size, I would just resize them. Because it would at least, gives a chance to the model to capture the feature in the image. However, in the second case (dropping the small images), you would not even give that chance to the model.


Primer on Random Forest Algorithm I wanted to ensure that I understood the random forest algorithm right. I think it's fundamental to understand how something work before using it!

Am I right with this:

If not: what did I understand wrong?

Ok. I think I got it now. It should be like this:

Am I right? If not: whats wrong?


Encoding Tags for Random Forest I have the following data set:

I want to use attributes Tags and Authors to classify each record into their respective Rating. In order to do so I want to use a random forest classifier. My concern is how to deal with Tags attribute. Each of the entry has an undetermined number of tags separated by a commas. There are a total of 4412 unique tags and the entry with more tags contains 20 tags. The first entry has tags ["Rhode Island","Economy", "Taxes", "Lincoln Chafee"].
How should I encode this attribute such that I can use Random Forest Classifier from sklearn?

You should use sklearn MultiLabelBinarizer

from sklearn.preprocessing import MultiLabelBinarizer
lb = MultiLabelBinarizer()

lb.fit_transform([['A', 'B', 'C'],[ 'A', 'D', 'E', 'B']])

array([[1, 1, 1, 0, 0],
$\hspace{1cm}$ [1, 1, 0, 1, 1]])

If required, remove the columns below a threshold value (sum of the column). This will reduce the Features count by removing the low variance Features


Neural network regression is not dynamic enough to predict target range? I'm working with a CNN on a regression task, using MSE as the loss function. Here's a plot of predictions vs targets for the training set.
Note: Legend is wrong. Blue = prediction vs target | Red = fit | Green = ground truth (x=x)

I know that the approximate mean for the target is 0.8 so I initialise the bias of the last fully connected layer to this. (In my case it's reasonable to assume this for unseen samples as well). That's why the predicted fit crosses the ground truth at (0.8, 0.8).
What's interesting is that the R^2 is pretty good, but the slope is off. If I wanted to calibrate my model without further training I could easily apply a linear transformation to change the slope (note that the validation data doesn't look too dissimilar).
I feel like the model should have learned this extra factor. What could cause this lack of "sensitivity" or "dynamism"?
For the signal processing people here, this reminds me a lot of when a PID regulation loop doesn't have a strong enough actuator to keep up with fast transients. Although it's a completely different concept, I feel like somehow the weights aren't large enough to produce the necessary gradient???

I've solved it myself. Actually the whole premise was incorrect. It was tempting to look at the plot and think that it just takes a linear transformation to change the slope of the line, but that's because one axis of the plot is the target data!
My claim was that the model should stretch the data along the y-axis (around the mean) because it's the obvious thing to do. But is it really? Have a look at these 4 examples. According to my "obvious fudge factor", the two points on the right should have been stretched up, and the two on the left should have been stretched down.

But then what would that mean for all the other points that were on target? It would shift them off target and overall the model would perform worse. Here's what would happen to some of the points stretched up away from the mean.

The tempting human thing to do, which is what got me, is to include the x-axis in your perception of the problem. But we have to remember that it's not there for the model.
Every time the model predicts deviations from the mean it needs to have a pretty strong reason to do it. If it hasn't actually found the real reason to do so, then it shouldn't, otherwise the overall performance would be worse.
As the model gets better, we don't just see a better R^2 (dots line up better), but we also see the slope increase, because the model gets more confident about deviating from the mean.


Multiple merging multiple convolutions (First post here)
I am rather new to neural networks, having used Tensorflow for a couple months now, and am looking for some advice I have on an idea to improve the accuracy of my model. I am looking at sequences using 1d convolution layer with spatial dropout, and an LSTM to look for spatial relatedness of these features.
My idea was that since these sequences have different types of features with at different sequence lengths I might try sending my input sequence to multiple 1dconv layers with different window sizes and then merging them back together.

Would this be effective at all? Is there a way to some how link the layers so that the layers with the larger window would not learn patterns from layers of the smaller windows? Or should I just make the original convolution layer larger with more filters?

PS The my original idea was to pair each individual convolution with an LSTM on top of it, and merge these into another convolution with an LSTM on top of that, but it seems to be prohibitively complex on my computer.

What you describe sounds a lot like the inception module. You can use Keras to concatenate convolutions with different filter sizes acting on the same input, such as:

This will increase the compute time, but might give you what you want. Above image is taken from here, where you can also find the code for building such modules using Keras in the section titled How to Implement the Inception Module.


Non-commutative distance formula I am trying to find a distance formula or a method that can give the non-commutative distance between two points in a feature space.
Suppose there are two movies represented in an R^n feature space. Now I want that when I try to find the distance/similarity between these movies using the feature vectors, I get different values with respect to which movie is the reference point i.e.,
Dist(Mov1, Mov2) != Dist(Mov2, Mov1)

I know this is slightly vague, but I am trying to understand my thoughts and would like your help. I found that there is a field called non-commutative geometry, but that seems a little too much and the last resort to dive into. Kindly direct my thoughts.

Take for example the Euclidean distance L2, defined by:
$$L_2(x,y) = \left(\sum_{i=1}^{d} (x_i-y_i)^2\right)^{1/2}$$
where $d$ is the vector dimension. You can easily add a term $\alpha \in (0,1)$ and put more weight on the first term, for example:
$$L_2(x,y) = \left(\sum_{i=1}^{d} (\alpha x_i-(1-\alpha)y_i)^2\right)^{1/2}$$
That will certainly be non-cummutative and basically it can be interpreted as putting more weight on the 'reference' vector.
Now, if you ask me, what would be a good use of it, I hardly understand why you would use a non-commutative distance or non-symmetric distance. First of all, it will not be a distance in mathematical sense, since symmetry is required by definition, secondly, many algorithms works with symmetric distances and can't be adapted easily.


Does GridSearchCV not save the best parameters? So I tuned the hyperparameters using GridSearchCV, fitted the model to the data, and then used best_params_. I'm just curious why GridSearchCV takes too long to run best_params_, unlike RandomSearchCV where it instantly gives answers. The time it takes for GridSearchCV to give the best_params_ is similar to the time it takes for GridSearchCV to tune hyperparameters, and fit the model to the data. It's as if it's doing it all over again when it has done so already. Is this the case? If not, what's taking it so long when it should have saved the best_params_ when I ran GridSearchCV the first time?

It doesn't, please try the following code
CELL1:
import numpy as np
from sklearn import svm, datasets
from sklearn.model_selection import GridSearchCV
iris = datasets.load_iris()

CELL2:
%%time
parameters = {'kernel':('linear', 'rbf'), 'C':np.linspace(0.1,100,1000)}
svc = svm.SVC()
clf = GridSearchCV(svc, parameters)
clf.fit(iris.data, iris.target)

CELL3:
%%time
clf.best_params_

Wall time of CELL2 will be about 7-9 seconds.
Wall time of CELL3 will be 0ns. ( instantaneous )
This is because best_params_ is an argument of GridSearchCV. It is however only created (and accessible) once you run .fit method.


Is there any difference between a weak learner and a weak classifier? While reading about decision tree ensembles Gradient Boosting, AdaBoost etc.
I have found the following two concepts weak learner and weak classifier.
Are they the same?
If there is any difference what is it?

A weak learner can be either a classification or a regression algorithm:

  Boosting (Schapire and Freund 2012) is a greedy algorithm for fitting adaptive basis-function models of the form in Equation 16.3, where the $\phi_m$ are generated by an algorithm called a weak learner or a base learner. The algorithm works by applying the weak learner sequentially to weighted versions of the data, where more weight is given to examples that were misclassified by earlier rounds.
  This weak learner can be any classification or regression algorithm, but it is common to use a CART model.

Source: "Machine Learning - A Probabilistic Perspective"; Murphy; 2012

So a weak classifier is simply a weak learner which is a classifier.


Find repeating patterns in data I have database of sequential events for multiple animals.
The events are represented by integers so it looks something like:
Animal A: [1,6,4,2,5,7,8]
Animal B: [1,6,5,4,1,6,7]
Animal C: [5,4,2,1,6,4,3]

I can manually see that for each event 6 event 1 first happens.
And event 4 happens quickly after a 1,6 combination.
But these are easy to spot in such a small dataset, the real lists are 10000+ events per animal.
Is there a way to use an algorithm or machine learning to search for these kinds of patterns?

I think this problem could be represented as a Markov Chain.
Maybe a Markov model (or Markov Random Fields) could be used to estimate the probabilities. Normally it's easier than with the more standard Hidden Markov Model since there is no hidden state.
My suggestion would be to estimate the probabilities of the model (parameters) from the data, then it should be easier to discover the patterns based on these probabilities.


How to tune bandwidth in machine learning kernel model? Gaussian kernel

$k(x,y) = \exp(-\lVert x-y \rVert^2/\sigma^2)$

has a hyperparameter $\sigma$.

I know grid search cross validation, but this would require a lot of computation since computational cost of kernel method scales with the number of samples to the power of 2.

As shown in wikipedia for KDE, a rule-of-thumb bandwidth estimator can be given if the underlying density for your data is Gaussian. This estimator is given by: $h = (\frac{4\hat{\sigma}}{3n})^{1/5}$, where $h$ is the bandwidth of your KDE estimation, $n$ the number of data and $\hat{\sigma}$ the estimation of the standard deviation of your sample.

If the underlying distribution for your data is not gaussian, you can still try with this bandwidth, but it might smooth everything too much. In that case, you should go for cross-validation with smaller bandwidths.


Normalize data with extreme outliers for forecasting Suppose I have input values that represent the change of a stock share from each time step to the next. Now I want to feed these values into an LSTM Neural Net. My problem is that most values are between -0.000001 and 0.000001, but some values go up to 0.1 or -0.1. How can I effectively scale such a dataset. I've tried applying a log function (to the value +1, so e.g. log(1.000001)) and scaling between 0 and 1 and I've tried capping values at around 1.5. Do you have any suggestions how to deal with such extreme outliers better?

Play a bit with your data:

try doing the scaling without these outliers and see what happens
have you tried training the model regardless and see what happens? then train the model without the "outliers"
you could also binarise the feature (and thus transform it into a category) - then in this case you don't have to deal with the large range while the information is still in there.


Binary softmax decision boundary In this paper, in figure 4:

the decision boundaries are shown for different $W1$ and $W2$s. If the softmax classification forces either $W_1.x > W_2.x$ or $W_1.x < W_2.x$ for class 1 and class 2 respectively, then the decision boundary is $(W1-W2).x = 0$ which means that that the decision boundary is perpendicular to $W1-W2$. However in this figure only in the case where $||W1|| = ||W2||$ this is visually correct. In other cases, this is not the case. If one draws $W1-W2$, it won't be perpendicular to the decision boundary.

Am I thinking right?

Yes you are right. But I think the authors only want to emphasize that if $||1|| > ||2||$, the decision boundary will be closer to $2$ and vice versa.


In Recommendation systems, Does we need to build each model for each product if we are using Logistic regression? I am reading this paper wide and deep learning recommedation system paper and haven't understood one thing.

To serve the latency, they actually first get the 100 best apps according to user query and rank them according to the probability score.

Does this mean that while buidling the model we have to build individual model for each app?

Are there any other Deep learning frameworks similar to this ?

I can only guess that since it is binary classifier based recommender model, it predicts whether a product is recommendable to a user or not with a probability score which can be set to a threshold.


Adding a new custom column in a python data frame I need to add a new column in a python data frame that has the dates of January in a column. Each date is repeated 24 times in the column without any intervention. The total number of entries will thus be 31*24 = 744 entries. Can you please help me code this part ?

This turns out to be fairly simple! There is a handy method called repeat on a datetime index. Here are the steps:

import pandas as pd

Define a date range, supplying start and end

jan = pd.date_range(start="1-Jan-2018", end="31-Jan-2018")    # could specify any year

Now provide how many times you want to repeat each date and create the repeated column

num_repeats = 24
repeated_jans = jan.repeat(num_repeats)

Let's create the random dummy dataframe as a base

total_dates = num_repeats * len(jan)    # 24 x 31 = 744
df = pd.DataFrame(np.random.randint(0, 10, total_dates))

This is how we add a column - the name of the column can be anything

df['repeated_jans_lalala'] = repeated_jans

Have a look at some dates:

print(df.iloc[[0, 24, 48, 71, 72]])    # multiples of 24...we can see one repeated date

    0 repeated_jans_lalala
0   7    2018-01-01
24  4    2018-01-02
48  3    2018-01-03
71  6    2018-01-03
72  3    2018-01-04

If try adding the column to a dataframe with a different number of rows, we get an error:

df_error = pd.DataFrame(np.random.randint(0, 10, 743))    # require 744!
#df_error['repeated_jans'] = repeated_jans                         # raises ValueError

If you want to change the way the dates look, you can use the strftime method on the dates

jans_fancy = jans.strftime('%d-%B-%y')
df['fancy_jans'] = jans_fancy

df.head()

0   repeated_jans_lalal  fancy_jans
0   7   2018-01-01       01-January-18
1   2   2018-01-01       01-January-18
2   8   2018-01-01       01-January-18
3   9   2018-01-01       01-January-18
4   7   2018-01-01       01-January-18

If you don't want to show the actual year, just leave out the %y part!


How to predict correct text from incorrect text with machine learning? I have a dataframe like this:

import pandas as pd

df = pd.DataFrame({'incorrect': ['jak', 'mya', 'kfka'],
                   'correct': ['jack', 'maya', 'kafka']})

incorrect  correct
jak        jack
mya        maya
kfka       kafka

I want to predict correct results from incorrect values with machine learning in python. Which model can I use for this case? Thanks.

It's unclear from your example how exactly you intend to predict if all you have are pairs of terms.
If you're assuming that you have all possible correct and incorrect pairs, then you can simply use your data as a hash table where each incorrect term is the key of a correct term.
Otherwise, assuming you have all the correct terms but you want to be able to handle slightly changed versions of the incorrect terms.
You can define a similarity measure (function) and use that to find the closest incorrect term, in order to return the correct term for it.
If you don't have all the correct words, then it's a problem.
You should look for a vocabulary of 'legal' terms (or names in your case) and use that as your dictionary.
There are some libraries in python that can do that for you, but as far as I know these are all based on regular dictionaries, like the ones used by LibreOffice for example (PyEnchant) is one example for that.
If you're looking for a dictionary for names, you should perhaps try looking for a specific dataset that consists of names.

Otherwise your data should be in a different format, which will have more information in it, such as context for example.

One way to 'predict' correct words based on incorrect words is constructing a LM (Language Model) for each (correct and incorrect) and based on probabilities returning the ones with maximal likelihood, like a basic translation model. But this will only make sense if you have a lot of text (not unigrams) and each word appears several times, otherwise you don't really have a way to construct the LMs.


What morphological operations can be used to vectorise road surface binary rasters? We are currently developing a Deep Convolutional Neural Network to extract road surfaces from aerial orthophoto's. Our current workflow involves using an existing road centreline to mask a large enough area to determine which of the pixels are road surface. The probability binary raster output are reasonably accurate. The final step within our workflow is that we use the original road centreline and average pixel width to determine the road width to buffer the road centreline as the output the road surface area. The problem with the following approach is that collector and arterial roads are not uniform in road width as well as the road bellmouth is lost at road intersections. Currently the team that has developed the following model is using Matlab, which I'm not familiar with, hence the following question to find out what morphological operations could be used to extract a better representation of the road surface by approximating road edges (straight lines), varying road widths and bellmouths at road intersections from the probability raster output from the CNN model.

CNN: Current Workflow

CNN: Red (Manually Derived Road Surface \ Training Dataset) Yellow (Vectorised Road Surface Output)

Perhaps another solution to your question: There are datasets that contain road surfaces, such as the (Dutch) BGT-database.


Linear Regression bad results after log transformation I have a dataset that has the following columns:

The variable I'm trying to predict is "rent".
My dataset looks a lot similar to what happens in this notebook. I tried to normalize the rent column and the area column using log transformation since both columns had a positive skewness. Here's the rent column and area column distribution before and after the log transformation.
Before:

After:

I thought after these changes my regression models would improve and in fact they did, except for Linear Regression.
If I don't do any type of transformations the models underperform. When I only transform the rent column all models improve including Linear Regression, but when I transform the rent column and the area column Linear Regression has a terrible result with a MAPE of 2521729.47.
Not transforming area MAPE results:

Transforming area MAPE results:

Can anyone tell me what's probably happening or guide me through any type of testing or verifications to understand what's happening to linear regression? Am I wrong to transform those columns even if the models are improving?
Edit:
After testing the models by removing and adding columns, I found that linear regression goes crazy after I insert the neighborhood column (which contains 66 neighborhoods) and create dummy columns. When I create this dummy variables the number of columns goes to 77, while the dataset has only around 3000 rows.
My thoughts are that after transforming the column into dummy columns the data becomes very sparse and with too many features for only 3000 rows, and that's why Linear Regression has this bad performance and Lasso Regression doesn't. Besides that, I should probably still use the other models since they perform well after the changes.
Am I correct?

Make sure you transformed back your predictions and actual values before calculating MAPE.
You can check which observations contributed the most to high MAPE.
MAPE is very sensitive to prediction errors at small actual values. Most likely worst performing observations ("from MAPE perspective") are those with small actual values.
Depending on the goal of your analysis you could check other metrics as well (eg: MAE).
Sparsity: Yes, you might have a neighborhood category in your test set, that does not exist in your training set (or has only a few examples). In this case predictions for that category might be very bad. Though this does not explain why you don't have high MAPE when you don't transform area.


Algorithm to find common sequence Assume that "1,2,3" are the ids of users, active means that person visited the stackoverflow in last one month (0=passive, 1=active), and there are positive and negative votes.

id  question       votes                 active
 1     1        -1, +1, -1, -1, -1         0
 1     2        -1, +1, -1, -1, +1         0
 2     1        +1, +1, -1, -1             0
 3     1        +1, +1, +1, -1, +1         1
 3     2        +1, +1, -1, +1, +1, +1     1
 3     3        -1, +1                     1

I want to know what makes the users stop using stackoverflow. Think that, I have already calculate the how many times did they get negative votes, total vote, average vote for each question...

I wonder what kind of information could I get from these sequences. I want to find something like this: these users who are passive have two negative votes sequentially. For example, one positive vote after two negative votes in the second question of user 1, doesn't prevent the user churn. User 3 doesn't have any 2 negative votes sequentially in any of his questions. Hence he is still active.

I'm looking for something like PrefixSpan Algorithm but order is important for me. I mean, I can't write the sequences like

<(-1 +1 -1 -1 -1) (-1 +1 -1 -1 +1 )>

or

<(-1) (+1) (-1) (-1) (-1) (-1) (+1) (-1) (-1) (+1 )>.

Because the first one loses the order, and the second one jumbled the questions together. Is there any algorithm to find these sequences which is common for churners?

Not a typical problem formulation indeed. I suggest two approaches:

Use a simple bayesian model where the states or nodes are each of your features. Connect the nodes directly when they haven a direct time dependency (for example vote1 -> vote2 -> vote3, but not vote3 -> vote1) and calculate the probability of a final (output) node to be something (for example user type 1, user type 2, etc). You could also use a hidden markov model that naturally models transitions between states. In your case it would output just 1 state (the prediction). I am not very fan of this model for your problem, but it could be nice to try. You will need a lot of data though.
You could use Fuzzy Inductive Reasoning (FIR). It is definitely not a simple algorithm, but works quite well for classification having time series or time-related features. In FIR you want to find an optimal mask of a given depth.
If you have, say, 5 features per sample, and for instance you want to know which features to select you will learn an optimal mask for your data that address this (and creates fuzzy rules, etc...). Also if you want to relate two or more examples you will define a depth higher than 1 for the mask. This will allow you to find a mask that for instance uses features 1,2 and 4 of sample t, features 2,3 ,4 and 5 of sample t+1 and features 1 and 4 of sample t+2. Moving t from 1 to N number of samples. So, interesting combinations of features from samples that have a time-relation (or some sort of sequence) are created.


Normalizing the final weights vector in the upper bound on the Perceptron's convergence The convergence of the "simple" perceptron says that:

$$k\leqslant \left ( \frac{R\left \| \bar{\theta} \right \|}{\gamma } \right )^{2}$$

where $k$ is the number of iterations (in which the weights are updated), $R$ is the maximum distance of a sample from the origin, $\bar{\theta}$ is the final weights vector, and  $\gamma$ is the smallest distance from $\bar{\theta}$ to a sample (= the margin of hyperplane).

Many books implicitly say that  $\left \| \bar{\theta} \right \|$ is equal to 1. But why do they normalize it ?

The Perceptron's output $f$ is
$$
f(\overline\theta \cdot \overline{x}) = \begin{cases} 1 &\text{ if } \overline{\theta }\cdot \overline{x} > 0 \\ 0 &\text{ if } \overline{\theta }\cdot \overline{x} \le 0\end{cases}
$$
Here, $\overline{x} = (1, x_1, \dots, x_n)$ where $(x_1, \dots, x_n)$ is the input vector.

You can see that the output only depends on the sign of the product $\overline{\theta }\cdot \overline{x}.$
Therefore the output will not change if we multiply $\overline\theta \cdot \overline{x}$ with a positive constant:
$$
f(\overline\theta \cdot \overline{x}) = f(c\times \overline\theta \cdot \overline{x}) \qquad \text{ for all } c > 0.
$$
In particular, we may choose $c = 1/\lVert \overline\theta \rVert.$ So we get
$
f(\overline{\theta }\cdot \overline{x}) = f(\tilde\theta\cdot \overline{x}),
$
where $\tilde\theta = \overline\theta / \lVert \overline\theta \rVert$ is the normalized vector.
Conclude that there is no loss of generality in assuming $\lVert \overline\theta \rVert = 1.$


Understanding a dataset (prior to applying ML models) with no metadata given How do you understand a dataset when there is no metadata given (no details about the attributes given in the dataset)? It is difficult to comprehend the attribute names as only the short forms are given.

It's given to me that 'pm2.5' is the target variable. How do I understand which independent variables will affect this target variable?

The aim is to predict pm2.5 ( target variable ).

Step 1: Data Cleaning. Remove unwanted features and fill the missing values.

Step 2: To learn about the features, perform data visualization. You can plot a linear plot with TEMP and pm2.5 and see how it varies with change in temp.

Step 3: The next step is to find the relationship between features. Some features are not needed for prediction. Remove those features.

Step 4: Apply a suitable machine learning technique and predict.


Expectations of the skill set of a Data Scientist Although there are more powerful tools than Excel which a data scientist should know how to use, would it be expected that someone indicating they are a Data Scientist know how to use Excel?

It depends on the level of expertise that is required. Personally I know how to do most of the standard operations you would need for Excel but I don’t think a Data Scientist needs to know more than that.

If this is a job advert where they are asking for advanced knowledge of Excel as a Data Scientist and do not require much in the way of programming/machine learning skills then I would give the job a wide berth. It would indicate to me that the company putting the job out don’t understand what a Data Scientist is and are putting a job out to say they have one.