Testing fit of probability distribution If I have fitted training data to a probability distribution, e.g. a poisson distribution, how can I test this fit on some test data?

To fit the poisson distribution I am using R's fitdistrplus package that using MLE for determining the optimal coefficients of a given distribution. Therefore, I have the estimated $\lambda$ for a poisson distribution based on my training data but I am not sure how to test this on some unseen test data.

Use chi-square test to check the goodness of fit to a specific distribution
http://courses.wcupa.edu/rbove/Berenson/10th%20ed%20CD-ROM%20topics/section12_5.pdf


Can we compare a word2vec vector with a doc2vec vector? I have set of categories and I want to compare a document vector with word vector of categories to find best matching category.

Is it possible to compare a word vector with document vector? If yes, is there any literature which gives proof of concept for this?

In paragraph vector, the vector tries to grasp the semantic meaning of all the words in the context by placing the vector itself in each and every context. Thus finally, the paragraph vector contains the semantic meaning of all the words in the context trained.

When we compare this to word2vec, each word in word2vec preserves its own semantic meaning. Thus summing up all the vectors or averaging them will result in a vector which could have all the semantics preserved. This is sensible, because when we add the vectors (transport+water) the result nearly equals ship or boat, which means summing the vectors sums up the semantics.

Before the paragraph vector paper got published, people used averaged word vectors as sentence vectors. To be honest, in my work these average vectors work better than document vectors. So, with these things in mind, in this way it could be compared.


Disproving or proving claim that if VCdim is "n" then it is possible that a set of smaller size is not shattered Today in the lecture the lecturer said something I found peculiar, and I felt very inconvenient when I heard it:
He claimed, that if the maximal VCdim of some hypothesis class is $n\in\mathbb N$, then it is possible that there is some $i<n$ such that for every subset C of size i the subset C is not shattered.
Is his claim true? I thought that we can take some subset of size $i,\forall i\in [n]$of the set C* which satisfies the condition for the case where $|C|=n$, and they will shatter as well. Am I missing something?

I agree, the claim as written is incorrect.  If $C^*$ is shattered by $\mathcal{H}$, and $C\subseteq C^*$, then $C$ is also shattered by $\mathcal{H}$; to be possibly over-pedantic:
For each $B\subseteq C$, because then also $B\subseteq C^*$, we have that there is an $H\in\mathcal{H}$ such that $C^*\cap H=B$.  Note that this implies $B\subseteq H$.  Now, $C\subseteq C^*$ implies that $C\cap H\subseteq C^*\cap H$, so we have $C\cap H\subseteq B$.  And having both $B\subseteq C$ and $B\subseteq H$ implies that $B\subseteq C\cap H$.  So $B=C\cap H$.


Exceptionally high accuracy with Random Forest, is it possible? I need your help to find a flaw in my model, since it's accuracy (95%) is not realistic.

I'm working on a classification problem using Randomforest, with around 2500 positive case and 15000 negative ones, 75 independent variables. Here's the core of my code:

# Splitting the dataset into the Training set and Test set
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.25, random_state = 0)

# Fitting Random Forest Classification to the Training set
from sklearn.ensemble import RandomForestClassifier
classifier = RandomForestClassifier(n_estimators = 900, criterion = 'gini', random_state = 0)
classifier.fit(X_train, y_train)

# Predicting Test set results
y_pred = classifier.predict(X_test)

# Making the Confusion Matrix
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, y_pred)

I've optimized the hyperparameters through grid search and performed a k-fold cross validation, reporting 0.9444 as accuracies mean.
Confusion matrix:

[[3390,   85],
 [ 101,  516]]

showing 97.6% accuracy.

Did I miss something?

NOTE: the database is composed by 2500 Italian mafia firms' financial reports, and 15000 lawful firms randomly sampled from the same regions as negative cases.

Thank you guys!

EDIT: I upload the metrics and cm. The model is actually performing well, but looking at the metrics and cm, it shows more realistic values regarding logloss and recall, so I assume it is fine.

In order to understand if the model is performing well, first would do the following:

Plot the distribution of the classes, to understand if sampling mechanisms are required.
In case the classes are not evenly distributed, would do stratified sampling during the test-train split.
After the prediction, would plot the confusion matrix that is supported by libraries such as matplotlib or seaborn
Based on the class distribution its also important to understand what sort of metrics are required, micro-averaging / weighted / macro-averaging of precision, recall and f1 score.

This should help you evaluate if you are model is truly learning the features or if one of the classes are imbalanced causing such a spike in the results.


Why Epochs take longer as learning proceeds? I am training a deep learning model in Tensorflow on GPU (Amazon AWS) and what I observe is that in the beginning each epoch takes only less than a second but let's say after 80 epochs one epoch takes more than 8 minutes.

What could be a possible reason?

As the number of epochs increase the error goes down and the neural network has less to learn from the given data. The learning rate also decreases towards the end which makes the calculations of the gradients and weight updates numerically unstable. These are some of the reasons that contribute to slow learning. You can have a look here Quora for more details where Ian Goodfellow has answered a similar question.


Can random forest algorithm provide customer churn prediction probability at each customer instead at class level? I have customer training data set from telecom industry along with its test data set containing churn values 0 & 1 for each customer. I also have customer data set whose churn value is to be predicted ie 0 & 1. It is also required to get the churn prediction probability at individual customer level so that they can be arranged in descending order of the propensity to churn

For brevity, showing limited features
cust_train.xls

cust_id Account Length  VMail Message   Day Mins    Eve Mins    Night Mins  Intl Mins   CustServ Calls

cust_train_output.xls

cust_id churn (0/1)

I want to know if it is possible to get the churn prediction probability at individual customer level & how by random forest algorithm rather than class level provided by:
predict_proba(X) => Predict class probabilities for X.

Goal is to arrange the customer in descending order of the propensity to churn.

Alterntively, is this possible with Logistic Regression Model?

Thanks

I want to know if it is possible to get the churn prediction probability at individual customer level & how by random forest algorithm rather than class level provided by: predict_proba(X) => Predict class probabilities for X.

I think the predict_proba(X) is actually the correct method to accomplish your task.  For each instance (person) the function will display the probability for each class label. So, if you know which class is churn, I'll just assume class 1, you just need to slice the results on that column.  For example:

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from tabulate import tabulate

X, y = make_classification(n_samples=1000, n_features=50,
                           n_informative=2, n_redundant=0,
                           random_state=1, shuffle=False)
clf = RandomForestClassifier(n_estimators=100, max_depth=2,
                             random_state=0)

clf.fit(X, y)

train_class_probability = clf.predict_proba(X)

print(tabulate(train_class_probability[0:4], ['Churn', 'No Churn']))

NewData, y = make_classification(n_samples=10, n_features=50,
                                 n_informative=2, n_redundant=0,
                                 random_state=1000, shuffle=False)
class_prediction = clf.predict(NewData)
class_probability = clf.predict_proba(NewData)

print(tabulate(class_prediction.reshape(-1, 1), ['0 - Churn / 1 - No-Churn']))
print(tabulate(class_probability, ['Churn', 'No Churn']))
# Just print the first column, Churn
print(tabulate(class_probability[:, 0].reshape(-1, 1), ['Probability of Churn']))

Output will be:

Class Probabilities of the Training Data (First four instances)

   Churn    No Churn
--------  ----------
0.595298    0.404702
0.620975    0.379025
0.601251    0.398749
0.610663    0.389337

Class Prediction on New Data

  0 - Churn / 1 - No-Churn
--------------------------
                         0
                         0
                         0
                         1
                         1
                         1
                         1
                         0
                         1
                         1

Class Probability Prediction on New Data

   Churn    No Churn
--------  ----------
0.553698    0.446302
0.602109    0.397891
0.587715    0.412285
0.423982    0.576018
0.419588    0.580412
0.419984    0.580016
0.369798    0.630202
0.572373    0.427627
0.414734    0.585266
0.40422     0.59578

Probability of Churn on New Data

  Probability of Churn
----------------------
              0.553698
              0.602109
              0.587715   <- new customer #3
              0.423982   <- new customer #4
              0.419588
              0.419984
              0.369798
              0.572373
              0.414734
              0.40422

So, the probability that "new customer #3" will churn is ~59% and the probability that "new customer #4" will churn is ~42%.

Hopefully, I have understood your question.

HTH


What is the best statistical measure tool to measure how close data is to fitted regression line if outliers are not fitted I am using a custom algorithm based on Gradient descent which computes the best fit on a training dataset. In this data set I have outliers i.e. data points that I do not want to fit. The algorithm is doing a good job in itself not fitting this data. However I want to quantify this 'goodness' with a statistical measuring technique.

I know that R squared is the most popular for regression, however I think that it is not the right approach for me. After trying r squared I am getting very low values because my algorithm is not fitting outliers which r squared does not take into account. rsquared does not know that I have outliers which the algorithm is on purpose not fitting with the rest of the data.

Is there another approach to this? maybe a different algorithm, or some changes I can make to r squared?

When you mention that your model does a good job in fitting the data (and not the outliers), I am assuming you are taking a look directly into actual vs predicted (or fitted) values. While doing this, you do not look for comparisons for outlier observations.

Similarly, you could still calculate the R squared by removing the outlier observations from the calculation, and it could still serve as a measure for goodness of fit. There is sufficient documentation available on how to do this manually (by hand) so you can write a function to do this by yourself.


Analyzing Videos using Deep Learning Is there any work done on analyzing sequence of frames from a video using Deep Learning techniques?

By "analyzing" I mean like memorizing them in order to classify or predict something (e.g. by taking into account first 10 frames of a video the model can make some sort of conclusion).

Yes there is. Actually using NNs on sequences is a big part of Deep Learning and one of the most powerful NNs come from this field of research. They are called Recurrent Neural Networks or RNNs. I would recommend you to read this article or watch this Stanford lecture, to learn more about them.

Especially a variant of RNNs, the so called LSTM that stands for Long Short Term Memory enables networks to memorize earlier inputs.

Based on your explanation I would suggest you have a many-to-one task where you input many frames and output one class. These types of tasks where you input images is most often solved using a combination of CNN and RNN. You could read this article to get more insights, to that technique.


What happens when we train a linear SVM on non-linearly separable data? What happens when we train a basic support vector machine (linear kernel and no soft-margin) on non-linearly separable data? The optimisation problem is not feasible, so what does the minimisation algorithm return?

I think basic support vector machine means hard-margin SVM. So, let's review:
What is a Hard-Margin SVM
In short, we want to find a hyperplane with the largest margin which be able to separate all observations correctly in our training sample space.
The optimisation problem in hard-margin SVM
Given the above definition, what is the optimisation problem which we need to solve?

The largest margin hyperplane: We want max(margin)
Be able to separate all observations correctly: We need to optimise margin and also satisfy the constraint: No in-sample errors

What happens when we train a linear SVM on non-linearly separable data?
Back to your question, since you mentioned the training data set is not linearly separable, by using hard-margin SVM without feature transformations, it's impossible to find any hyperplane which satisfies "No in-sample errors".
Normally, we solve SVM optimisation problem by Quadratic Programming, because it can do optimisation tasks with constraints. If you use Gradient Descent or other optimisation algorithms which without satisfying the constraints of hard-margin SVM, you should still get a result, but that's not a hard-margin SVM hyperplane.
By the way, with non-linearly separable data, usually we choose

hard-margin SVM + feature transformations
use soft-margin SVM directly (In practical, soft-margin SVM usually get good results)


How to deal with variable number of permutation invariant features? I want to learn from data where each record has a variable number of features that have no inherent order to them. Take as an example the task to predict whether a repair is worth it of some item. Each item can have a variable amount of features e.g. glass is broken, handle is missing etc. which are mostly categorical. There are up to +1000 different things to know about each item, but in reality there are only a handful present.
What ways are there to encode this kind of data or what models exist that can handle this natively?
I have a few things in mind

Binary encoding of set (resulting in a lot of dimensions)
Average features into single vector (dimensions become harder to interpret)

but I want to explore more options.

Technically your problem is not about a variable number of features, since you can have a finite list of all the possible features. The standard case is just to use all these features, even if only a few of them are "active" for a particular instance (your first option). If the number of features is too high, then you need dimensionality reduction.
The advanced option is to use a feature extraction method: an unsupervised method which "groups" similar features together, the goal being to reduce dimensionality while preserving the information contained in the features. By definition this method modifies the set of features: the "new features" are not directly interpretable, as opposed to the original ones.
Note that the other way to reduce dimensionality is feature selection: these methods don't modify the features, but they discard the least informative ones. Feature selection is supervised (as opposed to feature extraction) because informativeness is measured with respect to the response variable.


Theoretical and practical comparison of CTC and seq2seq loss in Tensorflow Tensorflow has built-in implementations for both, the Connectionist Temporal Classification (CTC) loss  and a special seq2seq loss (weighted cross-entropy).

Since CTC loss is also intended to deal with seq2seq mappings, I wonder about how the two loss functions compare. I went through an excellent explanation of CTC loss, finding out that the target sequence is restricted not to be longer than the input sequence whereas this restriction does not exist for seq2seq loss.

Some questions I have (ranked by relevance):

Are bidirectional LSTMs suited for seq2seq loss? I have only found b-LSTM implementations with CTC or even simpler loss functions. There is a post from 2016 saying that this was not possible at that time. Any updates?
Are there conceptual differences apart from the many-to-one restriction in CTC loss that seq2seq does not have? E.g. does the seq2seq loss make a similar conditional independence assumption like the CTC loss?
Do the loss functions have preferences regarding the optimizer (RMSProp, ADAM, Momentum etc.)
Do the loss functions work better for particular sets of tasks?

Theoretical arguments and practical experiences are both appreciated.

From my own work, I give a few, sparse empirical reports:

bLSTMs are suitable for seq2seq loss in Tensorflow in case they are used within the encoder of an encoding-decoding scheme. I have not tried to use bLSTMs within the decoder.
--
seq2seq loss could be trained almost equally well with both RMSProp and ADAM on both G2P and P2G tasks.
--


Ideal score of a model on training and cross validation data The question is little bit broad, but I could not find any concrete explanation anywhere, hence decided to ask the experts here.

I have trained a classifier model for binary classification task. Now I am trying to fine tune the model. With different sets of hyperparameters I am getting different sets of accuracy on my train and test set. For example:

(1) Train set: 0.99 | Cross-validation set: 0.72
(2) Train set: 0.75 | Cross-validation set: 0.70
(3) Train set: 0.69 | Cross-validation set: 0.69

These are approximate numbers. But my point is - for certain set of hyperparameters I am getting more or less similar CV accuracy, while the accuracy on training data varies from overfit to not so much overfit.

My question is - which of these models will work best on future unseen data? What is the recommendation in this scenario, shall we choose the model with higher training accuracy or lower training accuracy, given that CV accuracy is similar in all cases above (in fact CV score is better in the overfitted model)?

Accuracy on the training data basically doesn't count. I don't quite want to say to ignore it, because a train/test accuracy of 100/70 seems different to me than a train/test accuracy of 71/70, but you're mostly not interested in performance on the training data.

Using a test set mimics a real application of machine learning. Think about Siri or Alexa. The goal is to predict speech that it hasn't heard. There's no way to know how it will perform on such speech, so the next-best approach is to use some data where you know the answer but hide it from your model. After you train the model, assess how it performs on data where it has not seen the answer. If the model is accurate, then that's a good sign about its ability to perform on real speech recognition tasks.

Training data is like the practice questions or homework problems, while test data is the exam.


clustering before or after PCA? I'm newbie into data science, and I had some problems dealing with my project. I'm trying to visualize multidimensional data into 2D after clustering with using a lot of methods. (kmeans, DBSCAN, OPTICS, agglomerative, spectral...)

I have multidimensional data. (11 columns - attributes , 150K rows - number of data). It is slightly sparse-alike data, for example, which means one datum has numeric values like (0, 0, 6.5, 0, 0, 7.5, 0, 0, 4.5, 0, 0)

So, each datum has approximately 2~5 non-zero attribute values...

Below is not exactly same with my project, but it's similar.

https://scikit-learn.org/stable/auto_examples/cluster/plot_cluster_comparison.html

But, as I'm new to this, I am curious about the sequence of PCA and Clustering. I think there are 2 scenarios.

[1. Do clustering before PCA]

That means, I am using PCA just for visualization. But I have a question. In that case, If I process clustering with raw data, are all clustering algorithm (mentioned above) fit to my data type well.

[2. Do clustering after PCA]

In this case, I have other problems. My data's importance of components are like below.

                          PC1    PC2    PC3    PC4     PC5     PC6     PC7    PC8     PC9    PC10
Standard deviation     1.4173 1.1836 1.1141 1.0108 0.99109 0.95231 0.89091 0.8456 0.71542 0.64610
Proportion of Variance 0.2009 0.1401 0.1241 0.1022 0.09823 0.09069 0.07937 0.0715 0.05118 0.04174
Cumulative Proportion  0.2009 0.3410 0.4651 0.5673 0.66551 0.75620 0.83558 0.9071 0.95826 1.00000

As far as I've understood about visualizing multivariate data into 2D, I have to choose 2 PCs.(e.g.> PC1, PC2). However, my data's PV is slightly lower than I'd expected.

So, is it okay if I choose (PC1, PC2) to coordinates to be clustered and process clustering? Also, can I choose other PCs (e.g. PC5, PC8) to coordinates to be clustered?

This is often done to visualize if there is any structure in the data. Often you color the clustering differently to check if samples from the same cluster are close.
Often data contains a lot of redundant information. With many dimensions, you get the curse of dimensionality. This can lead to a few large and many small clusters. By reducing the dimensionality to a few informative features the clustering solution often improves. This is very dependent on the dataset. Perhaps all of your dimensions are informative then dimensionality reduction won't help much. Look at scoring metrics such as silhouette analysis/score, gap statistic, or elbow method.

In your case, most of the dimensions seem informative. Perhaps you can remove one or two dimensions, but it likely won't have much effect on the clustering solution.

You can choose whatever PC's you'd like but they are ordered according to the amount of variance they explain. You can look at how much the different original features contribute to the different PC's.


How to annotate labels in a 3D matplotlib scatter plot? I have made a 3x3 PCA matrix with sklearn.decomposition PCA and plotted it to a matplotlib 3D scatter plot.

How can I annotate labels near the points/marker? Here is my code:

from mpl_toolkits.mplot3d import axes3d
import matplotlib.pyplot as plt
%matplotlib notebook
fig = plt.figure()
ax = fig.gca(projection='3d')
ax.scatter(
       existing_df_3dx['PC0'], existing_df_3dx['PC1'], existing_df_3dx['PC2'],  # data
       s=60                                       # marker size
       )

plt.show()

Also, if you know of a better way to plot a 3D PCA, please post your code

Edit:

A solution provided from Stack Overflow:

for i in range(len(data_df_3dx)):
    x, y, z = data_df_3dx.iloc[i]['PC1'], data_df_3dx.iloc[i]['PC2'], data_df_3dx.iloc[i]['PC3']
    ax.scatter(x, y, z)
    #now that you have the coordinates you can apply whatever text you need. I'm
    #assuming you want the index, but you could also pass a column name if needed
    ax.text(x, y, z, '{0}'.format(data_df_3dx.index[i]), size=5)

In the following posts [1], [2] the plotting of 3D arrows in matplotlib is discussed.

Similarly Annotation3D class (inherited from Annotation) can be created:

from mpl_toolkits.mplot3d.proj3d import proj_transform
from matplotlib.text import Annotation

class Annotation3D(Annotation):
    '''Annotate the point xyz with text s'''

    def __init__(self, s, xyz, *args, **kwargs):
        Annotation.__init__(self,s, xy=(0,0), *args, **kwargs)
        self._verts3d = xyz

    def draw(self, renderer):
        xs3d, ys3d, zs3d = self._verts3d
        xs, ys, zs = proj_transform(xs3d, ys3d, zs3d, renderer.M)
        self.xy=(xs,ys)
        Annotation.draw(self, renderer)

Further, we can define the annotate3D() function:

def annotate3D(ax, s, *args, **kwargs):
    '''add anotation text s to to Axes3d ax'''

    tag = Annotation3D(s, *args, **kwargs)
    ax.add_artist(tag)

Using this function annotation tags can be added to Axes3d as in example bellow:

import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import axes3d
from mpl_toolkits.mplot3d.art3d import Line3DCollection

# data: coordinates of nodes and links
xn = [1.1, 1.9, 0.1, 0.3, 1.6, 0.8, 2.3, 1.2, 1.7, 1.0, -0.7, 0.1, 0.1, -0.9, 0.1, -0.1, 2.1, 2.7, 2.6, 2.0]
yn = [-1.2, -2.0, -1.2, -0.7, -0.4, -2.2, -1.0, -1.3, -1.5, -2.1, -0.7, -0.3, 0.7, -0.0, -0.3, 0.7, 0.7, 0.3, 0.8, 1.2]
zn = [-1.6, -1.5, -1.3, -2.0, -2.4, -2.1, -1.8, -2.8, -0.5, -0.8, -0.4, -1.1, -1.8, -1.5, 0.1, -0.6, 0.2, -0.1, -0.8, -0.4]
group = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 3, 2, 2, 2, 3, 3, 3, 3]
edges = [(1, 0), (2, 0), (3, 0), (3, 2), (4, 0), (5, 0), (6, 0), (7, 0), (8, 0), (9, 0), (11, 10), (11, 3), (11, 2), (11, 0), (12, 11), (13, 11), (14, 11), (15, 11), (17, 16), (18, 16), (18, 17), (19, 16), (19, 17), (19, 18)]
xyzn = zip(xn, yn, zn)
segments = [(xyzn[s], xyzn[t]) for s, t in edges]

# create figure
fig = plt.figure(dpi=60)
ax = fig.gca(projection='3d')
ax.set_axis_off()

# plot vertices
ax.scatter(xn,yn,zn, marker='o', c = group, s = 64)
# plot edges
edge_col = Line3DCollection(segments, lw=0.2)
ax.add_collection3d(edge_col)
# add vertices annotation.
for j, xyz_ in enumerate(xyzn):
    annotate3D(ax, s=str(j), xyz=xyz_, fontsize=10, xytext=(-3,3),
               textcoords='offset points', ha='right',va='bottom')
plt.show()


Compare similarities between two data frames using more than one column in each data frame This work started by comparing two columns in each data set in pandas.
Previous research:here

A lot of results online show how to compare 2 data frames with 1 column
I'm trying to learn how to compare and extract similarities between two data frames (same & different sizes if possible) using more than 1 column in pandas.
sample input:
df1=pd.DataFrame([[1,None],[1,None,],[1,None],[1,'item_a'],[2,'item_a'],[2,'item_b'],[2,'item_f'],[3,'item_e'],[3,'item_e'],[3,'item_g'],[3,'item_h']],columns=['id','A'])
df2=pd.DataFrame([[1,'item_a'],[1,'item_b'],[1,'item_c'],[1,'item_d'],[2,'item_a'],[2,'item_b'],[2,'item_c'],[2,'item_d'],[3,'item_e'],[3,'item_f'],[3,'item_g'],[3,'item_h']],columns=['id','A'])

 df1
        id  A
    0   1   None
    1   1   None
    2   1   None
    3   1   item_a
    4   2   item_a
    5   2   item_b
    6   2   item_f
    7   3   item_e
    8   3   item_e
    9   3   item_g
    10  3   item_h

df2
    id  A
0   1   item_a
1   1   item_b
2   1   item_c
3   1   item_d
4   2   item_a
5   2   item_b
6   2   item_c
7   2   item_d
8   3   item_e
9   3   item_f
10  3   item_g
11  3   item_h

What I've tried  so far:
   1: df1[df1.A.isin(df2.A) & df1.id.isin(df2.id)]

   2: df1[   df1[['id', 'A']].isin(df2[['id', 'A']])  ]

The output I got for 1 is close to what I desire:
    id  A
3   1   item_a
4   2   item_a
5   2   item_b
6   2   item_f #this specific row is not desired in the output
7   3   item_e
8   3   item_e #this specific row was raised due to a duplicate in `df1`. It's permitted to show duplicates. Duplicates values are allowed in `df1` but not `df2`.
9   3   item_g
10  3   item_h

Desired output:
    id  A
3   1   item_a
4   2   item_a
5   2   item_b
7   3   item_e
8   3   item_e
9   3   item_g
10  3   item_h

What's not shown: Two data frames have 2500+ rows. df1 can have the same items associated with an id. No duplicate items for an id in df2.
My 2nd try 2: df1[   df1[['id', 'A']].isin(df2[['id', 'A']])  ] is definitely the wrong approach as its matching row and column in df1 to row and column df2 (This output is similar to equals(), I get values from df1 instead of True and NaN instead of False)
Any code, links, suggestions are appreciated.

The last df is your required result, 2nd last is mine.
I think 3, item_f is not possible as its absent in df1.


How to build a model which can predict the probability of an event based on a set of timeseries data? I am trying to build a deep learning neural model using keras and tensorflow which can predict if a certain event will occur based on a set of timeseries data and some fixed data together . For example:
For a given set of entities say , Their price behavior is co-related due to rumor of some event that is going to happen in future.

1) A & B 's fixed data like type, group etc.
2) During a certain period of time - 1 Jan 2015 - 30 Mar 2015 , their price .

Data that I have is

INPUT : Name of entity, Type Of entity , Size , Country, Specific Attributes  and time series stock data from 1 Jan 2015 - 30 Mar 2015
OUTPUT : Y/N . Boolean output if event happend or not.

Now my question is how do I build this since I have some fixed data which doesn't changes over time and some time series data which changes over time.

Options that I thought of are
1) LSTM - But not sure if I should feed in fixed static data.
2) CNN - Not sure if it is the right approach ?

Please let me know what should be my approach to handle such a problem.

Since you have features that would be handled best with a recurrent neural net, AND some features that would be handled best with a feedforward net, what you can actually do is both and feed them into a main Dense layer which has a softmax output to give you the probability distribution.

This would be rather hard to do by hand, but luckily you are using Keras, which allows for this kind of modeling rather easily!

In the Keras functional API guide https://keras.io/getting-started/functional-api-guide/, there is a model actually very similar to what you are looking for, where the "Main" information is an LSTM layer (which you'd do for the stock prices), and the "Auxiliary" information would be (Name of entity, Type Of entity , Size , Country, Specific Attributes) etc...

The model looks like this:

The example model actually uses 2 loss functions (2 outputs), but you can easily build it to only have the one output. The code is all there so will be easy to replicate.

I basically use this kind of model for almost everything now and get great results, vs just LSTM alone.


NN Making Poor Averaging Fit whilst LGBM Regressor Fits Perfectly I have a simple toy dataset for which the features have been encoded using a Encoder-Decoder NN.
I am using the hidden feature vector from the Encoder as the X input for training a 1-step lookahead model of the data.
I know the feature vector from the Encoder is good, because when I use this X as input into an vanilla out of the box LGBM tree model it fits almost perfectly to the Y output.
However, trying to train a simple 2 layer NN network (ReLU activation in first layer, Linear in second) which should learn the relatively easy mappings of the X feature vector to the Y output, it basically just learns to just average the output.
Now there is a reason why I want to use a NN model to predict the output here rather than just using the LGBM model, so why (if the LGBM model can do it) is my NN struggling so hard.
Below is the Prediction/Target from the LGBM Model:

And here is the same but from the NN model:

The reference Pytorch code for the NN model is:
 alpha_hidden_1 = nn.Linear(encoder.hidden_size, hidden_alpha_size)
 alpha_out = nn.Linear(hidden_alpha_size, 1)

 x = F.relu(alpha_hidden_1(hidden_state_vector))
 x = alpha_out(self.dropout(x))

I'm training it for 1000 epochs and have tried varying various hyperparameters to no avail.

It looks like your neural network has a high bias. There are a couple of things you can do as a first step to attack this:

increase number of hidden layers
increase number of neurons
increase number of training epochs

Generally, neural networks have a lot of hyperparameters and so would be good to optimise using a gridsearch for example.
In terms of your first two models, it is of note that you need to double check they are not overfitting the training set.


How to read specific column with specific row in x_test using python I have a dataset with four input. here I am trying to predict X1 input in next time period using lstm model.

Here I put x_train 80% and x_test 20% .
In x_test first input some of rows are with 0 values.

What I want is I want to read that 0 value separately. I tried and it didn't work for me. Can anyone help me to solve this problem.
Here my code:

data=data.values
scaler_x = preprocessing.MinMaxScaler(feature_range =(0, 1))
x = np.array(x).reshape ((len(x),4 ))
x = scaler_x.fit_transform(x)
scaler_y = preprocessing.MinMaxScaler(feature_range =(0, 1))
y = np.array(y).reshape ((len(y), 1))
y = scaler_y.fit_transform(y)
train_end = 80
x_train=x[0: train_end ,]
x_test=x[train_end : ,]
y_train=y[0: train_end]
y_test=y[train_end :]
x_train=x_train.reshape(x_train.shape +(1,))
x_test=x_test.reshape(x_test.shape + (1,))

After writing code for lstm I tried to predict value for x_test. Before that I want to specify that rows with 0 values.

x_test_n = np.flip(x_test_n, axis=0)
curr_frame = x_test_n[0]

Then it just read first row with four input together.
What I want is to read first input with 0 values.

Image for what I got:

What I want to read

After  changing the code:

pred=[]
for index, row in x_test_n.iterrows():
  if index_row, index_col = np.where(x_test_n == 0)
    x_test_n.iloc[index] = pred[-1]
    row['X_test_n'] = pred[-1]
    pred1 = model.predict(x_test_n)
    pred.append(pred1)

It gave me an error:

  if index_row, index_col = np.where(x_test_n == 0)

For a 2D np array x_test_n, you could use index_row, index_col = np.where(x_test_n == 0) to return the indices with 0 values.


How to extract true positives data (complete row with data) after training and testing from test dataset? How do you extract true positive data from testing data after training and testing?
For example, in the test data, I have two rows and one row is true positives and the other is false negatives. However, I would like rows which only have true positive values. How do you extract the complete row from the testing data after training and testing?

One thing you can do is browse your prediction vector, get the indexes of "1" responses, and then check those indexes in y_test. if your y_test[index] is also a "1" class, then select the row by index in X_test
I tested this, it works for me. In my case, my X and y are pandas.DataFrame.
    import pandas as pd
    from sklearn.linear_model import LogisticRegression
    import numpy as np

    X_train = pd.read_csv("saves/cv_sets/X_train1.csv", sep=";", encoding="latin1")
    X_test = pd.read_csv("saves/cv_sets/X_test1.csv", sep=";", encoding="latin1")
    y_train = pd.read_csv("saves/cv_sets/y_train1.csv", sep=";", encoding="latin1")
    y_test = pd.read_csv("saves/cv_sets/y_test1.csv", sep=";", encoding="latin1")

    clf = LogisticRegression(class_weight="balanced", solver='lbfgs', C=0.1)
    model = clf.fit(X_train, y_train)

    pred = model.predict(X_test)

    pred1 = np.where(pred==1)

    TP_Indexes = []
    for k in pred1[0]:
        if(y_test.iloc[k][0] == 1):
            TP_Indexes.append(k)

    X_test_TP = X_test.iloc[TP_Indexes]


What will the required time to process 500GB of images using NVIDIA GEFORCE 930M GPU I have an image dataset of size 500GiB, and my system specs are NVIDIA GEFORCE 930M, 12GB of RAM and Intel Core i5.

I have the following questions:

Is it possible such a large dataset to be used in my local machine?
If yes, How much time will be required for one epoch or equivalently on iteration? Any links or reference on how to compute the required processing time will be helpful.
If my system is not good, what are the other possible solutions I have?

The large size of your data is acceptable for deep learning and big data projects. Your system is also acceptable, though it is not powerful. If you have enough hard disk to store them all, it will suffice which means you can train your network. The elapsed time for each epoch depends on multiple aspects. For instance, some elements which are important are, the batch size and your vectorized implementation, the bottle-neck between the disk and RAM, the bottle between RAM and GPU, the size of the model, the size of training data, the memory size of your GPU alongside the size of your RAM, the size of each data, the load which is imposed to your GPU by your OS, and so forth. The easiest way is to code your network and try it yourself.

As I've mentioned, by the current settings you can train your network, but you may not have very fast computation. However, you can use some techniques to faciliate your training phase as much as possible. For instance, you have two main bottle-necks. The first bottleneck, which exists between disk and RAM, can be dealt with using generators. Namely, you can employ generators to decrease the number of disk calls. The other bottle-neck, between RAM and GPU, can be handled using vectorized implementation of your neural network. After loading your network, you can find the appropriate batch size to use all available GPU memory.

I also want to point out that the current GPU you have may have space limitations. This can incur difficulties when your network is very large. In such cases, you won't be able to load your entire network to your GPU.


Ship detection on high resolution unlabeled image dataset I have been given a dataset of 4000 high resolution photos of  different size of ships entering a port, that need to be clustered in order to execute the following tasks
1)Ship detection
2)Count the number of ships in image
3)Find if the ship is parked or not
I am new to object detection and image recognition so i have the problem that the dataset is not labeled so i have to labeled to accomplish the above tasks.I am asking for guidance from people that faced similar problems

So, the question concerns how to go about detecting ships in images, count the number of ships in the image and get the model to predict whether the ship is parked or not.
From the problems you have posed, I think it is best that we implement supervised machine learning. This means that we need labelled data. For this, I would recommend taking a sub-sample of the data, making a data log which maps the images to the labels (e.g. the file name and then the number of ships and whether it is parked or not.).
In terms of preparing the images for ship detection, it would be a good idea to mask these images by drawing a box around the ship(s) in your subset of images.
When it comes to models, the obvious choice is going to be something which follows a convolutional network, for problems 2) and 3), you might need to use a CNN model as the encoder, which will encode the visual information into a hidden representation. From this hidden representation, you would then decode with a normal feedforward neural network.
For problem 2), you would get the overall model to output a scalar value, which will denote how many ships are in the image.
For problem 3), you would have a final 2-node softmax output layer, which will output a probability distribution over whether the ship identified is parked or not.
I think someone would need to fill in for me on problem 1, as I do not have much experience in the way of semantic segmentation (which is essentially what ship detection is in your case).


Word2Vec: Why do some dimensions of an embedding have an interpretation, and why does addition/subtraction of embedding vectors work? I'm reading about Word2Vec from this source: http://jalammar.github.io/illustrated-word2vec/. Below is the heatmap of the embeddings for various words. In the source, it's claimed that we can get an idea on what the different dimensions "mean" (their interpretation) based on their values for different words. For example, there's a column that's dark blue for every word except WATER, so that dimension may have something to do with the word representing a person.

Secondly, there's a famous example that "king" - "man" + "woman" ~= "queen", where the word in quotation means the embedding of that word.
My questions are:

I don't quite understand the mechanism as to how any dimension of an embedding goes on to have a tangible, interpretable meaning. I mean, the individual components of embedding vectors could've very well been completely arbitrary devoid of meaning, and the whole embedding approach still could've worked in that scenario, since we're interested in the vector as a whole. Is there an online explanation or a paper that I can look at to understand this phenomenon?
Why does this addition/subtraction of vectors to give the relevant embedding vector for "queen" work so nicely? In one source, the explanation is given as follows:

This works because the way that the neural network ended up learning about related frequencies of terms ended up getting encoded into the W2V matrix. Analogous relationships like the differences in relative occurrences of Man and Woman end up matching the relative occurrences of King and Queen in certain ways that the W2V captures.

This seems like a broad, vague kind of an explanation. Is there any online resource or paper that explains (or better yet, proves) why this property of embedding vectors should hold?

The dimensions for an embedding space are only be accidentally interpretable.
However, vectors through the space can be interpretable. That is why word analogies are possible in an embedding space . The addition / subtraction of word vectors describe another vector through the embedding space. For example, "king" - "man" + "woman" approximate the "queen" vector.
Words are consistently used in relationship to other words. Word embeddings can model these consistent relationship by finding which words co-occur together and projection the words into lower dimension space that retains the most common occurrences.


Why is the input to an activation function a linear combination of the input features? I'm new to deep learning, and there is something about the calculations I do not understand:
For a standard neural network, why is it that only the activation function is not linear, but the input to the activation function is a linear combination of each of the $x_i$'s? For example, with the sigmoid function, it would look like:
$$ \frac{1}{1+ e^{-(w_0x_0 + w_1x_1 + b)}} $$
where $w_i$ are the weights and $x_i$ represents the input to that layer.

For example, why is it that we don't have something like this:
$$ \frac{1}{1+ e^{-(w_0x_0^2 + w_1\sqrt{x_1} + b)}} $$

Is it because it would be redundant if we had enough layers? Or is it because a priori, you wouldn't know what the best function is?

The main reason is that a linear combination of the input followed by a non-linearity stacked on top of eachother is a universal function approximator. Which means that no matter how complicated the true underlying function is, a neural network can approximate it to an arbitrarily small error.

There's also the efficiency factor since a linear combination of $n$ inputs each having $m$ dimensions can be represented using a single matrix multiplication $h=X \times W$ where $X$ is an $n \times m$ matrix (where each row is an example and each column is a feature of that example) and $W$ is an $ m \times d $ weight matrix. And computers are VERY efficient at doing matrix multiplications. Thus, the more you build your model to use matrix multiplications the better.


Different accuracy of random forest and neural net I ran a set of time series data in  a neural net model and random forest model. For neural net I normalised the data split it into test 20%, validation and training 80%. Keras library neural net was used here.The accuracy is 85%(I get different accuracy every time).
For random forest I didn’t normalise the data and the test train split is 20% and 80%.I used scikit learn here. I am getting 93% accuracy .Here time series data is used as well. Am I not supposed to get similar accuracy as neural net? What am I doing wrong?
(I was reading some papers there they had similar kind of accuracy. That’s why I am asking. Also can random forest overfit? Is mine one overfitting?!

Different models will give different accuracies. Same model too could give you different accuracies. You can try setting the seeds to predefined values to bring in more consistency in the same model results


Extracting popular keyword terms or topics from blog posts based on post usage I have dataset of posts from blog and for each post I have the number of views.  I want to extract the topics (or phrases) that made the posts with more views.

I am planning divide all posts in two sets based on number of views (one set with low number of views and the other with higher numbers), then extract topics using LDA from each set and compare how they differ.

I am wondering if this is right approach and if there are other approaches that can be better or similar?

Seems right. However, establishing causality will not be as simple as extracting keywords and noticing the differences. And I would suggest not to divide the posts, instead club them together run LDA, extract the keywords, then analyse the differences. By separating you are introducing quite a huge bias into your model.


About neural network called "Self Organizing Maps" I have this simple neural networks ("Self Organizing Maps"):

What is the purpose of the lateral connections beetween the perceptrons (neurons) ?

Lateral connections exist so that the update of a neuron forces the neighboring neurons also to be updated but to a lesser degree.

Think of each neuron that has n inputs as a vector, a point in an n-dimensional space. It's coordinates are the values of the weights on its inputs.

When the network receives an input - also an n-dimensional vector, each neuron calculates the distance between its weights and the input vector. The neuron whose weights are the closest to the input is the winner, and is allowed to update its weights. It updates them by moving one step towards the input vector. The size of the step is equal to the learning rate.

While it moves, it uses the lateral connections to pull its neighbors (or push some of them away, depending on their distance to the winner and on the form of the function that you use for the lateral connections). As a result, the neighbors also move but they move less than the winner.

Only the winner and its neighbors are allowed to update their weights. All other neurons don't move. The neighbors are those who are closer to the winner than a certain radius. This radius is a hyper-parameter.

This type of learning allows to chart the space of the input vectors. The result is the weights of the neurons are distributed more or less uniformly in the set of the input vectors. After training, each neuron can be thought of as a representative of some region of the input space.


Feature relevance in PCA + kmeans algorythm Working on the World Happiness Report dataset, i have N countries with M features and a happiness score. This is the parameter I built 3 classes from: happy, medium, unhappy (numerical intervals of happiness score). I have run a 3 component PCA (only the first two components show relevant separation) and kmeans on the result (with 6 clusters, chosen in function of distorion). Plotting the two components with clustering predicted colouring gives very clear separation between classes, and I should now deduce what feature makes "happy" countries happy. My idea would be to implement some form of Bayes' theorem, but don't really know how since there is strong (>0.6) correlation between some features. How can I approach the problem?

If they are highly correlated, probably you can not easily tell which feature leads to a happy country. My suggestion is to perform multicollinearity test before fitting any model to remove highly correlated features. After that, there a chance that you be able to get more insights about the pattern in your data.


Doc2vec '-' symbol occurrence Currently working on resume parser and struggled with embedding words with '-' symbols in them. Such as 'IT-manager'.

Vector representations of these words are incorrectly classified by doc2vec.

  ['it-manager']
  [('salary', 0.23328335583209991), ('responsibilites', 0.22327110171318054), ('schedule', 0.14869527518749237), ('position', 0.12755176424980164)]

But when I remove '-' symbol, it is tokenized and classified right.

  ['it', 'manager']
  [('position', 0.9306046962738037), ('schedule', 0.6630333662033081), ('responsibilites', 0.6081600189208984), ('salary', 0.5934453010559082)]

How do you work with such data properly? For this kind of task, I guess, it is better to exclude the symbol. But there may be a way to tell Doc2vec to treat these words like two different ones. Or perhaps tell the word_tokenizer to tokenize them in this fashion?

gensim's Phrases module may also be helpful:

from gensim.models import Phrases
documents = [
    "the mayor of new york was there",
    "machine learning can be useful sometimes",
    "new york mayor was present"
]

sentence_stream = [doc.split(" ") for doc in documents]

bigram = Phrases(sentence_stream, min_count=1, threshold=2)

sent = [u'the', u'mayor', u'of', u'new', u'york', u'was', u'there']
print(bigram[sent])
# Expected output:
# [u'the', u'mayor', u'of', u'new_york', u'was', u'there']

That code is from this other answer (I've copy-pasted it above for convenience).

For more on the Phrases module, check this page out.


I am working on a pose dataset and want to convert this data into CSV format. How to convert this data into a single row of data This is the data for one pose. I have a lot of data like this which i want to convert to CSV to apply machine learning algorithm.
I am assuming I have to convert this data into a single row of comma separated values which will define a single pose.
Any suggestions will be appreciated.

You can use flatten numpy for this task. Simply convert your data to a numpy array and use the function as:

In [1]: import numpy as np

In [2]: x = [[[1,2], [3,4]]]

In [3]: x
Out[3]: [[[1, 2], [3, 4]]]

In [4]: np.array(x).flatten()
Out[4]: array([1, 2, 3, 4])


Keras input shape returning an error I am currently learning about Keras and have a problem with the input shape of a dense layer.I am currently trying to the mnist dataset.I understand that the input_shape for the train images is (60000,28,28) also i understand that keras ignores the first dimension as it is the batch size hence the input shape entered in the dense model should be (28,28) but when putting that i get an error yet when i put input shape of(784,) the model runs.Could someone please explain why is that so

(train_images, train_labels), (test_images, test_labels) =
 mnist.load_data()

 print(train_images.shape)

 network = models.Sequential()
 network.add(layers.Dense(512, activation='relu', input_shape=(28,28)))
 network.add(layers.Dense(10, activation='softmax'))

You always need to flatten your pictures when connecting the input to a Dense layer in Keras (Note that this is not the case for CNNs or RNNs). The reason is when the dense layer is built, based on the Dense layer code, the input dim is the last element you pass in the inputs (input_dim = input_shape[-1]). Therefore, although you are passing an input of (28,28) keras thinks that the shape is only 28. This also explains why the input of (,784) indeed works.

You can check the Dense layer code here


libraries for multiple machine NN training? As detailed here, the way to go to break NN training over multiple machines/threads, is decompose training data set on multiple chunks and send to each node, then sum results back in main node.
There is some library who already implements these techniques? Agents to install on each node?

TensorFlow and PyTorch both support distributed training.


How to find the initial hyperplane in a Support Vector Machine (SVM)? As far as I understand Support Vector machines, we are trying to find the optimal hyperplane, out of all hyperplanes that are equidistant from the support vectors. There are an infinite number of these hyperplanes, so we can start with an initial hyperplane, H1, which is not optimal. We can then vary the length of the normal vector w, so that we slightly change H1 into H2, which has a larger margin. We continue this process until we find H_max, the hyperplane with maximum margin.
My question is how do I find the initial formula of the first hyperplane H1? Given a linearly separable data set in the plane, how can I calculate a hyperplane that separates the data into two classes and is equidistant from the support vectors? Additionally, I am perhaps forgetting the significance of the variable b in this process.

Not sure how this is done in the any specific library, but here is what I would try.
Given two sets of points ($K$ points in the first set and $S$ in the second set): $\{\mathbf{a}_i\}_{i=1\dots K}$ and $\{\mathbf{b}_i\}_{i=1\dots S}$, all in $N$-dimensional space ($\mathbb{R}^N$)
I would compute the 'centres of mass' of the two sets:
$$
\begin{align}
\mathbf{A}=&\frac{1}{K}\sum_{i=1}^K \mathbf{a}_i \\
\mathbf{B}=&\frac{1}{S}\sum_{i=1}^S \mathbf{b}_i
\end{align}
$$
I would then use the mid-point between the two centres of mass,
$$\mathbf{M}=\left(\mathbf{A}+\mathbf{B}\right)/2$$
as the point for the hyper-plane.
Then I would use the vector connecting the two centres of mass,
$$\mathbf{C}=\mathbf{A}-\mathbf{B}$$
as the normal for the hyper-plane. Lets define
$$
\mathbf{\hat{n}}=\frac{\mathbf{C}}{\sqrt{\mathbf{C}.\mathbf{C}}}
$$
A single point and a normal vector, in $N$-dimensional space, will uniquely define an $N-1$ dimensional hyper-plane. To actually do it you will need to find a set of vectors
$$
\{\mathbf{v}_j\}_{j=1\dots N-1},\quad \mathbf{v}_j.\mathbf{\hat{n}}=0\,\mbox{for all } j
$$
This set can be created by Gram-Schmidt type process, starting from your trivial basis and then ensuring that every new vector is orthogonal to all vectors in the set and to $\mathbf{\hat{n}}$.
Once you did that, any point on the hyper-plane will be uniquely described by $N-1$ coordinates $\alpha_{i=1\dots N-1}$, and will correspond to the following point in the original $N$-dimensional space
$$
\mathbf{q}=\sum_{j=1}^{N-1} \alpha_j\mathbf{v}_j+\mathbf{M}
$$


Can I plug models like Linear Regression into a CNN feature map result? I was learning about image recognition on the Orange Software and I saw that I can feed my image database into a CNN(they call image embedding) that has as output a feature map of the image and then I can feed that feature map into models like Logistic/Linear Regression. And that is my objective, compare those models, but I want to do it without Orange.

I am thinking about using VGG16 as my CNN and extract the feature map from it. After that I want to plug those Linear/Logistic Regression to predict my image. Does that work at all? Is it possible?

(Using python and a proper labeled image database. Also I don't want to use the CNN alone, I really would like to use those 'simpler' models consuming the feature map)

What you're explaining is basically almost every CNN model where you basically have a fully connected layer at the end of the convolutions and that is equivalent to having a linear/logistic regression at the end (given there's only one output)

The only difference between all of those is whether there's an activation function or not, and if you are trying to build a classifier then you would definitely want an activation function to map the output values to a probability value between 0 and 1.

Example:

That "Fully Connected" layer is just basically a Linear/Logistic regression (10 of them actually since we have 10 outputs in this example) and a softmax applied to the output values to scale them all between 0/1 and ensure the sum is exactly 1.


Forecast vs Prediction: What is the difference? I use the two terms as follows:

A prediction model gets features (which can be a time series) as input and gives a fixed-length output (might be multiple values, but "atomic" in some sense)

Examples:

Dog vs Cat
Spam vs No Spam
Weather for tomorrow
Sales of next month

A forecasting model gets a time series as input and outputs a single time series.

Examples:

Amount of airline passengers: You can predict the value for the next month, but a forecast can be over an arbitrary time frame (with deteriating quality, of course)
Weather
Sales

I've never seen textbook definitions of those terms. Do you know any source that compares the two terms?

Bonus points if you give me a good German translation for both terms. By now, I call both "Vorhersage" - I have the impression there is no other German word for it.

Prediction is more general term.
It can be defined as a an expectation for a combination of different predictors (random variables).

Forecasting would be a subset of prediction and mostly related to time-series. Any time you predict into the future it is a forecast. All forecasts are predictions, but not all predictions are forecasts.

Moreover, forecasting always have error, that is why we have confidence interval of 5%, 2.5%, 1% and residuals for modeling. There is no such thing as so called "perfect forecasting", but we can maximize the model accuracy.

On contrast, prediction sometime can be perfect. I also believe that model should follow data not the other way around.


How to improve precision under imbalanced classification I am using an imbalanced dataset (rare positive cases) to learn models for prediction and the final good AUC is 0.92 but the F1 score is very low0.2.

Is it possible to add some key features which will change the class probabilistic distribution and thus we can get a threshold to generate a higher F1?

Here is an example:

In my original model, I get a class probabilistic distributions shown below:

I can adjust the threshold to make a better precision but meanwhile cut some recall off. It is due to the large overlapping area between two distributions.

Then I use an extreme dataset, i.e. include the target itself as a feature to learn. As a result, I can see I split the distribution completely disjointed.

Dose it mean if I introduce a strong feature, I can split the distribution to some extent and thus promote the precision and thus f1 score? Or please advise how to improve precision under imbalanced classification issue.

Many thanks

Introducing a Strong Feature would definitely help as it is "Strong" :-) .. if you do not have a sure-success feature then you may try changing penalty of miss-classification to start with.

You may try synthetic (i.e. SMOTE) or non-synthetic (domain based) approaches to bulk up the lower class.

Also, if this is a very very rare class the repeated sampling techniques may work


Learning Football Player Stats like FIFA's by only the game result It is a general question on how to learning representation of one entity but the dataset is mixed with a lot of other entities, which their statis are always waiting to be learnt.

The question is best be explained by an example. Let's say, the entity is a football player. How can we assign player stats to them by just learning from the match result.

Dataset: Football match data with startup line of the two teams and the final result score.

Player stats, something like this https://www.fifaindex.com/player/158023/lionel-messi/ but much simpler. We can define a player stats is just 3D vector v = (x , y , z) where x , y , z is between 0 to 1.0.  We don't need to know what actually x mean shooting power or anything.

So after learning, It output each player a v vector assigned.

Any idea or some material that you can suggest?

For that, you would probably need something more granular than just the score (e.g., in-game analytics, etc.) to help you assess the attributes of each player.  Even then, it would be difficult to map the outcome to multiple attributes (i.e., defending skills, offensive skills).

If you only care about a single score of a player (i.e., say, how effective is a player in helping her team to win, etc.), you could potentially look at something like Corsi which essentially measures puck possession (not without its criticism or flaws). There are many variants out there, but it should give you an idea how to leverage in-game analytics to assess a player.

But then, Corsi is merely a calculation from in-game statistic; you will need more than that to call it a machine learning exercise.


Can we fine-tune a model on the same dataset which it is pretrained on? So I was reading this paper (about a use case of pretraining then self-training) which got me thinking - suppose I pre-train a model on a particular dataset, then fine-tune it again on the same dataset.
Theoretically, if we pre-train it as a masked LM and fine-tune, it might lead to overfitting - but I am not sure. Maybe it won't be able to generalize well, but would still give an accuracy boost?
Does anyone know some research or other credible sources to explain why or why not this should be done?
Also, if this is indeed possible/advised do we have to take some certain extra steps in Tensorflow,  Keras, or HuggingFace (basically any Deep-Learning Framework or library) for doing so?
EDIT:- A simple example, I am saying like if we have a specific supervised task, and pre-train on the same features as we would fine-tune on - would that increase accuracy by any bit?

In the context of that paper, pre-train then fine-tune on the same dataset does not really make sense, as the pre-training is unsupervised, and the fine-tuning is with labelled data.
But, generally, if you have trained on dataset X for N epochs, and then you fine-tune one more epoch using the whole of X, it is just another way of saying you trained for N+1 epochs. Nothing wrong with it, except as you note in the question, if you were starting to overfit after N epochs, you are even more over-fitted now.
What does make sense, and something we have used in production models, is to do the initial training on a large dataset that is the combination of X1, X2, X3, ... and so on. Then once the learning curve starts to level out, we take a copy of the model and then fine-tune on e.g. just X1.
This is in the context of NLP Transformer models, so may not make sense for other domains, but we try to do initial training on as much data, from all domains, as possible, then we might fine-tune on just medical papers, or just economic reports, depending on what the model will be used for. (We've also done a final fine-tune on just a subset of the data to have it learn a specific style. It is amazing that even with just one epoch of this final fine-tune it can switch to using a different way of writing numbers, for instance.)


Data normalization before or after train-test split? Which one is the right approach to make data normalization - before or after train-test split?

Normalization before split

normalized_X_features = pd.DataFrame(StandardScaler().fit_transform(X_features), columns = X_features.columns)
x_train, x_test, y_train, y_test = train_test_split(normalized_X_features, Y_feature, test_size=0.20, random_state=4)
LR = LogisticRegression(C=0.01, solver='liblinear').fit(x_train, y_train)
y_test_pred = LR.predict(x_test)

Normalization after split

x_train, x_test, y_train, y_test = train_test_split(X_features, Y_feature, test_size=0.20, random_state=4)
normalized_x_train = pd.DataFrame(StandardScaler().fit_transform(x_train), columns = x_train.columns)
LR = LogisticRegression(C=0.01, solver='liblinear').fit(normalized_x_train, y_train)

normalized_x_test = pd.DataFrame(StandardScaler().fit_transform(x_test), columns = x_test.columns)
y_test_pred = LR.predict(normalized_x_test)

So far I have seen both approaches.

Normalization across instances should be done after splitting the data between training and test set, using only the data from the training set.

This is because the test set plays the role of fresh unseen data, so it's not supposed to be accessible at the training stage. Using any information coming from the test set before or during training is a potential bias in the evaluation of the performance.


Models that converged before aren't converging anymore in Keras I have two models with saved data that worked well previously but won't anymore.

First, it happened with one of my Jupyter notebooks. I can even load the saved model and weights that work. When I train more with the exact same model, the performance actually drops!
For example, I get a dice coefficient of -.39 with my previous training when it worked. Now if I load the same model, weights, and data, it drops to -0.04. (Loss of -1 is perfect).

So I load one of my older notebooks with a different model and saved data that worked well. It doesn't converge to nearly as high of a performance as it did previously either.

However, I tried setting up a simple MNIST CNN classifier and it worked fine.

Is there any way for there to be persistent changes to occur so that the exact same code/data that performed well before no longer does?

It was actually just the random weights and incredible luck that I got good convergence the first three tries in a row while didn't most other times afterwards.

I tried setting the seed and found certain seeds that gave me good results every time whereas most other seeds won't converge for many epochs.

from numpy.random import seed
seed(5)
from tensorflow import set_random_seed
set_random_seed(42)


Notion of cluster centers and cluster comparison in Density Based Algorithms I have done some research on clustering algorithms since for my goal is to cluster noisy data and identify outliers or small clusters as anomalies. I consider my data noisy because of my main feautures can have quite varying values. Therefore, my focus has been on density based algorithms with quite some success.

However, I am unable to grasp the idea of cluster comparison in such algorithms since the notion of cluster centers cannot be properly defined.

My dataset constists of network flows and I split the dataset in subsets based on an identifier. After applying clustering on each subset I want to be able to compare the clusters that are created on each subset so that I can compare the subsets themselves in some context.

Would appreciate some help from data scientist gurus on how to approach the concept of cluster comparison  or cluster center in such algorithms.

Thanks all!

You can either use the medoid, you can sometimes compute a centroid (and just ignore that it may be outside of the cluster), or you can do pairwise comparisons and take the average of that rather than comparing centers.


Prepare data for SVM, Is it valid to normalise the data before and after PCA dimension reduction Is it valid to normalise a dataset, reduce dimensionality with PCA and then to normalise the reduced dimension data.    Assuming this is performed on training data, should the same PCA coefficients be used to reduce the dimension of the test data.  Should the same max and min normalisation values be used for the test and training data. I have included a simplified example of the code I am using which may describe I said better. Thanks in advance.

%% Prepare Training Data

% Normalise training data
mindata=min(TRAINDATA); maxdata=max(TRAINDATA);
TRAINDATA = ((TRAINDATA-repmat(mindata,[size(TRAINDATA,1),1]))./(repmat(maxdata,[size(TRAINDATA,1),1])-repmat(mindata,[size(TRAINDATA,1),1])) - 0.5 ) *2;

% Perform PCA
mTRAINDATA = mean(mean(TRAINDATA));
TRAINDATA = TRAINDATA - mTRAINDATA;
[Cpca,~,~,~,~]=princomp(TRAINDATA,'econ');
EigenRange = 1:2;
Cpca = Cpca(:,EigenRange);
TRAINDATA = TRAINDATA*Cpca;
TRAINDATA = TRAINDATA + mTRAINDATA;

% Normalise training data second time
mindata2=min(TRAINDATA); maxdata2=max(TRAINDATA);
TRAINDATA = ((TRAINDATA-repmat(mindata2,[size(TRAINDATA,1),1]))./(repmat(maxdata2,[size(TRAINDATA,1),1])-repmat(mindata2,[size(TRAINDATA,1),1])) - 0.5 ) *2;

%% Prepare Test Data

% Normalise using first normalisation values from training data
TESTDATA = ((TESTDATA-repmat(mindata,[size(TESTDATA,1),1]))./(repmat(maxdata,[size(TESTDATA,1),1])-repmat(mindata,[size(TESTDATA,1),1])) - 0.5 ) *2;

% Perform PCA
mTESTDATA = mean(mean(TESTDATA));
TESTDATA = TESTDATA - mTESTDATA;
TESTDATA = TESTDATA*Cpca;
TESTDATA = TESTDATA + mTRAINDATA;

% Normalise using second normalisation values from training data
TESTDATA = ((TESTDATA-repmat(mindata2,[size(TESTDATA,1),1]))./(repmat(maxdata2,[size(TESTDATA,1),1])-repmat(mindata2,[size(TESTDATA,1),1])) - 0.5 ) *2;

Depending on the language of your choice, you may or may not have to normalise the data yourself - for example the e1071 package in R does this automatically for you. As this is built off the 'libsvm' library, it might be that this is also the case there - the library documentation is your best source. As for the normalisation values, you should definitely use the same [min,max] values from the training set for the test set as well. Also, I would reccomend reducing the dimensionality of your data first, than normalising before running a svm. hth


Is there a way to set a different activation function for each hidden unit in one layer in keras? I'm trying to set a different activation function for each hidden unit in a layer. Is this possible in Keras with 'Concatenate'?

If I get the point, you can use a similar code like the following:

from keras.layers import merge, Convolution2D, MaxPooling2D, Input

input = Input(shape=(256, 256, 3))

seq1 = Dense(1, activation = 'relu')(input)
seq2 = Dense(1, activation = 'sigmoid')(input)
seq3 = Dense(1, activation = 'tanh')(input)

acum = merge([seq1, seq2, seq3], mode='concat', concat_axis=1)

Depending on your task, specify concat_axis.


What is the difference between affinity matrix eigenvectors and graph Laplacian eigenvectors in the context of spectral clustering? In spectral clustering, it's standard practice to solve the eigenvector problem

$$L v = \lambda v$$

where $L$ is the graph Laplacian, $v$ is the eigenvector related to eigenvalue $\lambda$.

My question: why bother taking the graph Laplacian? Couldn't I just solve the eigenvector problem for the graph (affinity matrix) itself, like the guy did in this video?

PS: I made this same question in CrossValidated but I think this is a more appropriate channel. Forgive me if I'm wrong.

The concept is the same but you are getting confused by the type of data. Spectral Clustering as Ng et al. explain is about clustering standard data while the Laplacian matrix is a graph derived matrix used in algebraic graph theory.

So the point is that whenever you encode the similarity of your objects into a matrix, this matrix could be used for spectral clustering.

If you have standard data i.e. a sample-feature matrix, you can find the proximity or affinity or whatever you want to call it as a matrix and apply spectral clustering.

If you have a graph, this affinity would be anything like adjacency matrix, distance matrix or Laplacialn matrix and solving the eigenfunction for such a matrix gives you the corresponding result.

The point about using Laplacian rather than adjacency is to keep the so-called affinity matrix positive semi-definite (and normalized Laplacian matrix is a better choice as it gives you normalized eigenvalues between 0 and 2 and reveals the structure of graph much better).

So the long story short is that as long as you have a matrix containing the affinity of your data you can use spectral clustering in general. The difference is in details (i.g. the property of normalized Laplacian I just mentioned)


twitter data analysis? I am involved in twitter analysis data. I want to find trending topics in tweets  with some hashtags, like #finance or #technology. I have a hugh data set of tweets and now I need to analyze them.

I need to recognize topics, if there are. They way I'm approaching this is, first, performing a vector representation of each tweet, with a tfidf technique, and then, build groups of them based on their cosine similarity.

Are there common techniques in tweets analysis?

I believe that the algorithm that you want to use is something called a latent dirichlet allocation (LDA) model. This model is designed to uncover the topics in a corpus of documents.

Scikit learn has an implementation.

They even have a tutorial which teaches you how to extract topics. The tutorial also describes Non-negative Matrix Factorization (NNMF) as a method to extract the topics. I can't vouch for this algorithm, because I haven't used it personally (as opposed to LDA which I have used before), but from their tutorial NNMF does seem to give reasonable results.

Using cosine similarity will help you to group tweets that are most similar, but it wouldn't give you their topics. Which may be what you want? It really is hard to say, because only you would know how you should have the system behave. Unfortunately, that doesn't help you figure out what is trending, and you will need to do some heavy post-processing to make whatever algorithm you use spit out something that is useful to you.

Good luck!


R - Order each matrix inside a list of matrices I have a list where each element is a matrix, and I want to order each matrix individually

Order a matrix is as easy as this:

data <- data[order(data$value),]

But I am struggling to order a list of matrix individually

I have tried this with  some variations:

 for(i in 1:lenght(data)){
 data[i] <- data[i][order(data[i]$value),]
 }

data is the list that contains matrices

I have also tried with apply functions

Some ideas?

Thanks!!

All the code:

#The aim is to show one hospital for each state given an outcome
#in the posotion num of the ranking, its a learning exercise

rankall <- function(outcome, num = "best"){

if (outcome == "heart attack"){
  col <- 11
}else if(outcome == "heart failure"){
  col <- 17
}else if(outcome == "pneumonia"){
  col <- 23
}else{
  stop("invalid outcome")
}

data <- read.csv("outcome-of-care-measures.csv", colClasses = "character")

data <- data[, c(2, 7,col)]

colnames(data) <- c("Hospital", "State", "value")
data$value <- as.numeric(data$value)
data <- data[complete.cases(data),]
data <- split(data, data$State)

#data2 is the final result dataframe
data2 <- data.frame(matrix(nrow = 0, ncol = 3))
colnames(data2) <- c("Hospital", "State", "value")

#From here is what I do not know how to do it.

#data is a list of matrices, each matrix corresponds to a state
#data2 is the result dataframe, where I am going to add the row num of each
matrix in data

#1. Order each matrix in data by value and alfabetically if tie
#2. Add to data 2 the row num of each matrix in data once they are sort

for(i in 1:lenght(data)){

data[i] <- data[i][order(data[i]$Hospital),]
data[i] <- data[i][order(data[i]$value),]

if(num =="best"){
  num = 1;
}else if(num == "worst"){
  num = nrow(data[i])
}

num <- as.numeric(num)

rbind(data2,data[i][num])

}

return(data2)

}

Sorry about identations, it changed when I paste the code

Because I did not have your exact data (I don't have the csv file), I created a dummy dataframe (swiped from this tutorial):

numPeople = 10
sex=sample(c("male","female"),numPeople,replace=T)
age = sample(14:102, numPeople, replace=T)
income = sample(20:150, numPeople, replace=T)
minor = age<18

population = data.frame(sex=sex, age=age, income=income, minor=minor)
population

> population
      sex age income minor
1    male  34    130 FALSE
2    male  48     86 FALSE
3  female 101     36 FALSE
4    male  64     78 FALSE
5  female  97     92 FALSE
6    male  19     45 FALSE
7  female  42    124 FALSE
8  female  24     68 FALSE
9  female  24     96 FALSE
10 female  91     30 FALSE

This means you will have to adjust the variable names and so on to match your example, but can learn how to order a list of matrices. More specifically, you really have a list of data.frame objects, after using the split function. it is important to notice the difference between a data.frame and a matrix, as they can behave differently.

I will split the data.frame by the sex column:

l = split(population, population$sex)

You can test that each element of the resulting list is indeed a data.frame (and not a matrix) by running:

> is.data.frame(l[[1]])
[1] TRUE

Notice that you have to use the doublesquare brackets to access the actual element of the list. Single square brackets will return something else:

> is.data.frame(l[1])
[1] FALSE

Tt is a named list-element and needs to be referenced by using (in my case) one of the names of the group that we split on: male or female
Below I comment the code heavily to explain what each step does.

> l[1]
$female                          # this is the name of the list element
     sex age income minor
1 female  20     42 FALSE
6 female  75    103 FALSE
7 female  30    148 FALSE

So we can access the actual element two ways:

by name:

> l[1]$female
     sex age income minor
1 female  20     42 FALSE
6 female  75    103 FALSE
7 female  30    148 FALSE

by double square brackets:

> l[[1]]
 sex age income minor
1 female  20     42 FALSE
6 female  75    103 FALSE
7 female  30    148 FALSE

That all being said, he is a working example of sorting a list (vector, holding lists) containing data.frames:

l = split(population, population$sex)    # population is defined as shown above
N = length(l)

results <- vector("list", N)

for(i in 1:N) {

    results[[i]] <- l[[i]][order(l[[i]]$age),]
}

I specify the length of the vector when I create it, as we know how long the resulting list will be in advance.
... and check the results:

> results[[1]]
    sex age income minor
8 female  66     90 FALSE
5 female  69    131 FALSE
2 female  70     52 FALSE
9 female  95     43 FALSE

> results[[2]]
    sex age income minor
6  male  18     87 FALSE
4  male  61     97 FALSE
3  male  64    131 FALSE
7  male  72     93 FALSE
1  male  82    107 FALSE
10 male  92     42 FALSE

The results depend on random choice when creating population at the beginning, using sample().


Arima and Sarima in python I wanted to ask I am usually using R when I do time series but I am trying out python now.

When doing the Arima model I use ARIMA function
But Can I also use SARIMAX function for just Arima models?

When I do SARIMAX order $= (2, 0, 4)$ (noseasonal parametrs) and then when I do ARIMA order $= (2, 0, 4) $

Outputs are not the same even thou the model should be the same?

What am I missing?

It seems you are passing the same parameters to both Python classes: ARIMAX and SARIMAX... and this should produce the same answer, I agree.

The implementations are however separate, it seems - ARIMA inherits from ARMA models and SARIMAX inherits from MLEModel. Have a look at the source code for ARIMA (search fro "class ARIMA") and SARIMAX (search for "class SARIMAX").

There are probably some different ordering of the operations, or different limits on the tolerances for MLE.


Mask RCNN: Random predictions during inference for the same image I recently trained the Mask RCNN (matterport's implementation) on some satellite images, but during inference mode, I'm getting random predictions for the same set of weights for the same image.

That is to say, for a given set of weights and the same image I'm getting different bounding box and mask predictions.

Does anyone have an idea of why this is the case?

The following are the different predictions that the model produced for the same image.

On some cases, the model produces reasonable predictions like that of the 2nd image but other times it produces completely random results. I can't figure out where this randomness is coming from.

I would highly appreciate any insight into this problem.

The problem was that some layers were not getting their weights loaded and were being initialised randomly. The random initialisation of those weights were the cause of random predictions during inference.


What does xgb's scale_pos_weight parameter do for regression? From other posts (see Unbalanced multiclass data with XGBoost) and the documentation, scale_pos_weight in XGBoost appears to balance positive and negative cases, which seems to apply only to classification. However, it also appears to be an option in XGBRegressor (see  documentation). Before I dive into the source code, can someone explain what this does for regression?

There are a few unused or deprecated parameters in both XGBClassifier and XGBRegressor, so it might just be a matter of sloppy inheritance/c+p.

A couple of possibilities:

They copied over the params from Sklearn's GradientBoostedClassifier
They copied over the params from XGBClassifier
They inherited the properties from some class that already had those attributes.

There have been some consistency issues for a while now (I faintly recall nthreads versus n_job issue.)


How do I use keras NN to classify data after training? I have defined, trained and saved my tensor keras NN. Now that that is complete how do I use it output classifications to non training data?

import tensorflow as tf
import numpy as np
from tensorflow.keras import layers
from syslog import syslog_pred

model = tf.keras.Sequential()
# Adds a densely-connected layer with 64 units to the model:
model.add(layers.Dense(128, activation='relu'))
# Add another:
model.add(layers.Dense(128, activation='relu'))
# Add a softmax layer with 8 output units:
model.add(layers.Dense(8, activation='softmax'))

model.compile(optimizer=tf.train.AdamOptimizer(0.001),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

model.load_weights('.my_model')

x = np.array([arr[:-1] for arr in syslog_pred], dtype=np.float32)
dataset = tf.data.Dataset.from_tensor_slices(x)

answer = model.predict(dataset, steps=30)
print(answer)

The code at the end isn't what it should be but I'm a little lost.
Any help would be appreciated!

Once you have a trained model, you can pass new samples to it by using the predict method of the model, which will get you the probabilities of all classes. You than pick the class with the highest probability for each sample as the predicted class:

y_prob = model.predict(new_data)   # Get class probability vector for each sample
y_class = y_prob.argmax(axis=-1)   # Set prediction to class with highest probability

If you are using a Sequential() model, you can also use the predict_classes method, which will get you the same answer:

y_class = model.predict_classes(new_data)


Hierarchical Clustering customized Linkage function In my clustering project, I need to customize the linkage function, so that after each cluster merging I can update the inter-cluster distance in my own way.

Currently I'm using scikit-learn AgglomerativeClustering, which seems not having this customizable feature. After a quick glance in scipy, no luck there either. Does anyone know any python hierarchical clustering toolkit that has customizable linkage?

Fork sklearn and implement it yourself! The linkage function is referenced in cluster/hierarchical.py as

  join_func = linkage_choices[linkage]

and

  coord_col = join_func(A[i], A[j], used_node, n_i, n_j)

If you have time, polish your code and submit a pull request when you're done.


What is the next step after k fold CV? I came across this video lecture https://www.youtube.com/watch?v=wjILv3-UGM8
on k fold cross validation (CV). The algorithm given in the video lecture is presented below:

for k = 1:5
train on all except k
get model $M_{\tilde{k}}$
calculate accuracy on $k$ as $A_k$
end
Calculate final cross validation accuracy: $A =
> \frac{1}{5}\sum_{k=1}^5 A_k$

This is quite clear to me. Here  $M$ is I guess just a single type of ML algorithm. However at time stamp 6:35 the presenter raises the question that what do we do with all the 5 different models that were built? According to him, we either combine all the models and make decision based on that or take the best model out of the 5. Is this statement true?
In many sites including here (https://stats.stackexchange.com/questions/310953/doubt-about-k-fold-crossvalidation?noredirect=1&lq=1 ;  https://stats.stackexchange.com/questions/11602/training-on-the-full-dataset-after-cross-validation and https://stats.stackexchange.com/questions/11602/training-on-the-full-dataset-after-cross-validation) and research papers I have understood that:
-- for doing model training using k fold CV, we re-train on the entire dataset after the end of the CV loop and that is the final model.
-- We do not select any model from inside the CV loop if the idea of doing CV training is to check the accuracy of the ML algorithm on the entire dataset.
-- However, if we have multiple ML algorithms say random forest, neural network, SVM inside the CV loop then we select the algorithm with the highest accuracy.
-- Another technique, nested cross-validation is used for hyperparameter tuning.
Is my understanding correct?

I didn't watch the linked video but based on your explanations: yes, your understanding is correct.
A common confusion is to assume that cross-validation is similar to a regular training stage and therefore produces a model. This assumption is wrong: CV includes repeated training/testing for the purpose of evaluating the method/parameters. From this understanding it follows that:

for doing model training using k fold CV, we re-train on the entire dataset after the end of the CV loop and that is the final model.

Yes, since we want to obtain the final model as accurate as possible so we should use all the data. In this case the CV has been used to calculate a good estimate of the performance.

We do not select any model from inside the CV loop if the idea of doing CV training is to check the accuracy of the ML algorithm on the entire dataset.

Correct, otherwise there's no point using CV.

However, if we have multiple ML algorithms say random forest, neural network, SVM inside the CV loop then we select the algorithm with the highest accuracy.

Any case where multiple methods and/or parameters are being evaluated is a bit more complex than the regular case of a single method: evaluating multiple systems is by itself an additional layer of training, in the sense that we select some parameters (typically the best model) based on the data. This means that the selection itself is based on the whole data used in the CV stage, so the CV performance of the best model is akin to a performance obtained on a training set. This is why one needs another test set (or nested CV) in order to obtain the final performance of the model. An intuitive way to understand this is to imagine evaluating say millions of models with CV: the only way to know if the best performance is due to chance or not is to evaluate the corresponding model on some fresh test set.
Note: the case of combining the outputs of all the models is a different story, since this boils down to a single meta-model.


Python clustering and labels i'm currently experimenting with scikit and the DBSCAN algorithm. And i'm wondering how to combine the data with the labels to write them into a new file. I'd also like to understand how the labels array is used to filter the examples.

Please correct me anytime i say something wrong because i'd like to understand the whole process better.

For example my data looks like this:

city    x    y
A       1    1
B       1    1
C       5    5
D       8    8

So if i understand it correctly i first need to split the data i'd like to cluster. If i use the data like above it will consider the city column aswell (or even fail).

So my next array would look like this:

x    y
1    1
1    1
5    5
8    8

Now i'll use the DBSCAN on my Array and it will create a cluster model. The clusterlabels are now stored in the array foo.labels_.

As far as i know i can filter the data with those labels to get the items within the clusters. Let's assume my data is in the dataframe cities:

cluster0 = cities[foo.labels_ == 0]

What i don't understand is how this works. I somehow don't get what exactly happens here. I know that the labels are in an array where the column is a number and the value of the column is the cluster. So how do i get the correct index of my cities?

So after the clustering i'd like to export my data back into a CSV file with the following format:

city    x     y     cluster
A       1     1     0
B       1     1     0
C       5     5     1
D       8     8     2

My guess is to use to original dataframe and add another column like this:

cities = cities.assign(cluster=p.Series(labels_))

But i'm absolutely unsure if that's the correct way to achieve what i want to do.

I'd really appreciate some opinions and explanations.

As the algorithm should not change the order of the lists you could just add the clusters list

 cities["cluster"] = cluster

If you are really paranoid you can add your input parameters a second time to the dataframe in the same way and compare the diff in values (should be 0).


Are there any multi-type public datasets out there? Does anyone know of any good public datasets for supervised learning that would contain data where instances are described with different types of data (ie. timeseries data, categorical data, images, ...)

The types of datasets I have in mind are:

a dataset where each example is described with an image and some numerical features
a dataset where each example can be described with timeseries data and some categorical features
a dataset where each example is described with multiple images (ie. the image taken from two different angles)

An example would be a dataset of daily closing stock price data associated with news sentiment for that particular day.

I'm aware that it's possible to construct a dataset like that on my own with some scraping but I'm wondering whether there are any public datasets out there that serve this purpose.

You may want to take a look at: https://archive.ics.uci.edu/ml/index.php which is a great repository with tons of datasets.


Definition of likelihood function I am confused by the definition of the likelihood function in different contexts.

In the case of linear and logistic regression, it is defined as y given x

In the case naive bayes and LDA, it is defined over x and y

In the case of the EM algorithm (with observed variable x and unobserved variable z), it is defined over x

How do we know over what it should be defined?

I also found confusing the definition of likelihood the first time I encountered it. The only thing you need to remember is that likelihood refers to the probability of your data given a parameter. That is,

$$L(\theta| X) = P(X|\theta)$$

As you can see, the likelihood focuses on the parameter, so you're asking "What is the likelihood of this parameter given some observations?" In this case, the phrase "given some observation" assumes the observations $X$ are fixed and the parameter $\theta$ is not fixed. On the other hand, if you describe the same in terms of probability, you're trying to measure the probability of some observations $X$ given a fixed parameter $\theta$. It is only a matter of usage.

In the case of logistic regression, the likelihood function is described as:

$$L(\beta| y) = \prod_{i}^{N} \pi_{i}^{y_{i}}(1-\pi_{i})^{n_{i} - y_{i}}$$

where $\beta$ is the vector of parameters $\{\pi_{i}| i=1,...,N \}$and $y$ is the vector of Bernoulli variables. As you can see, this is the same (by definition) as $P(y|\beta)$ and that is what you have in the equation above: The product of the probabilities for each observation $y_{i}$ given its parameter $\pi_{i}$. As usual, each probability $\pi_{i}$ is equal to the inverse of the logit function.

In Naive Bayes, you have the same basic idea. The likelihood is defined as $L(p_{k}|x) = \prod_{i}^{N}p_{ki}^{x_{i}}(1-p_{ki})^{(1-x_{i})}$ where $p_{k}$ is a vector of parameters describing the probabilities of a class $k$ generating a word $x_{i}$).

In the EM algorithm, the likelihood is again $L(\theta|X,Z)$, where $\theta$ are your parameters, $X$ your observations and $Z$ are latent variables. The fact that you have a latent variables shouldn't confuse you, because this variable will be "integrated out"  in the expectation step of the algorithm.

In the previous equations, the calculation you need to do is self-evident if you think about it starting from your observations: First, I have some observations. Then I want to calculate the probability of obtaining these observation using some model. Most models require parameters to be fitted. So I want to calculate the probability of these observations given those parameters. That is, $p(X|\theta)$. Now, if you need to talk about likeliihood, then you simply write $L(\theta|X)$.


Wave form analysis ML algorithm I am trying to determine a person's emotions from their speech. This immediately rings Machine Learning bells and the first step in any ML problem is getting and processing data. My first question is, how do I know what data to provide a machine learning algorithm? Would I supply the entire sound file annotated with a score of "urgency"? Or would I break the sound file down into pitch and periodicity etc.? I guess what I am asking is, how do I quantify the level of significance of certain elements/dimensions of a piece of data in training a machine learning algorithm?

My second question, is what sort of machine learning algorithm would I use? I am relatively new to ML although I have some experience with TensorFlow, so even just a general concept would be helpful. The machine learning algorithm also needs to be applicable to real-time audio and return results within around 2000ms.

Thanks for the help, please let me know if I should update any component of my question with greater specificity!

First, for your problem you can read https://www.sciencedirect.com/science/article/abs/pii/S0031320310004619
Unfortunately there are no closed answers for these questions... One of the major tasks of a ML engineer is to decide what algorithm to apply and what features from the original data to use to solve a specific problem... Both decisions affect the other. Almost every single decision in ML is strongly related to others... Other important things to bear in mind are the amount available data, hardware constraints and so on...

how do I know what data to provide a machine learning algorithm?

In classical ML, the engineer has to decide which features to compute. My advice is consulting expert's papers and contributions that solved similar problems and take advantage of their expertise. Features are dependent on the specific problem, domain and data. In audio processing there are quite well known: MFCC, Spetrograms...
In audio, a typical approach is to divide the audio signals in overlapping frames and compute the features over them. In that way, the evolution or dynamics of the audio features can be analysed over time. Taking the spectrogram is an example of that, it can help analyzing the evolution of the frequency components.
Data preprocessing is really important too: audio can be normalized or filtered as a previous step to feature computation.
In Deep Learning, neural networks try to compute the best features for you. However, a raw audio signal is rarely directly fed into a ANN. Typical procedure here is to compute the spectrogram (which can be interpreted as an image) and use Convolutional Neural Networks or Convolutional LSTM networks, or even Transformers.

what sort of machine learning algorithm would I use?

Depending on the problem, there are state of the art methods for image classification, detection, speech recognition...
The first thing you need is to really well define what do you want to do, knowing the problem... then, once again visit other scientists approaches.
Once you have a clearer idea, if you have hardware and computing time constraints, you need to figure out what kind of solution will adapt to them.
Theses comments may not seem very clarifying, but you dropped two really broad and difficult to answer questions.