Poker tournament winner prediction I am trying to solve poker tournament winner prediction problem. I’ve millions of historical records in this format:

Players ==> Winner
P1,P2,P4,P8 ==> P2
P4,P7,P6 ==> P4
P6,P3,P2,P1 ==> P1

I want to find the best algorithm to predict winner from set of known players.
So far I have tried decision trees, XGboost without much success.

I’ve  done my research and could not find answer anywhere else.My apologies in advance if same problem is answered in different terms on stack-overflow.

As oW_ mentioned, this can be done with the Elo rating system.

More specifically, given a record of wins and losses for some player pool, you can fit ratings for each player such the expected result of player $A$ rated $r_A$ vs player $B$ rated $r_B$ is given by:

$$
E_{A\ vs\ B} = \frac{1}{1 + 10^{(r_B-r_A)/400}}
$$

When the only possible outcomes are win or lose then this is also the probability of $A$ beating $B$.

If we define the quantity $\gamma_i = 10^{r_i/400}$ then the above can be expressed as:

$$
E_{A\ vs\ B} = \frac{\gamma_A}{\gamma_A+\gamma_B}
$$

Furthermore, Elo ratings generalize to multiplayer competitions, such that the probability of player $A$ beating players $B$, $C$, and $D$, for example, is given by:

$$
E_{A\ vs\ B,C,D} = \frac{\gamma_A}{\gamma_A+\gamma_B+\gamma_C+\gamma_D}
$$

The simplest method for computing ratings of a pool of (previously unrated) players given their game histories is to give them all the same initial rating, say $0$. The number doesn't matter because the relative rating differences (which are used to estimate win probabilities) will be the same. Next, for each result, $S$, of player $i$ vs player $j$ in the data set make the following update:

$$
r_i' \leftarrow r_i + K \times (S - E_{i,j})
$$

where $S = 1$ if $i$ beat $j$, otherwise $0$, and $K$ is a factor chosen to determine how quickly ratings change. For simplicity you can use $K=32$.

Remember to perform the update in both directions, so that if $i$ beats $j$ then $i$'s rating increases while $j$'s decreases. Use the pre-updated rating for player $i$ when computing player $j$'s rating change, so that the result is symmetric. It's also typical to use the pre-updated ratings for all rating update calculations in a single tournament. So you would keep track of all deltas for each player and apply them at the end.

To handle multi-player tournaments, simply log one win for the winner against every other player. In your case you only have the winner, so this is the best you can do. (If you had a complete ranking you could try Tom Kerrigan's Multiplayer Elo method or Microsoft's TrueSkill.)

You also need to divide $K$ by the number of players minus $1$ in your update rule:

$$
r_i' \leftarrow r_i + \frac{K}{n-1} \times (S - E_{i,j})
$$

Here is a Python library that performs the calculation I have described.

An alternative approach is the TrueSkill rating system, which handles multiplayer games nicely. It expects a ranking of everyone in the tournament, however you could try it with the winner ranked first and everyone else tied for second. Here is a Python TrueSkill library.


Scale of the data after PCA I have 4 standard normal features on which I perform PCA. I then take the first principal component (with all of the components). Is it possible to a priori say what is the max and the min value that the transformed series will have?

I guess, if we assume that the original standard normal features never exceed +/- 5, then the max of the final transformed series would be the PCA coefficients product summed by 5? But is it possible that the PCA coefficients are [1, 1, 1, 1] in this case? Or is there some sort of a bound on what the PCA coefficients can be?

The short answer is that you can determine the max and min of your transformed series if you normalize after your PCA step, as your title implies.

Have you tried L2-norm after your PCA step? This might get you what you are looking for by constraining the values between 0 and 1.

You can, of course, customize this to suit your needs.


Why not higher accuracy in Otto data? On this site of Otto Group Product Classification Challenge, it is shown that best accuracy was possible with RandomForest method, but it was relatively low at 0.83. Accuracy with ANN and with Naive Bayes was even lower (0.72 and 0.65, respectively).

What is the cause of low accuracy achievable in some data? Are there any other methods with which accuracy can be increased here?

Edit: On this site: https://www.kaggle.com/c/otto-group-product-classification-challenge/leaderboard best model seems to have a score of 0.38. What will be corresponding accuracy for that?

I can mention two main reasons.

Complexity of dataset
high Bayes error

The former means your dataset is highly non-linear and the latter simly means you may have same input with contradictory outputs in your dataset.


Optimization technique for iterative distribution Apologies if this does not fit the proper format for this site, as it is a somewhat general question.
I have an application that sits on top of a SQL database, and need to handle a process which is very much like a linear algebra problem, whereby an amount a (any number between 0-1B) needs to get distributed among n entities (e), based on a few parameters (rank, weight, min/max requirements) set at the entity level.
Example:
a = 100

entity    weight    min    max
----------------------------------------------
X         0.25      10     40
Y         0.75      40     60
...

In this example, 25 (a * X[weight]) would go to entity X, and 75 would go to entity Y; however, 75 exceeds Y[max], so the remaining 15 need to go to another entity (in this case, X, since it stays at or under X[max]).
Intuitively, this is in iterative process. In a real example, there would be more entities, so more iterations would be necessary. SQL is not designed to iteration. I am looking for a way to better handle this in a set based method.
What I am looking for is something along the lines of:

a statistical method that I can use to minimize the number of iterations I need to make, or even better, a way that this can be distilled into a formula?

alternatively, maybe there is a way to store some of the data in a static way to minimize the steps needed to make the calculations on the fly?

Creating a lookup table, whereby for each entity there could be a stored outcome for min/max ranges, based on the other entities (they are in groups of 10 or less).

It looks like a resource allocation problem. I can't think of any statistical method which would help here, but maybe there is.
I think the process can be simplified by first calculating the amount in excess and in "shortage". I don't know if there's a formal method for this but I would try something like the following:

Distribute $a$ based on the weights. Then calculate for every entity:

its excess amount: X has 0, Y has 15 (75-60)
its missing amount: X has 0, Y has 0.
its capacity for additional amount. X has 15 (40-25) and Y has 0.
its capacity for giving away some amount. X has 15 (25-10) and Y has 20 (60-40)

Calculate the sum of excess amount $S_{excess}$, sum of missing amount $S_{missing}$, and capacity $S_{plus}$, $S_{minus}$. Let $S = S_{excess} - S_{missing}$:

If $S>0$

If $S>S_{plus}$, then no solution
Otherwise (1) fill every entity which has amount missing to their minimum, (2) distribute amount S by filling any entity which has capacity for more.

If $S<0$

If $|S|>S_{minus}$, then no solution.
Otherwise (1) unload every entity which has amount in excess to their maximum, (2) take total amount |S| from any entity which capacity for giving away.

If I'm not mistaken this requires only two steps, each step going over all the entities.


Which tool should I use for combining this large dataset? This is my first foray into data science and I've hit a snag before even getting to the analysis.

I have 40 CSV files; each one contains 2 columns - a time and value column.

I would like to consolidate this into one table by doing an outer join of the time column for all files, so that the final file would contain 1 time column and 40 value columns. I attempted this using pandas merge method, but my local machine ran out of memory before it could complete. I made sure there was nothing fundamentally wrong with the code by simply combining 8 of the 40 files and got the desired result.

At that point I decided to spin up a more powerful cloud compute instance on AWS; I chose one with absurdly high RAM so I'm working with 190 GB rather than 8. It got further, but got the same memory error around the 30th file. I should also mention that each file as quite a few rows - around 180K.

At that point I decided that I must be going about this the wrong way. I don't think pandas is meant as a tool for combining such large datasets. In my last job I used SQL pretty extensively and something like that seems more equipped. My next idea was to try to do it in AWS Athena, which is a SQL like service that can integrate with the csv files in S3. I assume there is no standard solution to this issue, but I just want to see if I'm way off base or going in the right direction.

Thanks!!

I would suggest trying to do it in batches. The underlying issue could well still be memory related in some way, as the merge method makes copies of its input and so is not memory efficient at all.

As an example, you could read in 10 files, create the desired output, as you have done already. Repeat this for files 10-20, then 20-30 then 30-40. The finally four the four files that you have created. It is a bit of an annoyance, but sometimes these little workarounds get the job done.

[EDIT]

Another option might be to use the more involved memory management during reading, option via the chunksize argument of pd.read_csv(). This will read parts of the file into memory in chunks, as the name suggests. If you do this in a loop, it should put an upper limit on the memory usage. For example (untested):

chunksize = 50e6        # 50 Mb

for single_file in list_of_file_paths:
    for i, chunk in enumerate(pd.read_csv(single_file, chunksize=chunksize):
        if i == 0:
            result = chunk
        else:
            result = pd.merge((result, chunk), copy=False, how= ...)

You may need to do something with the indivdual chunks before merging them.

Additionally, note that I set the copy argument in merge to False, which might help - the documentation is a little vague as to how it saves memory.


Stacking/Concatenating/Combining two vector space models I have two vector-space models,  with different dimensions.

The number of vectors in one model is the same as the number of vectors in the other. I.E: if I have vector representation for a car in one model,  I have vector representation for a car in the other model, but the number of dimensions can be different.

I want to combine these models (and then cluster using the combined model),  I cannot average (BoW) or add these models together as stated earlier they have different dimensions.

I was going to simply concatenate the vectors. Is this valid or  is there better way to do this?

Also before you concatenate should you normalize?

It is perfectly valid to concatenate the vectors from two different models. It will be necessary however to experiment with the two approaches, i.e. either using individual vectors from one of the models or concatenating the vectors, before you can determine which method will produce better clustering results.  It is also valid to normalize word vectors, but again requires experimentation to determine whether this approach will provide superior results.


Using GridSearchCV for custom kernel SVM in scikit-learn I would like to use scikit-learn's GridSearchCV() to do a grid search on custom parameters in a kernel I have specified. Specifically, the kernel is of the form

SeqKernel(x, y, orig_kernel, cut_off, order)

Here, orig_kernel is a kernel typically used in SVM learning (such as linear, polynomial, RBF, or sigmoid). I wish to perform a grid search over values of cut_off and order, with the additional caveat that only the pairs such that order $\le$ cut_off are considered. What would be the best way to implement this?

Look at the source code and modify it to meet your conditions.

https://github.com/scikit-learn/scikit-learn/blob/ab93d65/sklearn/model_selection/_search.py#L685


Website for Datasets - Miles and Shevlin book, "Applying Regression & Correlation" I've very much enjoyed Miles and Shevlin (2015) book, "Applying regression & correlation : a guide for students and researchers." The book mentions availability of datasets, but the website provided in the book no longer exists (http://jeremymiles.co.uk/regressionbook). Is there another website that I can find the aforementioned datasets?

That page is archived at https://web.archive.org/web/20161024210121/http://www.jeremymiles.co.uk/regressionbook/ , which will take you to https://web.archive.org/web/20160412090429/http://www.jeremymiles.co.uk/regressionbook/data/index.html .
Some of the datasets are available there; I imagine that some others were not crawled. Certainly I successfully obtained data1_1.xls from there. Do feel free to contribute to Internet Archive if you found their service useful.


How does SelectKBest order the best features? I'm trying to run a quick univariate filtering on some data, using a t-test of independence, since my target is binary. However, when I run the filter using sklearn's SelectKBest, I get the same top features returned doing  a manual filter, but in different order. The only information about SelectKBest I could find is here and the documentation, but both seem like they should work like my manual method. My code is

import numpy as np
from sklearn.feature_selection import SelectKBest
from scipy.stats import ttest_ind

np.random.seed(0)

data = np.random.random((100,50))
target = np.random.randint(2, size = 100).reshape((100,1))

X = data
y = target.ravel()

k = 10
p_values = []
for i in range(data.shape[1]):

    t, p = ttest_ind(data[:,i], target)
    p_values.append([i,p[0],t[0]])

p_values = sorted(p_values, key = lambda x: x[1])
p_values = p_values[:k]

# Indices of the ranked p-values
idx = [i[0] for i in p_values]

# SelectKBest features
mdl = SelectKBest(ttest_ind, k = k)

X_new = mdl.fit_transform(X, y)
# Manually selected k best features
X_new2=X[:,idx]

# Print first row of sklearn features
print(X_new[0])
array([0.4236548 , 0.96366276, 0.38344152, 0.87001215, 0.63992102,
   0.52184832, 0.41466194, 0.06022547, 0.67063787, 0.31542835])

# Print first row of manually selected features
print(X_new2[0])
array([0.67063787, 0.4236548 , 0.31542835, 0.87001215, 0.38344152,
   0.63992102, 0.06022547, 0.52184832, 0.41466194, 0.96366276])

It seems SelectKBest is not ordering the features based solely on their p-values or their t-values. How does SelectKBest order the features then?

No, SelectKBest and other *Select* transformers from sklearn.feature_selection do not change order of features, only drop not selected ones.
Anyway, generally, machine learning models do not utilize relative order of a feature.

If you need to check and reorder features, you can use scores_ and/or pvalues_ attributes of a fitted transformer (e.g. SelectKBest) object.

Under the hood those classes use a boolean mask to identify which features to keep. The boolean mask is composed using the first (or the only) set of scores. Those would be t-values in your case. p-values are not used by SelectKBest to select features for now, according to the source code.


What is the benefit of splitting tfrecord file into shards? I'm working on speech recognition with Tensorflow and plan to train LSTM NN with massive waves dataset. Because of the performance gains, I plan to use tfrecords. There are several examples on internet (Inception for ex.) where tfrecords files are split into shards.
My question is: what is the benefit of having tfrecords file into shards? Is there any additional performance gain of this split?

Splitting TFRecord files into shards helps you shuffle large datasets that won't fit into memory.

Imagine you have millions of training examples saved on disk and you want to repeatedly run them through a training process. Furthermore, suppose that for each repetition of the training data (i.e. each epoch) you want to load the data in a completely random order.

One approach is to have one file per training example and generate a list of all filenames. Then at the beginning of each epoch you shuffle the list of filenames and load the individual files. The problem with this approach is that you are loading millions of files from random locations on your disk. This can be slow especially on a hard disk drive. Even a RAID 0 array will not help with speed if you are loading millions of small files from random locations. The problem gets even worse if you are accessing the files over a network connection.

Another approach is to read the training examples in sequence from one large TFRecord file and shuffle the examples in memory using a shuffle buffer. However, the shuffle buffer typically cannot be larger than the DDR memory available to your CPU. And if the shuffle buffer is significantly smaller than your dataset, then it may not adequately shuffle the data. The data may be "locally" shuffled but not "globally" shuffled. That is, examples from the beginning of the dataset may not be shuffled with examples from the end of the dataset.

A good solution is to use a balanced combination of the above two approaches by splitting your dataset into multiple TFRecord files (called shards). During each epoch you can shuffle the shard filenames to obtain global shuffling and use a shuffle buffer to obtain local shuffling. A good balance will make the shards large enough to prevent disk speed issues but will keep the shards small enough to allow for adequately shuffling by a shuffle buffer.

Here are the exact steps:

Randomly place all training examples into multiple TFRecord files (shards).
At the beginning of each epoch, shuffle the list of shard filenames.
Read training examples from the shards and pass the examples through a shuffle buffer. Typically, the shuffle buffer should be larger than the shard size to ensure good shuffling across shards.
Pass the shuffled examples into your training process.


SVM - Making sense of distance derivation I am studying the math behind SVM.
The following question is about a small but important detail during the SVM derivation.
The question
Why the distance between the hyperplane $w*x+b=0$  and data point (in vector form) $p$, $d = \frac{w * p + b}{||w||}$  can be simplified to $d = \frac{1}{||w||}$?
My argument
Since data point $p$ is not on the hyperplane, then we have $w*p+b=k, k \ne 0$. Then $d=\frac{k}{||w||}$ but $k$ is not a constant as it depends on $w$. So optimizing $d=\frac{k}{||w||}$ is not the same as optimizing $d = \frac{1}{||w||}$.
Mentions of the distance calculation step in materials:

page 4 of https://www.ugpti.org/smartse/resources/downloads/support-vector-machines.pdf
slide 10 of http://web.mit.edu/6.034/wwwbob/svm-notes-long-08.pdf
slide 11 of https://www.cs.upc.edu/~mmartin/SVMs.pdf

I believe I have found the answer. A great thank-you to @Javier TG for pointing me to Andrew Ng's notes CS229 Suppor Vector Machines. I would not write down the full context as materials in the lecture notes and questions references do a much better job than I ever will. I will, however, try to fill in the tiny logic gap in many notes: why $w*x+b=1$ can represent any $w*x+b=\lambda, \lambda \ne 0$.
The key understanding is that the distance $d_i$from a point $x_i$ to the plane $w*x+b=0$ is invariant to scaling of $w$ and $b$. Below is a proof.
From the notes, we derived the distance as $$d_i=\frac{y_i(w*x_i+b)}{\|w\|}$$ This shows that $d_i$ is invariant to scaling of $w, b$. Meaning replacing $w, b$ with $2w, 2b$ gives the same $d_i$. You should see it for yourself. Then it is straightforward that the maximum of $d_i$,
$$
d=\max_{w} d_i, i=1,...,n
$$
Is also invariant to the scaling.
However, let $l_i$ represent the nominator in the distance equation, $l_i = y_i(w*x_i+b)$. You will see $l_i$ scales at the same magnitude as that of $w, b$. Meaning replacing $w, b$ with $2w, 2b$ gives $2l_i$. You should also see it for yourself. Then the maximum of $l_i$, $l$ as
$$
l=\max_{w}l_i, i=1,...,n
$$
scales with $w, b$.
Now revisit the optimization problem derived in the notes, $$
\begin{aligned}
\max_{w,b} \quad & d\\
\textrm{s.t.} \quad & y_i(w*x_i+b)\geq l,  i=1,...,n
\end{aligned}
$$
It is okay that we tweak $w, b$ to find the optimum solution, but we also got $d, l$ in the problem statement which makes the problem more complex. It will be great if we can get rid of $d, l$. We do this by leveraging the aforementioned findings. With $l$ scales with $w,b$, we can require that $l'=l*k=1$. To maintain the inequality constraint, we do the same scaling on the left hand side. $$y_i(kw*x_i+kb)\geq kl=1$$. Since $d$ is invariant to scaling, the objective function is not changed. We are still solving the same optimization problem.
Finally, we replace $w=kw, b=kb$, and $d=\frac{l}{\|w\|}=\frac{1}{\|w\|}$, and get the form of the same optimization problem stated in the notes:
$$
\begin{aligned}
\max_{w,b} \quad & \frac{1}{\|w\|}\\
\textrm{s.t.} \quad & y_i(w*x_i+b)\geq 1,  i=1,...,n
\end{aligned}
$$
Now we know, in our optimization problem, any constraint $y_i(w*x_i+b)\geq l$ is equivalent to $y_i(w*x_i+b)\geq 1$.


How can I perform multi-label classification if many labels are missing? I have a large set of documents, usually 500-2,000 words each, and for several different labels, there are about 20-100 samples with those labels, and hundreds to millions more that should be labeled but do not have that label. Over time, I expect new labels to be created and new documents to be added to the data set, although relatively few old labels added to any documents. It is possible that some documents will have many labels or possibly no labels. It is also possible for a label to be a subclass of another label, though not manually indicated.

I have a keyword tagging system in place that utilizes TF-IDF metrics to extract ngrams from the documents, which is my current feature set from them.

My initial thought is to use Naive Bayes classifiers (one for each label) and perform ROC analysis to see how well it classifies the category. However, if, say, 5% of the data should actually be labeled with Label A, and the amount of data supplied with Label A amounts to 1% of the data, then I would expect an ideal false discovery rate of 80%. On top of that, since the prior distribution is unknown, dealing with this issue is even more difficult.

Another idea I had is to use the vectors of word/ngram counts and look at the distribution of the cosine similarities of the documents within the sample and use a simple one-sided significance test with p-values to determine if they are similar enough. This relies heavily on the idea that the ngram vectors for each label are dissimilar enough to consider this form of hypothesis testing. A big problem with this is that I am heavily relying on the assumption that similarity is a highly discriminative metric, and because labels can be extremely unbalanced (by up to 5 orders of magnitude), I may need to employ additional methods, such as limiting the number of labels (perhaps by only taking the top N labels by p-value).

The training portion of the algorithm does not need to be extremely fast, but the labeling portion should be very quick once the document is parsed/tokenized.

There is a greater penalty for mislabeling a document than leaving it unlabeled, but future, human-feedback data may be provided to indicate incorrect labels. I will not have access to anything of the sort in the meantime, though.

This is called Automatic Document/Text Categorization (a lecture note on this). There're some approaches you can take, such as ATC with semi-supervised learning or building one-vs-all classifier and labeling a new test sample with a new label when all classifiers reject it. This was discussed here as well.


Dynamic clustering for text documents I have few hundred thousands of text documents. Some of them are pretty similar - they differ just in ex. names or some numbers, all other text is the same. I would like to cluster these documents, so when I list them, the most similar are listed together in groups. That's how I would avoid having numerous (almost) same documents listed one, after another.

I was thinking some kind of clustering would come handy. But the problem is, I don't know how many clusters I need. Also the number would have to be dynamic. And still, most of the documents wouldn't belong to a cluster, because they don't have any similar documents. So I would cluster just similar documents.

Can anyone point me to the direction that would help me solve this problem, or provide some examples of similar problems.

It sounds as if you don't need clustering.

But rather you are trying to detect near duplicates.

The difference is that clustering tries to organize everything with a focus on the larger, overall structure. But much of your data probably isn't duplicate. Clustering is difficult and slow. Near duplicates is much easier, and much faster (e.g., with MinHash or similarity search)


Does AWS Sagemaker continue running when I go offline? I'm pulling data from web services in an AWS Sagemaker Notebook.

I'd like to be able to close my computer for an hour, reconnect my computer and see that the data collection has continued to run.
Additionally, I'd like my notebook's kernel sessions to remain active so that I can access the data in memory without having to write to file and reload, etc.

Is there a setting to accomplish this, and is it enabled by default?

I couldn't stay at work forever, so I left the Sagemaker notebook running, closed my computer and commuted home.  That evening when I returned to my Sagemaker Notebook, the work had completed running and all the cells and their variables continued to be accessible in memory.

Long story short, if you disconnect from Sagemaker but leave your notebook running, it will continue to process for you.


How to get a (descriptive) overview of a large database? I'm facing a data framework with

~ 20 k observations and
151 variables across
2078 subjects

At first I am primarily interested in how the data looks like related to a single parameter. But I cannot plot 2078 subjects on the x-axis and make a bar plot out of it or so.
What would be useful methods for such a situation? I prefer some visualizations but I think they won't be applicable. I'm afraid even non-visualization methods are not really helpful as well.

There's no way to have a complete summary of a large dataset like this, you have to analyze what can be relevant, decompose into more specific pieces of information and then find the best way to visualize each specific part on its own.
The first thing would be to plot the distribution of this parameter of interest across subjects and/or observations.
If you want to look at the individual level and there are too many values, you can simply pick a random subset (say 100 subjects) and plot these. Then you do it again with a different random subset in order to distinguish real patterns from variations due to chance.


how to interpret a hierarchical clustering in the heatmap in the picture bellow? I am trying to interpret the heatmap which was created based on a agglomerative hierarchical clustering. I am not sure what exactly the heatmap does, having in mind that I see on left hand side clustering done on symptoms and on the top of the heatmap I get a clustering of the bellow labels.
Can someone please help? Here is the picture
Also, this is the code I have used to generate the heatmap:
library(pheatmap)
pheatmap(t(dt_wd[-1]), cluster_rows = TRUE, cluster_cols = TRUE, clustering_distance_cols = "euclidean",
         clustering_distance_rows = "euclidean", clustering_method = "ward",  cutree_cols = 4,
         main = "Cluster of comorbidities based on Covid-19 Symptoms")

This type of plot is refereed as Double Dendogram or Coloured Heatmap.
As seen in the code you have used Single Linkage Method for clustering.It yields clusters in which individuals are added sequentially to a single group.
From the example we can see that label dia2,ht and ob belong to one group but ht and ob are more correlated with each other.

I am not sure what exactly the heatmap does

The color of a cell is proportional to its position along a color gradient.
Heatmap is helpful in finding the correlation coefficient. It can helps us to visualize the difference in value within clusters. It's more like a color representation of value.


empty fields to be filled with previous values In the above sample data, I have empty fields and now the task is to fill the fields with previous values.
my columns are dates and the values are a number of items present for that particular article for the specific date.
which would be a faster way to interpolate the missing fields. Any suggestions to build the function is appreciated.

You can use fillna function with ffill method:

df.fillna(method='ffill', inplace=True)


Machine learning to predict pollution at a single location I have 2 months of air pollution measured data for a location of interest. I am highly interested to play with machine learning algorithms to predict pollution at the same location (let say, for the next two months).
Since air pollution is a geographic problem, does it make sense to apply and predict pollution at a single location using machine learning? Could someone shed a light on prons and cons of this?

Not enough karma to simply comment, so I have to log this as an answer.
Does it make sense? Sure! But keep in mind that if you train your model on a single location, it can only really be trusted to be used on future data from that same location. Could your model generalize to other locations perhaps? Maybe, but your model didn't see anything so you can't be sure. Same thing applies to having only 2 months worth of data. You can't be sure if this model will be accurate when it's scoring data that comes from a month it hasn't seen before.
If you really want to work on this type of problem and just don't have the data for it - this link has a similar dataset that might interest you.


What ML techniques can be used to determine context across sentences? What ML techniques can be used to determine relevance or context between sentences?

For example:

"I am going to the event at 9pm. Can you meet me there?"

Here, there is context between the two sentences - the person is indicating they will be at the event at 9pm, and is asking the recipient to meet them there.

Conversely:

"I am going to the event at 9pm. Can you meet me at school tomorrow?"

Based on this text alone, it appears as if the sentences represent two disjoint thoughts.

Additionally, although the examples above are of adjacent sentences, the context may exist across multiple sentences, or sentences that are not adjacent:

"I am going to the event at 9pm. I've heard it is fabulous. Can you meet me there?"

What you want to do is to extract the relation within the sentence and in this case beyond - so basically a cross-sentence relation extraction.

Ambiguity and the context of words is a big deal when doing this kind of task. There might be multiple ways to get there, but it also depends on your goal (production environment, research or just doing it for fun to get into NLP).

I would recommend you looking into Graph LSTMs (e.g. this Paper)


What makes the validation set a good representative of the test set? I am developing a classification model using an imbalanced dataset. I am trying to use different sampling techniques to improve the model performance.
For my baseline model, I defined an AdaBoost model like so:

    from sklearn.model_selection import KFold
    kf = KFold(n_splits=5, shuffle=False)

    ada = AdaBoostClassifier(n_estimators=100, random_state=42)

    params = {
        'n_estimators': [50, 100, 200],
        'random_state': [42]
    }

    grid_ada = GridSearchCV(ada, param_grid=params, cv=kf, n_jobs=-1,
                           scoring='precision').fit(X_train, y_train)

    # The best precision score is
    grid_ada.best_score_

0.5294693068932419

    # look at all the validation scores
    grid_ada.cv_results_['mean_test_score']

array([0.51916435, 0.52946931, 0.48800155])

    # check the test scores are in line with what we expect from the CV scores
    precision_score(y_test, grid_ada.predict(X_test))

0.4423076923076923
In this case, I am not able to determine if my validation result (~53%) was a good representative of my test result (~44%). And if it isn't, why is that the case
I suppose my question can be split into 3 parts:

When do we determine that a validation set is a good representative of a test set? Should the difference between the two results be between a certain range?
What are some of the reasons for large discrepancies between the validation and test set results? I know from a previous question that data leakage from training set into validation set by upsampling the data before splitting it can cause this. But are there any other obvious reasons?
Does class imbalance influence the reliability of the validation results? So should I be using StratifiedKFold as the scikit learn documentation states:

Some classification problems can exhibit a large imbalance in the distribution of the target classes: for instance there could be several times more negative samples than positive samples. In such cases it is recommended to use stratified sampling as implemented in StratifiedKFold and StratifiedShuffleSplit to ensure that relative class frequencies is approximately preserved in each train and validation fold.

UPDATE:
I have taken two additional steps that have made my validation set more representative of my test set:

I am now using the StratifiedKFold for the cross validation, like so:

    from sklearn.model_selection import StratifiedKFold
    skf = StratifiedKFold(n_splits=5, shuffle=False)

Regarding the initial split of the data between and train and test split, I am now using the stratify option in this method because I expect that future data this model will receive to make predictions will have a similar, imbalanced distribution between classes, so stratification by class percentages make sense:

    X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y,
                            shuffle=True, test_size=0.2, random_state=42)

The general idea of using a validation set is to analyse or control the training of the model. Usually, the model tends to under-fit or over-fit, and this can be observed using the validation split. In practical applications, we just have the visible training set and unknown test set. So, with some assumption on the test set, a simple assumption can be such that test set belongs to the similar distribution as of the training set, we use part of training set as validation set. Once the validation set is chosen, we train the system on the training set and at every required step we observe how the trained model performs on the validation set. (Here, we are still under an assumption that the test-set is not visible). Hence, we can use the validation to check if the model is being trained well and to control the training (early stopping) especially in the case of over-fitting. With this assumption a good validation set should be something which has similar distribution as that of the test set. However, most of the times since we split the available dataset into training, validation and test set, the validation set usually has similar distribution as that of test set.

As mentioned above, difference in the distributions of test set and validation set can cause some discrepancies in the results. As you pointed out, if we upsample the training set before obtaining the validation split, then the validation set will contain different distribution than that of the test set. The discrepancies also occur when the the test set is not provided before-hand, and if it has different distribution than that of training/validation set.

Generally, a random-uniform-split produces the similar distribution, i.e, the ratio of  imbalance reflects in all the splits. However, when the number of samples are small (either overall number of samples or number of samples in minority class), the ratio may not be proportional across the splits. Hence, given an option to have stratified-split, you can prefer the stratified split. (As mentioned, it may not make much difference if random-uniform-distribution produces similar ratios across the split).


K-fold cross validation of scikit-learn with confusion matrix of Keras I intend to display confusion matrix using Keras while K-fold of scikit-learn. My code using Keras is:

import numpy
import pandas
from keras.models import Sequential
from keras.layers import Dense

seed = 7
numpy.random.seed(seed)

# load dataset
dataframe = pandas.read_csv("BolMov.csv", header=None)
dataset = dataframe.values
X = dataset[:,0:24].astype(float)
Y = dataset[:,24]

model = Sequential()
model.add(Dense(16, activation='relu'))
model.add(Dense(7, activation='softmax')
model.compile('adam', 'categorical_crossentropy', metrics=['accuracy'])

y_cat = to_categorical(Y)
result = model.fit(X, y_cat, verbose=0, epochs=50)

plot_loss_accuracy(result)

y_pred = model.predict_classes(X, verbose=0)

print(classification_report(y, y_pred))
plot_confusion_matrix(model, X, y)

How should I use kfold in this code? Here the author is calling a function. What I believe is that if I do so in my code, the model.fit() will be executed twice - once for my Keras code and another time (internally) for the KerasClassifier(). I want that the model.fit() executes once only. Help from anyone is appreciated.

KFold() is meant for cross-validation purpose where multiple models are created over the subsets of the entire dataset and discarded after the validation procedure is over. So the model.fit() should be called explicitly to create the model for purpose. Both the tasks can be easily done through the wrapper named KerasClassifier() by packaging all the details of the model design.


Prove GDA decision boundary is linear My attempt:
(a) I solved that $a=\ln{\frac{P(X|C_0)P(C_0)}{P(X|C_1)P(C_1)}}$
(b) Here is where I'm running into trouble. I'm plugging the distributions into $\ln{\frac{P(X|C_0)P(C_0)}{P(X|C_1)P(C_1)}}$ and I get $a=\ln{\frac{P(C_0)}{P(C_1)}}+\frac{1}{2}(x-\mu_1)^T\Sigma^{-1}(x-\mu_1)-\frac{1}{2}(x-\mu_0)^T\Sigma^{-1}(x-\mu_0)$.
I can see that $b=\ln{\frac{P(C_0)}{P(C_1)}}$ and $w^Tx=\frac{1}{2}(x-\mu_1)^T\Sigma^{-1}(x-\mu_1)-\frac{1}{2}(x-\mu_0)^T\Sigma^{-1}(x-\mu_0)$.
I'm not sure how to simplify $w^Tx$ so that I can solve for $w$. Or is there something that I did wrong?

If you expand the terms, you can see that the quadratic terms cancel out.

\begin{align}
a &= \ln \frac{P(C_0)}{P(C_1)} + \frac12(x - \mu_1)^T\Sigma^{-1}(x - \mu_1) - \frac12(x-\mu_0)^T\Sigma^{-1}(x-\mu_0)\\
&=\ln \frac{P(C_0)}{P(C_1)} + \frac12\left[x^T\Sigma^{-1}x-2x^T\Sigma^{-1}\mu_1+\mu_1^T\Sigma^{-1}\mu_1\right]\\& - \frac12\left[x^T\Sigma^{-1}x-2x^T\Sigma^{-1}\mu_0+\mu_0^T\Sigma^{-1}\mu_0\right]\\
&= (\mu_0-\mu_1)^T\Sigma^{-1}x+\ln \frac{P(C_0)}{P(C_1)} +\frac12\left[\mu_1^T\Sigma^{-1}\mu_1-\mu_0^T\Sigma^{-1}\mu_0\right] \\
\end{align}


Looking for a rough explanation of additive hidden nodes and radial basis functions I'm working on a neural networks project right now and for that I'm reading a bunch of scientific papers, in a few of those the terms additive hidden nodes and radial basis functions are thrown around, but I seem to have trouble to get a clear explanation of the terms there and anywhere else on the internet. I seem to have gathered that these are classifications for neuron types but I would love to get a more clear intuitive explanation of the terms. Preferably one that doesn't require me to be very mathy to understand. I'm fairly new to neural networks so beyond sigmoid neurons and backpropagation algorithms I'm still a bit lost when it comes to the common termonology.

Albeit not wrong, Huang seems to be the only person in the world to use the term "additive hidden nodes". By this, he means that the neuron computes the sum of weighted inputs. In other words, the kind of neural network you're already used to.

An RBF neuron, on the other hand, computes a distance (usually the Euclidean distance) from input to some center (which can be thought as the weights if you see them as a vector) and applies a exp(-dist²) function in order to obtain a Gaussian activation. Thus, RBF neurons have maximum activation when the center/weights are equal to the inputs.


How to set the number of neurons and layers in neural networks I am a beginner to neural networks and have had trouble grasping two concepts:

How does one decide the number of middle layers a given neural network have? 1 vs. 10 or whatever.
How does one decide the number of neurons in each middle layer? Is it recommended having an equal number of neurons in each middle layer or does it vary with the application?

The consideration of the number of neurons for each layer and number of layers in fully connected networks depends on the feature space of the problem. For illustrating what happens in the two dimensional cases in order to depict, I use 2-d space. I have used images from the works of a scientist. For understanding other nets like CNN I recommend you taking a look at here.

Suppose you have just a single neuron, in this case after learning the parameters of the network you will have a linear decision boundary which can separate the space to two individual classes.

Suppose that you are asked to separate the following data. You will need d1 which specifies the upper decision boundary and somehow is doing AND operation to determine whether the input data is on the left side of it or on the right side. Line d2 is doing another AND operation which investigates whether the input data is upper than d2 or not. In this case d1 is trying to understand whether the input is on the left side of line to classify the input as circle, also d2 is trying to figure out whether the input is on the right side of the line to classify the input as circle. Now we need another AND operation to wrap up the results of the two lines which are constructed after training their parameters. If the input is on the left side of d1 and on the right side of d2, it should be classified as circle.

Now suppose that you have the following problem and you are asked to separate the classes. In this case the justification is exactly like the above's.

For the following data:

the decision boundary is not convex and is more complex than the previous boundaries. First you have to have a sub-net which finds the inner circles. Then you have to have another sub-net which finds the inner rectangular decision boundary which decides the inputs which are inside of the rectangle are not circle and if they are outside, they are circle. After these, you have to wrap up the results and say if the input data is inside the bigger rectangle and outside of the inner rectangle, it should be classified as circle. You need another AND operation for this purpose. The network would be like this:

Suppose that you are asked to find the following circled decision boundary.

In this case your network would be like the following network which was referred to but with much more neurons in the first hidden layer.


Correct LSTM model to predict shuffled data For a year I've been collecting data from my RPi:
[0 core load, 1 core load, 2 core load, 3 core load, environment temperature, fan speed, CPU temperature]
Now I want to build a model with Keras and LSTMs to make a prediction of core temperature based on a vector
[0 core load, 1 core load, 2 core load, 3 core load, environment temperature, fan speed].
The main problem is that I can not compose this data into time series, because I've sorted it by 0 core load unintentionally. So this data can be considered as shuffled.
Current model does not work at all:
model = Sequential()

model.add(Dense(128, activation='relu'))
model.add(Dense(128, activation='relu'))
model.add(Dense(128, activation='relu'))
model.add(Dense(128, activation='relu'))
model.add(LSTM(128))
model.add(LSTM(128))

model.add(Dense(1, activation='sigmoid'))

Training summary:
Epoch 1/128
6483/6483 [==============================] - 45s 7ms/step - loss: 140033.1094 - val_loss: 119426.7812
Epoch 2/128
6483/6483 [==============================] - 45s 7ms/step - loss: 140032.5156 - val_loss: 119426.7812
Epoch 3/128
6483/6483 [==============================] - 45s 7ms/step - loss: 140034.2500 - val_loss: 119426.7812
Epoch 4/128
6483/6483 [==============================] - 45s 7ms/step - loss: 140040.8438 - val_loss: 119426.7812
Epoch 5/128
5071/6483 [======================>.......] - ETA: 9s - loss: 140302.2812

How can I predict such shuffled data with LSTM? I'm currently stuck.

The input for an RNN like LSTM should be in the shape of (batch, timestep, features).
Base on my experience, if your current X_train.shape is in 2D you would like to reshape it into 3D array by X_train = np.reshape(X_train, (X_train.shape[0], X_train.shape[1], 1)) Then It will be something like (total number of samples, 7,1); 7 is because you have 7 different features.
By the way, you probably would like to start with a single layer of Dense and LSTM.
Further, using multiple layers of LSTM would require you to set return_sequences to True for all LSTM layers before the last one
model.add(LSTM(128,return_sequences = True))
model.add(LSTM(128))


Predictive Model for Customer Payment Pattern I am working for a logistics firm and there are approx. 750+ customers who avail our services. I am in the process of building and generating some insight for the business based on the payments made by these customers for the last 1 year. Some make payment on time whereas some are late & some are extremely late. Could you please advice which modeling technique or statistical approach would be best in this case.
I can think of creating clusters based on the payment history and highlights & placing all defaulters in one cluster where our company can focus.
PLease suggest.

Here are some things that come to mind:

1) ARMA/ARIMA models: These may show if there is any seasonal trends in their payment history.

2) Logistic Regression: Will allow you model the conditional probability of a missed payments based off of number of previous missed payments.

From here you need to determine what the threshold for risk is since that's a little more subjective.


Text-Classification-Problem: Is Word2Vec/NN the best approach? I am looking to design a system that given a paragraph of text will be able to categorize it and identify the context:

Is trained with user generated text paragraphs (like comments/questions/answers)
Each item in the training set will be tagged with . So for e.g. ("category 1", , "text paragraph")
There will be hundreds of categories

What would be the best approach to build such a system? I have been looking at a few different options and the following is a list of possible solutions. Is Word2Vec/NN the best solution at the moment?

Recursive Neural Tensor Network fed with averaged Word2Vec data
RNTN and The Paragraph Vector (https://cs.stanford.edu/~quocle/paragraph_vector.pdf)?
TF-IDF used in a Deep Belief Network
TF-IDF and Logistic Regression
Bag of words and Naive Bayes classification

1) Max-Entropy(Logistic Regression) on TFIDF vectors is a good starting point for many NLP classification task.

2) Word2vec is definitely something worth trying and comparing to model 1. I would suggest using the Doc2Vec flavor for looking at sentences/paragraphs.

Quoc Le and Tomas Mikolov. Distributed Representations of Sentences and Documents. http://arxiv.org/pdf/1405.4053v2.pdf

Gensim(python) has a nice Doc2vec model.


why nobody uses matlab I wonder why most people now don't use matlab. I guess the reason is matlab is not free, so companies don't want to use it, then interviewees don't use it, then schools don't encourage it, then nobody ends up using it.
But I like it, compared to C++, Python, for it's convenient to plot figures without needing to import any libraries or header files, let alone those powerful ODE solvers.
I might be wrong on any of the above points, as I'm not most familiar with other languages. Any comments?

Most people are using Python or R for data-science, I like Python more because python have enormous amount of libraries for datascience related tasks (for data-analysis, plotting, out of the box machine-learning & deep learning algorithms) and huge open source community to support/help.
You can read more about individual language capabilities everything can be done using any of these languages (Python/R/Matlab) it's just the matter of perspective.


What is difference between "cv2.filter2D" vs Keras "Conv2D" function When I have to sharpen an image using opencv, I use:

#Create our shapening kernel
kernel_sharpening = np.array([[0,-1,0],
                              [-1, 5,-1],
                              [0,-1,0]])# applying the sharpening kernel to the input image & displaying it.

sharpened = cv2.filter2D(image, -1, kernel_sharpening)

In above code sharpened is our resultant image. As you can see in above code I used opencv function named filter2D to perform convolution of input image with the kernel, and as a result I got sharpened image.

Recently I went through this link regarding image Super-Resolution (link)

And found out Keras has something similar to filter2D and Keras calls it  Conv2D.

Its syntax is as follows:

dis2 = Conv2D(filters=64, kernel_size=3, strides=2, padding='same')(dis1)

My question is what is the difference between opencv filter2D, and Keras Conv2D ?

(I assume both do the same role of convolution of image with a kernel, I may be wrong pls correct)

So from an architectural viewpoint you are right, both are 2D Convolutional Kernels with size of (3,3).

But there are some major differences. While cv2.filter2D(image, -1, kernel_sharpening) directly convolute the image dis2 = Conv2D(filters=64, kernel_size=3, strides=2, padding='same')(dis1) only constructs a Conv2D Layer which is part of the Graph ( Neural Network). So Keras Conv2D is no operation for directly convolute an image.

Also the weights are different. In the cv2 Part [0,-1,0],
                              [-1, 5,-1],
                              [0,-1,0] are your weights.

dis2 = Conv2D(filters=64, kernel_size=3, strides=2, padding='same')(dis1)

has standard weight initializer which is glorot uniform. so the weights would not even match.

Additionally the weights of a Conv2D Layer which represents the Keras way, will be learned during training stage of the neural network.

However if you neural network would have only this convolution layer and yields the same weights as the cv2 convolution, the result should be exactly the same.


How do I run SMOTE on image data using the packages available? I need to balance some image datasets, how do I use SMOTE variants or the imblearn SMOTE method with images? I can't figure out how to, since they seem to be working only with numerical datasets.

This might be helpful - https://github.com/tgsmith61591/smrt. Synthetic Minority Reconstruction Technique uses a variational auto encoder.


How do I explain the relationship between Health and Weight? I have two columns in my dataset:  Health and Weight,  both being of numeric type:

    Health<-number of days when health is not good,
    Weight<-weight

All I want to check if there  is a relationship between Health and Weight. In other words, does a Weight increase yield an increase of the number of days  where Health is not good, or the  converse?  I just want to check the relation between those two columns in dataset by plotting some graphs.

Here my sample dataset:

| Health     | Weight      |
|:-----------|------------:|
| 0          |      30     |
| 3          |      63     |
| 2          |      31     |
| 10         |      169    |
| 1          |      9      |
|0           |     139     |

Suggestion

You could bin the 'Health' vector to make a categorical variable (ideally consisting of similar number of observations, for example : 2-bin would be median-split of 'high' and 'low', 3-bin would be tertiles of 'high', 'medium' and 'low', and so on), and then do box plots of 'Weight' per bin. You might see that the bin of 'low' is different from the bin of 'high'.
The number of bins you choose depends on the distribution of the 'Health' variable and you can play around with that.

Implementation

library(dplyr) # for modifying datasets
library(ggplot2) # for plotting
library(magrittr) # for piping
stackodato <- data.frame("Health" = sample(0:10, 10), "Weight" = sample(0:200, 10)) # creating a pseudo dataset

stackodato %>%
mutate(binnedHealth = factor(dplyr::ntile(Health, 2), labels=c("low", "high"))) %>% # add "binnedHealth" column which has the "Health" variable categorized into two factors : "high" and "low"
ggplot()+geom_boxplot(aes(x=binnedHealth, y=Weight)) # boxplot showing the distribution of "Weight" split by the "binnedHealth" factor

You can also try this :

stackodato %>% mutate(binnedHealth = factor(dplyr::ntile(Health, 2), labels=c("low", "high"))) %>% ggplot()+geom_boxplot(aes(x=Health, y=Weight, group = binnedHealth))


After some researches and trial, the dimensions need to be normalised.

In this case, I consider 0 as normal and 1 be outliers.

I normalise the dimensions by take the 10% and 90% of the samples.

clf = svm.SVC(max_iter=1, kernel='rbf')


Why the invariant reward helps training? I am new to Machine Learning, and I am trying to solve MountainCar-v0 using Q-learning. I can solve the problem now, but I am still confused.

According to the MountainCar-v0's Wiki, the reward remains -1 for every step, even if the car has reached the destination. How does the invariant reward help the agent learn? If every step gives the same reward, how can the agent tell if it is a good move or a bad move?

Thanks in advance!

Well, some inefficient agents will need more steps to reach the goal. Others will have a more target-oriented, efficient way to reach the goal. The efficient agents will need less steps and have a better/bigger score.


Finding outliers in multiple dimensions I'm working on a dataset which isn't normally distributed. The dataset contains three dimensions like cost, discount and profit.

I'm trying to find possible outliers in all these dimensions. I used Z-score to detect outliers in single dimension to find which high cost is causing outliers.

As a next step I tried to find outliers with high cost and high profit and low discount.

I came up with a formula of :

Zscore(cost) + Zscore(profit) - Zscore(discount)

negative sign because I want to find outliers with low discount.

Is this approach meaningful to do? or is there any further proven way to achieve this?

Have you considered using the Mahalanobis Distance? It's can be thought of as the magnitude of a multi-dimensional Z-score.

The way I prefer to view the the Mahalanobis Distance is as the square root of the exponent of the Multivariate Normal Distribution. This is similar to the Z-score, which is the exponent of the univarate normal distribution before the square is applied. A big difference between these two is the Z-score is signed, while the Mahalanobis Distance is unsigned, which doesn't matter for finding outliers anyway.

I understand you don't have normally distributed data, but sometimes pretending your data is normally distributed can have good results, so using the Mahalanobis Distance can be worth investigating.


Smaller alternatives to sklearn that doesn't require scipy? I am packaging my model for deployment in aws lambda which has a size limit of 250mb for all dependencies.
Sklearn, if you include its dependencies of numpy and scipy is a huge package.
Are there any alternatives to sklearn that don't require scipy that are smaller than sklearn?
Thanks!

Did you check tinynumpy?
Anyway, I rarely found alternatives to famous packages (except scikit-image instead of opencv). What usually works for me is:

Slim the model as much as I can (e.g. weights quantization)
Check in the code which functions I use from each module. Once I have a list of them, I retrieve the corresponding python files and get rid of the rest
Try to split my process in multiple functions (e.g. one function to perform data processing, one function where to implement the model and make the inference)

The second point is crucial. In my experience, one rarely needs entire packages.
However, depending on the case, it could also be that AWS Lambda does not fit your needs.


Random Forest Classifier gives very high accuracy on test set - overfitting? I have a financial dataset, where I'm trying to predict company types, based on the amount dollars, what time of day, and whether they buy or sell (currency pairs). It looks like this:

The features I use to predict:

X.head():

Dollars   | Hours      | Buy | Sell
-0.761916   0.364838     1     0
-0.924413   0.377558     1     0
-0.573336   0.397836     0     1
-0.561639   0.399144     0     1
-1.164036   0.423715     1     0

The features I want to predict could look like this:

y.head()
Bank  Tech  Fund  Holding  Defence  Financial Services  Pharma
1     0     0     0        0        0                   0
1     0     0     0        0        0                   0
1     0     0     0        0        0                   0
1     0     0     0        0        0                   0
1     0     0     0        0        0                   0

Agriculture  Commodities  Energi  Pension
0            0            0       0
0            0            0       0
0            0            0       0
0            0            0       0
0            0            0       0

In this snippet, the first five companies are banks.

Using a training/test ratio of 0.25, I get an accuracy of 0.99, which seems too good to be true:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25)

rand_forest = RandomForestClassifier(max_depth = None, random_state = 0)

rand_forest.fit(X,y)

predictions = rand_forest.predict(X_test)

The result of the classification_report:

         precision    recall  f1-score   support

      0       0.98      0.95      0.97      5074
      1       0.98      0.91      0.94      2292
      2       0.98      0.82      0.89       572
      3       0.99      0.83      0.90       235
      4       0.98      0.91      0.94       261
      5       0.99      0.81      0.89       411
      6       0.98      0.83      0.90       239
      7       1.00      0.70      0.82       144
      8       1.00      0.81      0.89       384
      9       0.99      0.81      0.89       200
     10       1.00      0.81      0.90       232

avg / total   0.98      0.90      0.94     10044

Adjusting the max_depth parameter of the classifier changes this number significantly though, but I'm still reading up on what the consequences of that parameter actually are.

It is worth mentioning that there is only 50,000 entries in this dataset, across 11 different companytypes, which might be too little?

Using a simpler DecisionTreeClassifier yiels an accuracy of about 50%.

UPDATE:

I used the entire dataset for training, not the actual training set. Switching these two outs gives an accuracy of 54%, which sounds much better (or more realistic anyways).

rand_forest.fit(X,y)

Why are you using the whole data set for training? You are using the test set for training then evaluate the performance on it again?

In your code, I didn't see you actually used the training set you created.


big combination with repetition problem Let me describe my problem:
I've got around 1500 items, all of them described by 15 numeric (rational numbers, positive and negative) attributes.

As an minimalized example:

Item A:
- value I: 34
- value II: 12335
- value III: -10
- value IV: 0
....

Item B:
- value I: 500
- value II: -2332
- value III: 0
- value IV: 9
...

No i have to combine them and find the best combination.
There is space for about 70 items and each item can be used n times.

The performance of each combination can be determinated by a simple math formular using 3 of the attributes, while the other attributes has to match min or max requirements to determinate if a combination is valid or not.

because some items have good values for a single attribute, but negative for other attributes, other items have to be taken to compensate these negative aspects

Now I'm looking for an approach/algorithm/tooling to find the best combination or at least get near to it.

My approaches were to (in pseudocode)

//approach A
while(true)
  var best = generateRandomCombination()
  var challenger = generateRandomCombination()
  best = betterPerformanceIndicator(best, challenger)

at some point i have to stop it and use the result.
tried it, let it run for an hour and loggt every single improvement. horrific performance, to much randomness means repetition. and of course it does not came near to a result i'd produce on my own.

//approach B
var best = generateRandomCombination();
while(true)
  var challenger = replaceSingleItemWithRandomOne(best)
  while (!challenger.isBetter(best) && untestedItemsLeft()){
    challenger = replaceWithOtherItem(best);
  }
  best = betterPerformanceIndicator(best, challenger)

This approach is some kind of approximation, but i think because of the needed compensations it won't be very efficient to change only one item.

So i see two possible ways to improve:
- perhaps i should give the items some kind of weighting
- replace more than on item

Do you have any suggestions to improve my approaches or replace it with better ones?
Is this some kind of data-mining problem? What should i learn to deal with it?

(if my question is wrong in here, i'm sorry. would be nice if you would redirect me to the correct stack exchange section)

added information due to comments:

in the beginning i have to fix the container properties, to determinate if there is space for 70, 71 or an other amout of items. And to to determinate the multiplicator of each attribute. As an example: the containers multiplicator of "value I" would be 1,1 - adding Item A and Item B wour result in (34 + 500) * 1,1 = 587,4.
So you see, adding the items is simple math (addition and multiplication)
Concerning speed, my first approaches were written in java using Lists, but i have not tracked the speed of these single operations.
the order of the item does not effect the outcome, so it's about building an unordered set, just like Neil asked.
The formula to compute the total value is:

square root of ( (sum of all valueI) multiplied by (sum of all valueII and valueIII) )
min and max requiremants are just natural numbers, that the sum of all valueN should (not) reach at least

That sounds like a knapsack problem. Knapsack problem can be solved a variety of ways. Given your constraints, an approximate solution, like first-fit decreasing (FFD), would probably be acceptable.


Topic models for non-textual data? I am looking to employ an unsupervised clustering on a dataset where each observation has a mix of textual and non-textual features.

For each observation, I combine the features into a single vector of ~1000 dimensions. To cluster I have two potential ideas:

Using an autoencoder (or an embedding?) to reduce the dimensionality of the data and then cluster using k-means.
Could I use a topic model? If so, isn't this the superior method in most circumstances to the above?

Why are topic models (in my experience) not commonly used for non-textual data? Is this just a relic of their name/original application, or is there something more fundamental?

Thanks!

I think that you can use a topic model such as Latent Dirichlet Allocation (LDA). For example, in this paper https://pdfs.semanticscholar.org/9e6f/33bdd04df0536f6ad6783d33cccfbc54b1b1.pdf it is used for music and images. I suggest you to take a look at it :) . In general, in topic modeling you end up with a list of topics, where each topic contains a set of associated keywords. In clustering, depending on the algorithm, you might have hierarchy of dependencies. You can also use algorithms which assign each sample to only one class. In addition to this, when doing clustering, you usually have a distance metric which you have to pre-define (e.g. Euclidean distance). The topic models, especially LDA are based on the assumption that your data represents a distribution of topics with their corresponding distribution of keywords (one keyword can be contained in many topics). In other words, you already assume how the texts/documents have been generated.


Assigning manually a score based on some statistics? I am trying to assign manually scores to identify bad accounts (not in Stacks community!) based on some specific conditions, for example if they have account names containing only numbers and a limited number of review, a past history poor, poor score, banned, and so on.
To do it, as I mentioned, I am doing the following:
-if number of reviews is < 2 then assign -1;
or
-if past history is poor then assign -1;
or
-if account score is less than 3 then assign -1;

and so on.
Since I have variables such as number of reviews, past history, account score, for all the users in my dataset, I am just wondering it I should assign a score in a different way, maybe using a more statistical approach based on average.
My goal is to determine an algorithm which can predict if an account is bad or not thought time based on the above conditions.
I would appreciate it if you could let me know what you think.
Thanks

If you use a ML model to learn a score that you built based on your features the best thing that could happen is that your model learn exactly the rules you applied on your features. It would not be usefull as it would be equivalent to applying your rules in the first place. Supervised ML is only usefull to learn unknown rules.
Basically you know have two approaches :

Don't use ML and keep your rules as is. Even if it wouldn't be the most predictive model it might already be usefull to observe unusual behaviors.
If you want to go the ML way you need to build relevant target. Generally speaking that means having a solid, undiscutable definition of your target. In your case the target could be an action from the moderation. Then ML could help you find the behavior that leads to moderator intervention.


Q&A answer comparison multiple sentences using I have been working on a Q&A app that has a template of questions and answers. The hope is to take answer text from the user, and compare it to the correct answer. I’d like to weight it on the keywords/buzzwords and general accuracy of the response.

I’m a mobile developer, not quite a data scientist or well versed in machine learning, so I’m a bit lost in the woods here and not sure if I’m going down the right path.

I’ve been looking into natural language process and some related python libraries/models that evaluate sentences.

Am I headed in the right direction? or should I go about this another way?

It depends on how sophisticated you want the system to be:

the most basic way is to compare the user answer with the gold answer using a simple string similarity measure, such as the overlap coefficient. Basically it just counts the words in common, and there would be a minimum threshold to count the answer as correct (e.g. 80% words in common). It's not very good because a small typo is enough to make the score wrong and it gives the same importance to every word.
The same idea but with TF-IDF weights, typically with cosine similarity. This requires a corpus on which to calculate the IDF weights (which reflect the importance of the words in general).
Still based on string similarity measures but more advanced: a hybrid similarity measure which combines character-level similarity between words (e.g. Jaro, Levenshtein edit distance) and similarity across words. Soft-TFIDF is a common example. Disadvantage: can be tricky to properly adapt to the task.
Beyond that there are a lot of fancy options: using semantic similarity with WordNet (synonyms), words embeddings, etc.

Note: fyi this is not related to the task called Question Answering, which is about a computer-generated answer to a question.


Which algorithm can be used to reduce dimension of multiple time series? In my dataset, a data point is essentially a Time series of 6 feature over a year per month so in all, it results in 6*12=72 features.
I need to find class outliers so I perform dimensionality reduction hoping the difference in data is maintained and then apply k-means clustering and compute distance.

For dimensionality reduction I have tried PCA and simple autoencoder to reduce dimension from 72 to 6 but results are unsatisfactory.

Can anyone please suggest any other way to reduce dimension of this type of data.

So, to be clear, you observe $N$ samples with $P$ measurements at $T$ times (where $P = 6$ and $T = 12$).

I would first determine whether or not you expect the joint distribution of your $P$ features to change over time. For example, if these features are $X$, does $\mathcal{P}(X) = \mathcal{P}(X | T)$ for all values of $T$? If so, you could do something like stack your data over time (so, you'd have $NT$ samples of $P$ features) and perform PCA, then use the top $k$ loadings to rotate the measurements at individual timepoints, leaving you with $N$ $k$-dimensional time series of length $T$. You could choose $k = 1$ if necessary, and use it to visually identify outliers or set a simple cutoff. If not, you'd want to be more careful with this technique - especially if the covariance changes. If it's the variance that's changing but correlation is remaining the same, perhaps some procedures can be done (e.g. perform PCA on the correlation matrix/standardized data) and this technique would still work.

What form do you expect your outliers to take? If it's a temporary deviance from the rest of the sample (e.g. only at one or two measurement times), your technique of PCA on the concatenated data will not be very sensitive. The technique I just described would be more sensitive.


LSTM for time series forecasting with H20.ai I want to implement a time-series prediction model using LSTMs like the one mentioned here: https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/

The model works pretty well using a similar Keras code above, though I want to implement the same using H20.ai

This documentation here on H2O Deep Water says "The H2O Deep Water project supports CNNs and RNNs though third-party integrations of other deep learning libraries such as TensorFlow, Caffe and MXNet.". Though there are no demos for the same.

In this conversation here in Jan '16 on 'Recurrent Neural Networks in H2O for time series prediction?', they say it hasn't been implemented yet.

Any advice or links to similar implementations on H2O would be deeply appreciated

Note: I am looking for a data parallel architecture implementation rather than model parallelism

Active development on H2O's Deep Water project has stopped, see here: https://github.com/h2oai/deepwater

I don't think LSTM was included in the last release, and I would use Keras instead.

A list of Jupyter notebook demos is found here as well: https://github.com/h2oai/h2o-3/tree/master/examples/deeplearning/notebooks


How to interpret a decision tree correctly? I'm trying to work out if I'm correctly interpreting a decision tree found online.

The dependent variable of this decision tree is Credit Rating which
has two classes, Bad or Good. The root of this tree contains all 2464
observations in this dataset.
The most influential attribute to determine how to classify a good or bad credit rating is the Income Level attribute.
The majority of the people (454 out of 553) in our sample that had a less than low income also had a bad credit rating. If I was to launch a premium credit card without a limit I should ignore these people.
If I were to use this decision tree for predictions to classify new observations, are the largest number of class in a leaf used as the prediction? E.g. Observation x has medium income, 7 credit cards and 34 years old. Would the predicted classification for credit rating = "Good"
Another new observation could be Observation Y, which has less than low income so their credit rating = "Bad"

Is this the correct way to interpret a decision tree or have I got this completely wrong?

Let me evaluate each of your observations one by one, so that it would be more clear:

  The dependent variable of this decision tree is Credit Rating which
  has two classes, Bad or Good. The root of this tree contains all 2464
  observations in this dataset.

If Good, Bad is what you mean by credit rating, then Yes. And you are right with the conclusion that all the 2464 observations are contained in the root of the tree.

  The most influential attribute to determine how to classify a good or
  bad credit rating is the Income Level attribute.

Debatable Depends on how you consider something to be influential. Some might argue that the number of cards might be the most influential, and some might agree with your point. So, you are both right and wrong here.

  The majority of the people (454 out of 553) in our sample that had a
  less than low income also had a bad credit rating. If I was to launch
  a premium credit card without a limit I should ignore these people.

Yes, but it would also be better if you consider the probability of getting a bad credit from these people. But, even that would turn out to be NO for this class, which makes your observation correct again.

  If I were to use this decision tree for predictions to classify new
  observations, are the largest number of class in a leaf used as the
  prediction? E.g. Observation x has medium income, 7 credit cards and
  34 years old. Would the predicted classification for credit rating =
  "Good"

Depends on the probability. So, calculate the probability from the leaves and then make a decision depending on that. Or much simpler, use a library like the Sklearn's decision tree classifier to do that for you.

  Another new observation could be Observation Y, which has less than
  low income so their credit rating = "Bad"

Again, same as the explanation above.

  Is this the correct way to interpret a decision tree or have I got
  this completely wrong?

Yes, this is a correct way of interpreting decision trees. You might be tempted to sway when it comes to selection of influential variables, but that is dependant on a lot of factors, including the problem statement, construction of the tree, analyst's judgement, etc.


Datasets for Topic Modeling I'm looking to try and use deep learning methods for topic modeling as opposed to the more traditional methods of lda and word embedding methods. However, I'm having trouble finding good labeled datasets for this task.  So far the best that I've seen is the New York Times Dataset which I can't use due to licensing constraints. I've also seen the 20News Dataset but it only has twenty categories so it probably won't scale well to other domains.

Are there any other good datasets out there that I'm missing that can be used for topic modeling?  I'm happy to use a dataset that isn't explicitly meant for topic modeling; as long as it has some sentences/paragraphs that are tagged or labeled that should be fine.

Since topic modeling is unsupervised, it's not usually evaluated against labeled data. Instead people devise measures which evaluate the clusters, typically based on how much the most probable words for a topic are semantically relevant.

You might find data ideas in this paper: https://www.cs.cornell.edu/~laurejt/papers/authorless-tms-2018.pdf


Random forest minimum number of observations I have around 5000-6000 observations of nearly 8-10 variables (of which 2 are discrete, categorical) and a single numerical target parameter. As per initial evaluation, random forest regression might be a good algorithm for the current case.

Is the current observations/variables count adequate for the proposed method? If other regression algorithms are recommended as things are described currently, kindly let me know.

The important is not the number of observation but the quality of this observations. If you have a look at toy datasets of sklearn they are way smaller than that.

Random forest is a good algorithm when there is small data since it is a bagging of decision trees with bootstrap. Each decision tree is feed with a sample of data with replacement, in this way even if the data is small there are bigger chances of making a good model.

In a high level, yes it seems a good way to go, but with out knowing more at the data is hard to tell.

I would suggest to give it a try with a Generalized Linear Model, a support vector machine and a gradient boosting.  Since your data is small you will not need much computation time for it.


Determining extreme outliers from boxplot (by eye) I have a normalised dataset with range 0,1. I have created boxplots for every feature in  the dataset and need to identify which features have extreme outliers (by looking at their boxplots).
However, because of the look of my boxplots I am struggling to identify which would be extreme outliers or mild (or even have any at all). You can't see clearly the whiskers etc of the boxplots (due to the range of 0-1) but can see some outliers.
I'm hoping someone could help me identify if these boxplots (1, 2 and 3) in the attached image have extreme outliers or not?
(im particularly stuck with the 3rd one - because you can still see a circle)
Thanks in advance!

These images aren't very useful to make decisions by eye, because the scale is far too compressed. In the second and third images the mean, entire box and both whiskers are all represented by a single vertical line, making it impossible to tell if those values are distributed over a range of 0.01, or 0.00001 or 0.000000001. In the third image, the circle looks overlaid directly onto the vertical line, so any attempt to guess how far outside the interquartile range that point is would be just that, a guess. That point in the circle could be just barely outside the whisker, or a million times greater than the mean value, we simply cannot tell from that plot. All we know from the circles is that those point are outside of the whiskers (typically the mean + 1.5 * IQR), but with no meaningful way to discern mean or IQR from the plots, that's not very valuable information.
When making a visual assessment, it's important to put your data on a scale that can actually be visualized. You wouldn't try to visually estimate the distance from your house to the pub on a map of the universe - you'd need a map that can discern distances on the order of miles, not millions of miles.  There's not really any reason to plot the range from 0 to 1 when all your points fall in the range of 0 to 0.01, and it will compress potentially meaningful differences to the point where there is no difference whatsoever on a visual scale. You could also try log-transforming your data, which will compress the high end of the range and stretch the low end of the range, which could be useful for looking at data with a right tail.
Also consider that these might not be outliers at all, depending on the distribution of your data - a boxplot may not tell the whole story. For an exponentially distributed value, for example, it'll be completely expected that you'll have some values skewed to the right which are far from the mean. Those aren't weird outliers, they're just an expected part of the exponential distribution.


Why not using linear regression for finetuning the last layer of a neural network? In transfer learning, often only the last layer of the network is retrained using gradient descent.
However, the last layer of a common neural network performs only a linear transformation, so why do we use gradient descent and not linear (or logistic) regression to finetune the last layer?

The common approach to fine-tuning an existing pre-trained neural network is the following:

Given an existing pre-trained neural network model (e.g. imagenet), remove the last layer (which does classification in the pre-training task) and freeze all weights in the remaining layers of the model (usually with setting the trainable parameter to false).
Add a new final dense layer that is to be trained on the new task.
Train the model on the new task's dataset until convergence.
[optional] After fine-tuning is converged, possibly unfreeze the all the layers and train further until convergence with a lower learning rate.

A reason to use gradient descent over a different ML algorithm as you suggest is to enable further training after initial fine-tuning (step #4 above). However, it's not necessary to do this. The approach you suggest (to use take the output of the pre-trained model as input to another ML model) may provide satisfactory performance and be more computationally efficient.
Tradeoffs between these approaches is also discussed in the Keras Transfer Learning guide in the section on the "Typical Transfer Learning Workflow".


Semi-gradient TD(0) Choosing an Action I am trying to write an optimal control agent for a simple game that looks like this:

The agent can only move along the x-axis, and has three actions available to it: left, right, and do nothing. A random number of falling rocks are spawned at arbitrary positions along the top row. The goal is to survive as long as possible by avoiding collision on each time step.

Here's what I've done so far:

1) I use a feature vector $Φ$ with $Φ_0(s)$ being the current x-coordinate of the agent and $Φ_1(s)...Φ_n(s)$ taking on a 0 or 1 (1 indicating the presence of a rock).

2) The corresponding weight vector $θ$ is initialized to 0 for all weights. So I have the linear function approximation $$\hat v(s,θ)=\sum_{i=0}^nΦ_i(s)θ_i$$

3) The reward on each time step is 1, and 0 upon collision.

I'm actually trying to implement the algorithm below from Sutton and Barto's 2017 draft.

Semi-gradient TD(0) for estimating $\hat{v} \approx v_{\pi}$

Input: the policy ${\pi}$ to be evaluated
Input: a differentiable function $\hat{v} : \mathbf{S^+} \times \mathbb{R}^n \rightarrow \mathbb{R}$ such that $\hat{v}(terminal,·) = 0$
Initialize value-function weights $\theta$ arbitrarily (e.g., $\theta = 0$)
Repeat (for each episode):
$\qquad$Initialize $S$
$\qquad$Repeat (for each step of episode):
$\qquad\qquad$Choose $A \sim \pi(·|S)\qquad$ # Not sure how to choose action here
$\qquad\qquad$Take action $A$, observe $R, S'$
$\qquad\qquad\theta \leftarrow \theta + \alpha[R + \gamma \hat{v}(S',\theta) − \hat{v}(S,\theta)]\nabla\hat{v}(S,\theta)$
$\qquad\qquad S \leftarrow S'$
$\qquad$until $S'$ is terminal

My problem is this:

1) Is this even an appropriate algorithm to apply to this sort of task? It feels like a policy gradient approach would be more suitable.

2) If yes to 1), how do I choose an action? In the algorithm above, this appears as "Choose $A \sim \pi(·|S)$" Since the policy is implicit,  I compute the approximate value of the state as shown in 2) above and then greedify over the three actions (no ε-greedy, though), but the only thing that changes is $Φ_0(s)$. I'm definitely missing something.

1) Is this even an appropriate algorithm to apply to this sort of task?

No, you have selected an evaluation algorithm from chapter 9 of the book. None of the algorithms in chapter 9 are control algorithms. They are all designed to estimate the value function for a policy supplied as input. The corresponding control algorithms are discussed in chapter 10.

The current draft of the book does not give the corresponding TD(0) control algorithm with linear estimator. However, that algorithm does exist and might be suitable (with caveats). In fact in your case it could even have benefits over action-value based methods, because you reduce the scope of estimates needed by a factor of 3. This is something that you can take advantage of only if you have a full model of the environment, so can look ahead one time step to determine the best action. Without a model of the environment, or if you don't want to use the model in your agent, then you must use an action value based algorithm like Monte Carlo, SARSA or Q Learning.

  2) If yes to 1), how do I choose an action?

Well it was a no, but you could use the control version of TD(0). Then you have the problem of using your state value function to figure out the policy. The rule here is that to use state values you need to use a model of the environment. Specifically you need to be able to predict the next state and immediate reward given the current state and action. In the book, this is usually represented by the transition function $p(s′,r|s,a)$ which gives the probability of each possible successor state and reward. In a fully deterministic game, the probability is just 1.0 for one target state and reward caused by each action. In your case you have new rocks appearing randomly at the top. To be complete you would probably have to model this in detail (which would be painfully slow). However, given the very low influence of this top row, and how little planning the agent can do to deal with it, I'd be tempted to just sample it.

Assuming you want to choose the greedy action, then you can find a policy by taking $argmax_a$ over the next step. When you have a state value function, then you have to run the step forward in simulation to figure out the expectation over each action. This is the same calculation for greedy policy as used in dynamic programming (back in chapter 4 of the book):

$\pi(s) = argmax_a \sum_{s'} p(s',r|s,a) (r + \gamma \hat{v}(s',\theta))$

Of course this is a lot simpler if you used action values istead (e.g. in SARSA):

$\pi(s) = argmax_a \hat{q}(s,a,\theta)$

. . . so despite the fact this is less efficient, you might want to use action values for less effort in this part.

One additional thing you are likely to have problems with: Your choice of state representation does not have good linear relationship with the true value function. The linear estimator is quite limited in what it can do. A better estimator here would be a neural network with at least one hidden layer.

However, you can probably tweak the representation slightly to get something that will work a little bit with a linear estimator - the trick is to have the rocks part of the state represented relative to the agent - i.e. don't have a grid of absolute positions of rocks, but make the grid relative to the agent. That way, a rock directly above the agent will always contribute the same to the current state value, which is important. Without this tweak, and using a linear approximator, your agent will not learn optimal control, it will instead learn a kind of fatalistic "with these many rocks, I have roughly this long to live" value function and probably just take random actions (if the distribution of rocks is not even it might learn to move to a particular column . . .)


Are mini batches sampled randomly in Keras' Sequential.fit method() When you .fit a Keras Sequential() model, you can specify a batch_size parameter. I have noteiced it is sometimes defined independently from the actual dataset size. Does it mean that mini-batches are sampled randomly, instead of scrolling down the dataset from top to bottom, slice by slice?

If you set shuffle=True as an argument of the model.fit method, Keras will shuffle the dataset before splitting it into batches (source), otherwise the dataset will be processed sequentially.


How to improve machine learning model using 2+ datasets I am building a supervised machine learning model which (for example) predicts heart failure (yes/no). I have two datasets from 2 different labs A and B, which both have decent distribution, aka it's not like A has way more young people than B, but somehow A has much lower rate of heart failure.
Having created a separate model for each, they each achieve around 90% or higher accuracy, but my aim is to make one model which can use a concatenated dataset containing information from A and B.
Currently this combined model gets 75% accuracy for samples from B and 90% for A.
What I have tried:

sample in different ways / create 'even' datasets
add feature indicating which lab
normalise numerical features
create categories for numerical features

I'm aware the overall results would of course not become as good as the individual models, but how can I make the platforms at least get more similar results?
I appreciate any advice / methods I could try out to tackle this type of problem!!

You may wish to combine the two models using an approach called Ensemble Methods.
However, in your case I'm surprised that using the combined data in a single model is so much worse if as you say the distribution of the features is similar.
So two potential outcomes:

if the distributions of the data are the same, the it could be the individual models are overfitted and given the larger data set this is exposed.

if the distributions are different then each model is being tuned for specific features and hence a single general model is hard to achieve.

To get a better feel of what to expect, I would suggest doing some distribution plots for each feature to see if they are as you say similar.
Another simple test is to take Model A and use it to predict everything in Dataset B and vice-versa. If the actual accuracy is around 70% then everything is working as expected.


Appending to numpy array for creating dataset I want to create a dataset from three numpy matrices - train1 = (204,), train2 = (204,) and train3 = (204,). Basically all sets are of same length. I am applying a sliding window function on each of window 4. Each set become of shape =(201,4) I want a new array in which all these values are appended row wise. Like for first train1 then train2 then train3. And final output set is of size =(603,4).

This is a sliding window function which converts array of shape (204,) to (201,4)

def moving_window(x, length, step=1):
    streams = it.tee(x, length)
    return zip(*[it.islice(stream, i, None, step) for stream, i in zip(streams, it.count(step=step))])

Create dataset fucntion is:

def create_dataset(dataset1,dataset2):
    dataX=[]
    x=list(moving_window(dataset1,4))
    x=np.asarray(x)
    dataX.append(x)
    y=list(moving_window(dataset2,4))
    y=np.asarray(y)
    dataX.append(y)
    return np.array(dataX)

data_new=create_dataset(train1,train2)

It is returning a dataset of shape 0(2,201,4). I think this is appending differently, but I want row wise appending. so that the new _dataset is of shape= (402,4) with two sets and (603,4) with three sets. I want to generalize as well like if I want for 10 training sets or twenty training sets. How can I do that?

I think its because of the way you are appending the datasets in list and then converting it to numpy array.

Solution 1

One quick solution is to reshape your array as -

data_new = data_new.reshape(data_new.shape[0]*data_new.shape[1], data_new.shape[2])

So, your data of shape (2,201,4) will become (2*201,4) = (402,4).

Solution 2

Another solution is to append the arrays in the function you have defined, instead of returning np.array(dataX), use -

return np.append(x, y, axis = 0)

So, you don't have to use dataX anywhere.


What is a Recurrent Heavy Subgraph? I recently came across this term recurrent heavy subgraph in a talk. I don't seem to understand what it means and Google doesn't seem to show any good results. Can someone explain what this means in detail.

The term may best be expressed as a Recurrent, Heavy Subgraph. That is, a subgraph which is both Recurrent and Heavy.

Heaviness of a subgraph refers to heavily connected vertices- that is, nodes which are connected many times ("many" being relative to the network in question).

Recurrent refers to the propensity of a subgraph to occur more than once.

Thus, a Recurrent Heavy Subgraph  is a densely connected set of vertices which occurs several times in the overall network.

These subgraphs are often used to determine properties of a network.

For example: In a network of emails interactions within a company organized into 4-person teams with one member acting as the lead, each team's email activity (if they email between themselves sufficiently to be considered "heavy") could be described as a Heavy Subgraph. The fact that these subgraphs occur many times in the network make them Recurrent Heavy Subgraphs. If one was searching for structure in the network, noticing that these recurrent, heavy subgraphs exist would go a long way toward determining the organization of the network as a whole.


Long term time series forecasts with small dataset I have a small dataset which has timestamp and temperature values for 6 months(I.e. one temperature value per day). I would like to forecast 2-3 months of temperature.
I would like to know, what kind of models would be suitable for such purpose.

I have used LSTM but the forecasts degrades as I go further  and the data size is pretty small for LSTM to be trained.

You can use reinforcement learning, in instance mbrrl2 package.

As temperature is very stational dependent variable. You should have at least 1 year, better 2 years, to make some predictions.

Other way to solve this, is having a collection of historical temperatures, clustering them, and matching your dataset with the closest cluster.


Derivative of activation function used in gradient descent algorithms Why is it necessary to calculate the derivative of activation functions while updating model( regression or NN) parameters? Why is the constant gradient of linear functions considered as a disadvantage?

As far as I know, when we do stochastic gradient descent using the formula:

$$\text{weight} = \text{weight} + (\text{learning rate}\times (\text{actual output} - \text{predicted output}) * \text{input})$$

then also, the weights get updated fine, so why is calculation of derivative considered so important?

As the name suggests, Gradient Descent ( GD ) optimization works on the principle of gradients which basically is a vector of all partial derivatives of a particular function. According to Wikipedia,

  In vector calculus, the gradient is a multi-variable generalization of the derivative.

At its core, GD computes derivatives ( in terms of Neural Networks ) of a composite function ( a neural network is itself a composite function ) because of the gradient descent update rule, which is,

$\Large \theta = \theta - \alpha \frac{\partial J}{\partial \theta}$

Where $\theta$ is the parameter which needs to be optimized. In a neural network, this parameter could be a weight or a bias. $J$ is the objective function ( loss function in NN ) which needs to be minimized. So for $\frac{\partial J}{\partial \theta}$, we need to repeatedly apply the chain rule till we have a derivative of the loss function with respect to that parameter.

Intuition:

Sorry for the weird image. When GD is far away from the function minima ( where it tends to reach ) the value of $\frac{\partial J}{\partial \theta}$ is greater and therefore the updated value of $\theta$ is smaller than the previous one. This updated value is scaled by the learning rate ( $\alpha$ ). The negative sign indicates that we are moving in the opposite direction of the gradient.


How to figure out if two sentences have the same meaning with AI? I have two sentences which might be similar in meaning.
Are there a useful and successful (machine learning) algorithms, which is able to determine the semantic similarity?

Are there any approaches which can handle this task and are easy to use?

An approach via Keras or Tensorflow as a Framework would work perfectly in my use case.

Thank you!

This corresponds to an NLP task called paraphrase detection. It's an active area of research, as far as I know there's no ready-to-use system able to perform this task very well, but there are probably a good few methods and prototypes around. A quick search gives these links for example:

https://aclweb.org/aclwiki/Paraphrase_Identification_(State_of_the_art)
https://arxiv.org/abs/1712.02820


Churn Prediction Training Set I don't understand how to form my dataset from activity(logins etc.) and characteristic(location, age etc.) raw user data.

Ultimately, each row of the training set will have N activity features for a certain period, M characteristic features and a binary outcome - churn or not after the end of this period.

My problem comes from defining the period and the number of rows per users.

The options I see are the following:

Define period from start of user lifetime, 1 week for example. Then each row is 1 user (activity and characteristics) and outcome is whether they churned in week 2 or not.
Break down a user's lifetime into periods. Predict all users every day on the data from their last week. Let's say user has 2 weeks lifetime. Training data will be:

data_week_1, not churn

data_week_2, churn

Looking for any advice or links related to the viability of these or other methods of dataset formation.

A simple way to go would be to use Option 1. Thus, each row could be uniquely identified and you could perform classification easily. You could additionally add a column which mentioned the week in which they churned. This would be very similar to a Type 2 Slowly Changing Dimension. Go wide with your dataset, ie: keep adding as many columns as necessary. That way, even if you have to look at it in Excel before creating a classification model, it'll be easier. If you are using a Random Forest classification algorithm or any other tree based algorthim, this option would make good splits on nodes.

Option 2 would be possible, however, it would result in a very deep dataset which would grow over time if you decide to update it frequently.


How can I access to loss value in Keras LSTM implementation? I use Keras library and it's LSTM model. When I train my network I can see loss value in my program execution console. I like to know how can I access to this value in my code?

You can access it by assigning a variable when calling fit

hist = model.fit(X, y)

Where hist is a dictionary containing history of various variables during training. To get your training loss you would do hist['loss']