How to calculate the fold number (k-fold) in cross validation? I am confused about how I choose the number of folds (in k-fold CV) when I apply cross validation to check the model. Is it dependent on data size or other parameters?

The number of folds is usually determined by the number of instances contained in your dataset. For example, if you have 10 instances in your data, 10-fold cross-validation wouldn't make sense. $k$-fold cross validation is used for two main purposes, to tune hyper parameters and to better evaluate the performance of a model.

In both of these cases selecting $k$ depends on the same thing. You must ensure that the training set and testing set are drawn from the same distribution. And that both sets contain sufficient variation such that the underlining distribution is represented. In a 10-fold cross validation with only 10 instances, there would only be 1 instance in the testing set. This instance does not properly represent the variation of the underlying distribution.

That being said, selecting $k$ is not an exact science because it's hard to estimate how well your fold represents your overall dataset. I usually use 5-fold cross validation. This means that 20% of the data is used for testing, this is usually pretty accurate. However, if your dataset size increases dramatically, like if you have over 100,000 instances, it can be seen that a 10-fold cross validation would lead in folds of 10,000 instances. This should be sufficient to reliably test your model.

In short, yes the number of folds depends on the data size. I usually stick with 4- or 5-fold. Make sure to shuffle your data, such that your folds do not contain inherent bias.


Pyspark Matrix Transformation Let's assume I have the following dataframe in PySpark:

Customer    |  product  |   rating
customer1   |  product1 |   0.2343
customer1   |  product2 |   0.4440
customer2   |  product3 |   0.3123
customer3   |  product1 |   0.7430

There can be several customer product combinations but every combination is unique already. I want to archive the following outcome in the most efficient manner:

Customer (Index) | product 1 | product 2 | product 3
customer 1       |   0.2343  |  0.4440   |  0.0000
customer 2       |   0.0000  |  0.0000   |  0.3123
customer 3       |   0.7430  |  0.0000   |  0.0000

Each combination which is not represented in the first table will be set to zero. It has to be efficient because the output matrix will have a size of 59578 rows × 21521 columns and I want to avoid the computational cost as good as possible.

Is there any solutions for this? I didn't found a good solution on the web so far.

Thanks for your help up front.

The way to do this in PySpark is to use groupBy and pivot. Since you don't want to do any actual aggregation, just the pivot, you can use first here.

from pyspark.sql.functions import first

(df.groupBy("Customer")
  .pivot("product")
  .agg(first("rating"))
  .fillna(0))

pivot will give nulls when there is no value so fillna needs to be used as well to give the wanted result.


How to fill missing value based on other columns in Pandas dataframe? Suppose I have a 5*3 data frame in which third column contains missing value

1 2 3
4 5 NaN
7 8 9
3 2 NaN
5 6 NaN

I hope to generate value for missing value based rule that first product second column

1 2 3
4 5 20 <--4*5
7 8 9
3 2 6 <-- 3*2
5 6 30 <-- 5*6

How can I do it use data frame? Thanks.

How to add condition to calculate missing value like this?

if 1st % 2 == 0 then 3rd = 1st * 2nd
else 3rd = 1st + 2nd

1 2 3
4 5 20 <-- 4*5 because 4%2==0
7 8 9
3 2 5 <-- 3+2 because 3%2==1
5 6 11 <-- 5+6 because 5%2==1

Assuming three columns of your dataframe is a, b and c. This is what you want:

df['c'] = df.apply(
    lambda row: row['a']*row['b'] if np.isnan(row['c']) else row['c'],
    axis=1
)

Full code:

df = pd.DataFrame(
    np.array([[1, 2, 3], [4, 5, np.nan], [7, 8, 9], [3, 2, np.nan], [5, 6, np.nan]]),
    columns=['a', 'b', 'c']
)
df['c'] = df.apply(
    lambda row: row['a']*row['b'] if np.isnan(row['c']) else row['c'],
    axis=1
)


Are there any objections to using the same (unlabelled) data for pre-training of a BERT-Based model and the downstream task? I'm looking to train an Electra model using unlabelled data in a specific field. Are there any objections to using the same data for unsupervised learning and then using the same data downstream for the supervised learning task?

Not at all. A recent ACL paper by AllenAI even says this is the best way. They recommend continuing pre-training on the task data and claim that it reduces the problems caused by domain mismatch. So, if you train the model on the in-domain data from the very beginning, it is probably a good thing given you have enough data for that.


Fully Convolutional Variational Autoencoder I would like to make a neural network which uses black and white images as input and outputs a colored version of it. The important thing in that process is that the size of the images must stay the same.

Usually this is done by using a Fully Convolutional Network with GAN or AE architecture. Now I have decided to implement a VAE version, but when i looked it up on the internet I found versions where the latent space was Linear/Dense meaning it breaks the Full Convolution.

Is a VAE not a valid Approach for this type of neural network? Or is there a solution/some code someone can provide which would help in that situation? (Pytorch prefered)

In my experience, when people say "fully" convolutional, even when doing something typical like ImageNet classification, they are still typically referring to a network with at least one final dense layer.

If I understand your question correctly, you're looking to create a VAE with some convolutional layers which has the same sized output as input, but you're confused how to upsample in the decoder such that you go from fewer latent dimensions to an output of the same size as your input. There are a few ways people typically handle this, two of which are deconvolution and sub-pixel (description of the difference, good deconv tutorial). Both allow you to use something like a convolution to take the output from $ d << n $ latent dimensions and upsample it into an $ n $ dimensional output.

Here's an example written in Keras that does this using deconvolutional layers.


Multivariable real time system for fraud detection This is the scenario:

Client -> Server

The client sends multiple voice calls to Server.
Call info:

calling number
called number
call duration
source IP
destination IP
SIP header information (User-Agent, Version, Proxy information)

Other patterns:

Numeric Patterns - Destination.
Time Patterns - Calls occurring too regularly, activity at unusual times or dates.
Geographic Patterns - Proximity relationships between apparently unrelated entities.

In case of Toll fraud and client being insecure, the attacker can send calls via original agent hence the Service Provider won't reject calls immediately at least based from IP information. (Toll fraud).
I'm exploring which approach is the best to implement an ML model, instead of static rules to be able to detect Toll fraud in a live system:

Classification (Fraud or not)
Reinforcement Learning (If fraud then reward)

I'm new to RL, does this scenario makes sense for live systems?
Can I use offline data to train an RL model? Offline data includes Fraud cases detected historically.

Reference.

I think it makes sense to stick with Classification here, since you already have examples of fraudulent and non-fraudulent calls that you can train on.

It might also be beneficial to train several models for different regions based on IP, as well as your 'global' model, and apply the region specific and global models to incoming calls. Just a few ideas.

From what I understand, real-time learning would require immediate feedback, which most fraud detection systems can't provide. For example, it may take a few days or weeks to have a case of fraud resolved (labelled fraud/not-fraud), and therefore take some time for the learning system to receive the feedback on it's prediction.


Understanding the mathematics of AdaGrad and AdaDelta I have been building some models for a project, but I can't wrap my head around the math of Adagrad and Adadelta algorithms.

I do understand how vanilla gradient descent works and I have written code for making it work successfully.

I will be grateful if anyone explain these two things to me or provide some resource to understand them.

With regard to resources:

In my opinion, ADADELTA: An Adaptive Learning Rate Method (the original ADADELTA paper) explains (in sections 1-3) both ADAGRAD and ADADELTA in a quite accessible manner.
I found Adaptive Subgradient Methods for Online Learning and Stochastic Optimization to be less accessible, but it is the original ADAGRAD paper, so it is probably worth a shot.
An overview of gradient descent optimization algorithms (a blog post by Sebastian Ruder) also helped me understand both ADAGRAD and ADADELTA.

Here are some central quotes from ADADELTA: An Adaptive Learning Rate Method, along with some examples and short explanations:

ADAGRAD

  The update rule for ADAGRAD is
  as follows: $$\begin{matrix}\Delta x_{t}=-\frac{\eta}{\sqrt{\sum_{\tau=1}^{t}g_{\tau}^{2}}}g_{t} &  &  & (5)\end{matrix}$$
  Here the denominator computes the $l2$ norm of all previous
  gradients on a per-dimension basis and η is a global learning
  rate shared by all dimensions.While there is the hand tuned global learning rate, each dimension has its own dynamic rate.

I.e. if the gradients in the first three steps are $g_{1}=\left(\begin{gathered}a_{1}\\
b_{1}\\
c_{1}
\end{gathered}
\right)\,,\,g_{2}=\left(\begin{gathered}a_{2}\\
b_{2}\\
c_{2}
\end{gathered}
\right)\,,\,g_{3}=\left(\begin{gathered}a_{3}\\
b_{3}\\
c_{3}
\end{gathered}
\right)$, then: $$\begin{gathered}\Delta x_{3}=-\frac{\eta}{\sqrt{\sum_{\tau=1}^{3}g_{\tau}^{2}}}g_{3}=-\frac{\eta}{\sqrt{\left(\begin{gathered}a_{1}^{2}+a_{2}^{2}+a_{3}^{2}\\
b_{1}^{2}+b_{2}^{2}+b_{3}^{2}\\
c_{1}^{2}+c_{2}^{2}+c_{3}^{2}
\end{gathered}
\right)}}\left(\begin{gathered}a_{3}\\
b_{3}\\
c_{3}
\end{gathered}
\right)\\
\downarrow\\
\Delta x_{3}=-\left(\begin{gathered}\frac{\eta}{\sqrt{a_{1}^{2}+a_{2}^{2}+a_{3}^{2}}}a_{3}\\
\frac{\eta}{\sqrt{b_{1}^{2}+b_{2}^{2}+b_{3}^{2}}}b_{3}\\
\frac{\eta}{\sqrt{c_{1}^{2}+c_{2}^{2}+c_{3}^{2}}}c_{3}
\end{gathered}
\right)
\end{gathered}
$$
Here it is easier to see that each dimension has its own dynamic learning rate, as promised.

Problems of ADAGRAD that ADADELTA tries to counter

  The idea presented in this paper was derived from ADAGRAD in order to improve upon the two main drawbacks
  of the method: 1) the continual decay of learning rates
  throughout training, and 2) the need for a manually selected
  global learning rate.

The second drawback is quite self-explanatory.

Here is an example for when the first drawback is an issue:
Consider a case in which the absolute value of each component of $g_2$ is much larger than the absolute value of the respective component of the gradient in any other step.
For any $t>2$, it holds that every component of $\sqrt{\sum_{\tau=1}^{t}g_{\tau}^{2}}$ is bigger than the absolute value of the respective component of $g_2$. But the absolute value of every component of $g_2$ is much bigger than the absolute value of the respective component of $g_t$, and so $\Delta x_t$ is very small.
Moreover, as the algorithm progresses, it gets closer to a minimum, so the gradient gets smaller, and so $\Delta x_t$ becomes smaller and smaller.
Thus, it might be that the algorithm virtually comes to a standstill before reaching a minimum.

ADADELTA

Instead of considering all of the gradients that were calculated, ADADELTA considers only the last $w$ gradients.

  Since storing $w$ previous squared gradients is inefficient,
  our methods implements this accumulation as an exponentially
  decaying average of the squared gradients. Assume at
  time $t$ this running average is $E\left[g^{2}\right]_{t}$ then we compute:
  $$\begin{matrix}E\left[g^{2}\right]_{t}=\rho E\left[g^{2}\right]_{t-1}+\left(1-\rho\right)g_{t}^{2} &  &  & (8)\end{matrix}$$
  where $\rho$ is a decay constant [...].  Since we require the square root of this quantity
  in the parameter updates, this effectively becomes the $\text{RMS}$
  of previous squared gradients up to time $t$:
  $$\begin{matrix}\text{RMS}\left[g\right]_{t}=\sqrt{E\left[g^{2}\right]_{t}+\epsilon} &  &  & (9)\end{matrix}$$
  where a constant $\epsilon$ is added to better condition the denominator

($\text{RMS}$ stands for Root Mean Square.)

Similarly:
$$E\left[\Delta x^{2}\right]_{t-1}=\rho E\left[\Delta x^{2}\right]_{t-2}+\left(1-\rho\right)\Delta x_{t-1}^{2}$$
$$\text{RMS}\left[\Delta x\right]_{t-1}=\sqrt{E\left[\Delta x^{2}\right]_{t-1}+\epsilon}$$
And finally:

  [...] approximate $\Delta x_{t}$ by compute the exponentially
  decaying $\text{RMS}$ over a window of size $w$ of previous $\Delta x$ to
  give the ADADELTA method:
  $$\begin{matrix}\Delta x_{t}=-\frac{\text{RMS}\left[\Delta x\right]_{t-1}}{\text{RMS}\left[g\right]_{t}}g_{t} &  &  & (14)\end{matrix}$$
  where the same constant $\epsilon$ is added to the numerator $\text{RMS}$ as
  well. This constant serves the purpose both to start off the first
  iteration where $\Delta x_{0}=0$ and to ensure progress continues to
  be made even if previous updates become small.[...]The numerator acts as
  an acceleration term, accumulating previous gradients over a
  window of time [...]

I.e. if the gradient in step $r$ is $g_{r}=\left(\begin{gathered}a_{r}\\
b_{r}\\
c_{r}
\end{gathered}
\right)$ and $\Delta x_{r}=\left(\begin{gathered}i_{r}\\
j_{r}\\
k_{r}
\end{gathered}
\right)$, then:
$$\begin{gathered}\Delta x_{t}=-\frac{\text{RMS}\left[\Delta x\right]_{t-1}}{\text{RMS}\left[g\right]_{t}}g_{t}=-\frac{\sqrt{E\left[\Delta x^{2}\right]_{t-1}+\epsilon}}{\sqrt{E\left[g^{2}\right]_{t}+\epsilon}}g_{t}=\\
\\
-\frac{\sqrt{\rho E\left[\Delta x^{2}\right]_{t-2}+\left(1-\rho\right)\Delta x_{t-1}^{2}+\epsilon}}{\sqrt{\rho E\left[g^{2}\right]_{t-1}+\left(1-\rho\right)g_{t}^{2}+\epsilon}}g_{t}=\\
\\
-\frac{\sqrt{\rho\left(\rho E\left[\Delta x^{2}\right]_{t-3}+\left(1-\rho\right)\Delta x_{t-2}^{2}\right)+\left(1-\rho\right)\Delta x_{t-1}^{2}+\epsilon}}{\sqrt{\rho\left(\rho E\left[g^{2}\right]_{t-2}+\left(1-\rho\right)g_{t-1}^{2}\right)+\left(1-\rho\right)g_{t}^{2}+\epsilon}}g_{t}=\\
\\
-\frac{\sqrt{\rho^{2}E\left[\Delta x^{2}\right]_{t-3}+p^{1}\left(1-\rho\right)\Delta x_{t-2}^{2}+p^{0}\left(1-\rho\right)\Delta x_{t-1}^{2}+\epsilon}}{\sqrt{\rho^{2}E\left[g^{2}\right]_{t-2}+p^{1}\left(1-\rho\right)g_{t-1}^{2}+p^{0}\left(1-\rho\right)g_{t}^{2}+\epsilon}}g_{t}=\\
\\
-\frac{\sqrt{\rho^{t-1}E\left[\Delta x^{2}\right]_{0}+\overset{t-1}{\underset{r=1}{\sum}}\rho^{t-1-r}\left(1-\rho\right)\Delta x_{r}^{2}+\epsilon}}{\sqrt{\rho^{t-1}E\left[g^{2}\right]_{1}+\overset{t}{\underset{r=2}{\sum}}\rho^{t-r}\left(1-\rho\right)g_{r}^{2}+\epsilon}}g_{t}
\end{gathered}
$$

$\rho$ is a decay constant, so we choose it such that $\rho\in\left(0,1\right)$ (typically $\rho\ge0.9$).
Therefore, multiplying by a high power of $\rho$ results in a very small number.
Let $w$ be the lowest exponent such that we deem the product of multiplying sane values by $\rho^w$ negligible.
Now, we can approximate $\Delta x_{t}$ by dropping negligible terms:
$$\begin{gathered}\Delta x_{t}\approx-\frac{\sqrt{\overset{t-1}{\underset{r=t-w}{\sum}}\rho^{t-1-r}\left(1-\rho\right)\Delta x_{r}^{2}+\epsilon}}{\sqrt{\overset{t}{\underset{r=t+1-w}{\sum}}\rho^{t-r}\left(1-\rho\right)g_{r}^{2}+\epsilon}}g_{t}=\\
\\
-\frac{\sqrt{\overset{t-1}{\underset{r=t-w}{\sum}}\rho^{t-1-r}\left(1-\rho\right)\left(\begin{gathered}i_{r}^{2}\\
j_{r}^{2}\\
k_{r}^{2}
\end{gathered}
\right)+\epsilon}}{\sqrt{\overset{t}{\underset{r=t+1-w}{\sum}}\rho^{t-r}\left(1-\rho\right)\left(\begin{gathered}a_{r}^{2}\\
b_{r}^{2}\\
c_{r}^{2}
\end{gathered}
\right)+\epsilon}}\left(\begin{gathered}a_{t}\\
b_{t}\\
c_{t}
\end{gathered}
\right)\\
\downarrow\\
\Delta x_{t}\approx-\left(\begin{gathered}\frac{\sqrt{\overset{t-1}{\underset{r=t-w}{\sum}}\rho^{t-1-r}\left(1-\rho\right)i_{r}^{2}+\epsilon}}{\sqrt{\overset{t}{\underset{r=t+1-w}{\sum}}\rho^{t-r}\left(1-\rho\right)a_{r}^{2}+\epsilon}}a_{t}\\
\\
\frac{\sqrt{\overset{t-1}{\underset{r=t-w}{\sum}}\rho^{t-1-r}\left(1-\rho\right)j_{r}^{2}+\epsilon}}{\sqrt{\overset{t}{\underset{r=t+1-w}{\sum}}\rho^{t-r}\left(1-\rho\right)b_{r}^{2}+\epsilon}}b_{t}\\
\\
\frac{\sqrt{\overset{t-1}{\underset{r=t-w}{\sum}}\rho^{t-1-r}\left(1-\rho\right)k_{r}^{2}+\epsilon}}{\sqrt{\overset{t}{\underset{r=t+1-w}{\sum}}\rho^{t-r}\left(1-\rho\right)c_{r}^{2}+\epsilon}}c_{t}
\end{gathered}
\right)
\end{gathered}
$$


Recognition of numbers from audio I am interested in the question, is there any research on the recognition of numbers from audio?
For example, there are call records and you need to get the numbers that speak on the phone.

Not aware of research per se but I did this in C# before. The concept you need to know about to recognize only certain patterns in speech is "Grammar".

Grammars in terms of speech or text recognition consist of multiple rules and word tokens that can be identified by your speech recognizer.

As further inspiration for your idea maybe look at SpeechRecognizer and the Grammar class in the VB.net Framework.

ie:
https://docs.microsoft.com/en-us/previous-versions/office/developer/speech-technologies/hh361633(v=office.14)?redirectedfrom=MSDN


Training LSTM for time series prediction with nan labels I have a time series of features $x_1,x_2,x_3,...,x_n$. I want to make a prediction $y_1,y_2,y_3,...,y_n$ for each timestep. However, in my training data some of the $y$ can be nan. I'd like the fit to just ignore these (i.e. the cost for this pair measured $y$ and predicted $y$ is zero).
I'm currently using tensorflow through Keras.

Is there an analogue of the masking layer for the label? I'm currently using tensorflow through Keras. Alternatively, it might be possible to change the loss function, but I don't know how, expecially while retaining numerical efficiency.

I suggest implementing it this way :

Set the nan value to 0 or any other value
when compiling keras model use parameter sample_weight_mode='temporal'
You can use masking on top of this by supplying the weight as the mask (sequence of values 1 if not nan 0 otherwise).

The steps above should give you the desired result.


Is there any point in learning Hadoop in 2018? I'm a data analyst/scientist working mostly with the Python open source stack i.e. Pandas, scikit-learn, matplotlib, seaborn etc. I want to expand my toolbox and learn a distributed computing framework.

Hadoop had created a lot of fuss in around 2013-2014. As per my limited knowledge on the subject, Apache Spark has improved upon Hadoop multi-fold in all aspects.

So, apart from the obvious case when you need to maintain legacy Hadoop applications, is there any reason as of today to prefer Hadoop over Apache Spark?

At this point of time, if I had to start a project from scratch and I had to choose between Hadoop and Spark, I would certainly choose Spark over Hadoop. There are several reasons for this:

Spark is more efficient than Hadoop given the fact that processing in Spark is in-memory whereas Hadoop requires to store intermediate results in disk.
Spark provides dozens of different operations and it is not constrained to just Hadoop's map-reduce.|

However, I still think that there is value in learning Hadoop before trying to learn Spark, even if it is at a high level. These are some reasons:

You may came across some legacy applications/systems based on Hadoop technologies
It provides a gentle introduction to some of the concepts used in Spark
Spark is sometimes used in combination with some technologies in the Hadoop ecosystem, like Hive and HDFS (Spark does not incorporate a way of storing data, but it can fetch data from multiple sources, including HDFS, and a Spark cluster with HDFS based data storage is a usual combination).


Metrics for unsupervised doc2vec model I have just built a simple doc2vec model using the gensim library, pretty much followed the tutorial located here.

The methods provided for checking the quality of the model are very manual and require reading the similar docs, is there a way to calculate some other metrics from the model to try and improve its performance?

Evaluation is based on the task, not the type of model used for it. In the tutorial that you link the task would be simple document similarity. Afaik a more common variant is the information retrieval setting, where the goal is to rank documents according to their similarity against a query document.

The tutorial mentions that they use a test set for which a link to a paper is provided. This paper explains that human annotators rated the similarity level between pairs of documents. That would be the standard way to obtain annotated data, but of course it takes a lot of manpower. It doesn't seem that the ground truth information is provided with the test set used in the tutorial though, I assume that this is why the author only proposes a manual inspection of the results.

In general there are probably many benchmark datasets (with ground truth annotations) publicly available for similar tasks, but they are not always easy to find and I'm not very knowledgeable about these. I think a good place to start would be the SemEval series of competitions, every year they release various datasets related to this kind of task.


Tread wear detection I have an assignment in which I have to provide a solution to measure the tread depth of a tire using a single rgb image.

I thought of two possible solutions:

Using CNN to measure the image similarity between a tire image and an image of a known depth.
Using a regression model to predict the tread depth.

Is there any other method I could try that can work over a large dataset?

Any suggestions would be helpful.

I understand the data you have is just a bunch of unlabeled images of tires and you need to predict the tread depth.

Your solution 1 requires a different dataset of labeled images. But if you had that, you could use it to train a model and forget about your unlabeled data. If the images are not very similar, maybe you could label a few and use transfer learning to tune the last layers of your network to the actual data.

Solution 2, treating it as a regression problem, also requires labels. But I think it’s the correct approach. You need to label the data. If labeling is a lot of work, then perhaps you can train a model on a different but similar problem. I can’t think of any similar problems that might be useful, so I would say you’re down to putting in some manual labor and label a part of your dataset.

When that’s done, I would expect that a good solution is a CNN with some FC layers to generate the prediction. You’re not going to have too many labels though, so you want to use a model that is as simple as possible.

There could be some domain-related knowledge that you can use, like the fact that tires have markers on them, in the tire grooves, and if these markers are flush with the outer surface, then you know that the tread depth is at or below minimum and the tires need to be replaced. But I don't think that's the question you're asking, so I would insist you need to label the dataset.


Best Data Science & Machine Learning Conferences 2019? What are the best data science conferences you have attended? I am looking for a conference that will help me further develop my Data Science and Machine Learning competencies. Topics that are particularly interesting to me include model explainability, model pipelines, and model governance. I am specifically looking for a conference that have hands-on Spark training. Do you have any conference suggestions that align with my interests?

Any of the Strata conferences (East/West/Europe). Should hit all of your requirements.


Why Decision Tree boundary forms a square shape and SVM a circular/oval one? I was going through a Udacity tutorial wherein a few data points were given and the exercise was to test which of the following models best fit the data: linear regression, decision tree, or SVM. Using sklearn, I was able to determine that that SVM is the best fit followed by decision tree. I got a very distinct decision boundary when these two algorithms were applied:

Is there any specific reason for the said shapes or does it just depend on the data sets?

The code was quite straightforward; just reading the CSV, separating the features and then applying the algorithms as shown below:

from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.svm import SVC

import pandas
import numpy

# Read the data
data = pandas.read_csv('data.csv')

# Split the data into X and y
X = numpy.array(data[['x1', 'x2']])
y = numpy.array(data['y'])

# import statements for the classification algorithms
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.svm import SVC

    # Logistic Regression Classifier
    classifier = LogisticRegression()
    classifier.fit(X,y)

    # Decision Tree Classifier
    classifier = GradientBoostingClassifier()
    classifier.fit(X,y)

    # Support Vector Machine Classifier
    classifier = SVC()
    classifier.fit(X,y)

Shape of the SVM decision boundary depends on the kernel (similarity function) used. The "standard" version of SVM has linear decision boundary. The one displayed could be using Gaussian kernel.

Decision boundary of a decision tree is determined by overlapping orthogonal half-planes (representing the result of each subsequent decision) and can end up as displayed on the pictures.

See more here:

https://shapeofdata.wordpress.com/2013/07/02/decision-trees/

https://www.quora.com/What-are-Kernels-in-Machine-Learning-and-SVM


Do I need to convert booleans to ints to enter them in a machine learning algorithm? My dataset contains a lot of columns with booleans do I really need to change them so I can insert them into the algorithm?

I'm gonna use KNN right now but will test other algorithms later so I'm trying to ready up my dataset

In Python, True and False are cast implicitly into integers:

True == 1     # True!
False == 0    # True!

Although they are not the same objects - you can test this with True is 1, which returns False.

This means that an algorithm running in pure Python should work without conversion. Many libraries/algorithms have some part implemented in C/C++ in the background, in which case you might run into problems.

You could try the model on your Pandas DataFrame as boolean. If it crashes, you know you must convert to integers/floats.

Even if it doesn't crash, you could convert the values to integers or floats and run it for comparison. Here is a short example:

In [1]: import pandas as pd
In [2]: df = pd.DataFrame({'a':[0, 2, 3, 4, 5], 'b':[True, False, True, False, False]})
In [3]: df
Out[3]:
   a      b
0  0   True
1  2  False
2  3   True
3  4  False
4  5  False

Convert everything to boolean

In [4]: df.astype(bool)
Out[4]:
      a      b
0  False   True
1  True   False
2  True    True
3  True   False
4  True   False

In [5]: df.astype(float)
Out[5]:
     a    b
0  0.0  1.0
1  2.0  0.0
2  3.0  1.0
3  4.0  0.0
4  5.0  0.0


Which is the best Machine learning technique for this Load forecasting problem? I am trying to use Machine Learning to predict the load of a residence at any point in time for a whole year. I have past data pertaining to that house. So I have the training data and I need the algorithm to predict future loads of the house.

Based on my knowledge, I have found the "supervised" machine learning technique to be the one I must adapt. I figured this out since I have labelled test data, I have a prediction requirement and I can get feedback for my prediction (cross-checking with the actual value). Am I correct here?

Also, I read online that "Unsupervised" learning is to be used at places where we need to find "Hidden data structure". I assume it means pattern. If so, what is the difference between the unsupervised and supervised learning in my case. Both of them will give me a prediction about the future load pertaining to that house at any point in time.

My background

I am doing my Masters in EE (Power systems). I am new to Machine Learning as well.

I think this is one of the time series forecasting because you want to predict the future load of residence with the past dataset of residence's load by time. By my experience I recommend using LSTM RNN for the solution. An LSTM is well-suited to classify, process and predict time series given time lags of unknown size and duration between important events. Of course, this is supervised learning.

As for difference between supervised learning and unsupervised learning on your case, you don't need to use unsupervised learning for this solution because you have already dataset with input/correct output. if you are training your machine learning task for every input with corresponding target, it is called supervised learning, which will be able to provide target for any new input after sufficient training. Contrary, if you are training your machine learning task only with a set of inputs, it is called unsupervised learning, which will be able to find the structure or relationships between different inputs. It says you should use supervised learning - RNN or CNN.

PS: as for framework, I recommend using Tensorflow or Keras.


How to calculate the Probability for the Unconditional Node in the Bayesian Belief Network? In the popular example for Bayesian Belief network Burglary Alarm how is the probability for burglary P(B) and earthquake P(E) calculated as 0.001 and 0.002 respectively?

Is it an assumption made or there is some calculation involved ? I can understand the conditional probabilities for the child nodes but not sure how the probability for the nodes Burglary and Earthquake are getting calculated?

As the comment says, Burglary and Earthquake only have prior probabilities in the Bayes net. These priors are not calculated from other variables/distributions. They are just assumptions in this example, or, sugarcoated, expert knowledge. In fact, it is often hard to find good priors (from expert knowledge). Extracting them from large non-biased datasets can help. Anyway, one needs to understand the dataset and the context/circumstances/constraints of its collection.


Cross validation when training neural network? The standard setup when training a neural network seems to be to split the data into train and test sets, and keep running until the scores stop improving on the test set.  Now, the problem:  there is a certain amount of noise in the test scores, so the single best score may not correspond to the state of the network which is most likely to be best on new data.

I've seen a few papers point to a specific epoch or iteration in the training as being "best by cross-validation" but I have no idea how that is determined (and the papers do not provide any details).  The "best by cross-validation" point is not the one with the best test score.

How would one go about doing this type of cross validation?  Would it be by doing k-fold on the test set?  Okay, that gives k different test scores instead of one, but then what?

I couldn't say what the authors refer to by

  best by cross-validation

but I'll mention a simple and general procedure that's out there:

You are correct that analyzing one estimate of the generalization performance using one training and one test set is quite simplistic. Cross-validation can help us understand how this performance varies across datasets, instead of wonder whether we got lucky/unlucky with our choice of train/test datasets.

Split the whole dataset into k folds (or partitions), and train/test the model k times using different folds. When you're done, you can compute the mean performance and a variance that will be of utmost importance assessing confidence in the generalization performance estimate.


Textbook definition of Data augmentation I'm trying to find a textbook definition of Data augmentation. We all know what it is, but I'm having a hard time finding a reliable source which I can cite for my paper:

The Wikipedia article on Data augmentation misquotes an article by Shorten & Khoshgoftaar which doesn't actually present a definition.
There are many definitions in blog posts, but these are not reliable sources.

It doesn't help that there is a Markov chain Monte-Carlo algorithm with the same name cluttering up search results.
Any clues?

How about the definition from "The Art of Data Augmentation" by David A van Dyk & Xiao-Li Meng:
The term data augmentation refers to methods for constructing iterative optimization or sampling algorithms via the introduction of unobserved data or latent variables.
Citation:
David A van Dyk & Xiao-Li Meng (2001) The Art of Data Augmentation, Journal of Computational and Graphical Statistics, 10:1, 1-50, DOI: 10.1198/10618600152418584


Why is it wrong to train and test a model on the same dataset? What are the pitfalls of doing so and why is it a bad practice? Is it possible that the model starts to learn the images by heart instead of understanding the underlying logic?

Yes, you put it quite correctly.
As a teacher, you wouldn’t give your students an exam that’s got the exact same exercises you have provided as homework: you want to find out whether they (a) have actually understood the intuition behind the methods you taught them and (b) make sure they haven’t just memorised the homework exercises.


Is there any standard or normal range for the amount of LSTM loss function? I am working on a LSTM network that I get loss amounts around 4.7 e-4 . It seems adding more layers and increasing epochs don't help to decreasing it. I also using a Dropout = 0.2 for each of my layers and implemented all the jobs with Keras library.

I like to know about this loss amount? is this large or is OK> Are there any rule of thumb for loss?

And why I can't decrease my loss amount? Is there any problem here?

Although your loss function is an indication of how well the model is training, usually one uses other more intuitive metrics to assess how good the model is.

If you are looking to a classification problem, your loss function is most probably the cross entropy. In what regards the loss function what matters is to understand its behaviour during training, more than its value.

A loss function that reduces its value during training is an indication that the model is effectively training. The loss function will, at some point, start reducing its value, and that means that the model has arrived to a minimum. One need to understand also the interaction of the loss function in training and validation set, and how to detect things like overfitting. If you are not aware of that, there is plenty of literature about the topic.

To know how good a model is, I would use other metrics that give a better indication and intuition. For example, in a classification problem, pne can look to things like Precision, Recall, Accuracy (if the classes are not very unbalance) or even ROC AUC. If it is a regression problem, maybe you are more interested in Mean or Median Absolute Percentage Error (MdAPE or MAPE).


API for historical housing prices I'm looking for an (ideally free) API that would have time series avg/median housing prices by zip code or city/state.

Quandl almost fits the bill, but it returns inconsistent results across different zip codes and the data is not as up to date as I'd like (it's mid November, and the last month is August).

I also looked at Zillow, but storing their data is against TOS, and at 1,000 calls daily--it would take forever to pull in the necessary data.

Any suggestions (even if they aren't free) would be much appreciated!

The United States Census Bureau has many free housing datasets (some of which are updated more than once every 10 years). There is an API for American Community Survey 1 Year Data that includes housing data. There are raw data sets at American Fact Finder.


Why does exploration in DQN not lead to instability? Why does action exploration in DQN not lead to instability?

I see in DQN algorithms, that it selects random actions even after some iterations.

My question is how does this approach not lead to instability? Even the final value of epsilon (the probability of a random action) is non-zero!

If we set the final epsilon=0.01, we will select a random the action one time in a hundred, which means that we are not going to get the same answer in different runs and probably our answers are not stable. Why does this still work?

All Reinforcement Learning (RL) control algorithms have to deal with the exploration/exploitation trade-off. Do you trust that you have learned enough about the environment that you can act optimally, or do you still need to check alternative actions.

Related to this, there are two basic approaches when trying to learn an optimal policy in RL. An agent can either learn:

on-policy where the agent assesses the actions that it is taking, and adjusts behaviour directly. To solve the exploration/exploration trade-off with on-policy learning, a common approach is to decay $\epsilon$ (or a similar parameter which controls the amount of exploration), based on some idea of how much learning is necessary to learn the task.
off-policy where the agent has separate behaviour and target policies. The agent takes actions from an exploring behaviour policy, and adjusts the results it gets mathematically so that it calculates a different target policy, which is often optimal in some way.

Q learning is an off-policy algorithm, which learns the action value function of following deterministic greedy action choices at all times. In the simplest single-step version, the adjustment required is really simple - instead of estimating the action value based on what the behaviour policy does, Q learning updates using whatever the best next action would be, regardless of what gets chosen.

Q learning is proven to be stable and to converge when using a table of q values (i.e. not a neural network). However it is actually not very stable when used with neural networks for estimating $\hat{q}(s,a, \theta)$ (where $\theta$ is the NN weights). This is why DQN needs to use experience replay and a separate target network - in order to combat the instability. The instability is not directly caused by random actions though, it is more complex than that.

  If we set the final epsilon=0.01, we will select a random the action one time in a hundred, which means that we are not going to get the same answer in different runs and probably our answers are not stable. Why does this still work?

The agent will not behave consistently on each run, as it is still exploring. However, if it has been set up correctly, it will still converge to and learn an optimal policy, and in fact the constant exploration should help it get values for non-optimal action choices correct, and maintain the correct maximising actions.

When you use a Q learning policy after it has finished learning, typically you would set epsilon to zero and have the agent act greedily.

It is worth noting that even with a deterministic policy, RL is designed around the concept of an environment where random events can happen. For instance it will learn the best way to move when navigating, even if sometimes the distance travelled is not the same due to random events. When you test an agent in a non-deterministic environment, then different things will happen each time, and this is not a problem for the learning process. It might be a problem for you to evaluate the agent - you would need to run a test multiple times and take an average to assess how well it is performing.


Chossing between gradient boosting algorithms I just stepped in machine learning competitions and it looks like most of the mid-sized dataset competitions are won by Gradient boosting based models. However I came accross case where LightGBM,Catboost or Adaboost had very different scores.
Is there a method to choose between those algorithms?

I would say Catboost and lightgbm perform similarly and its purely a matter of choice.
Some of my colleagues prefered Catboost when dataset has lots of categorical columns, but I rarely saw any advantage over lightgbm.
there is a great article comparing CatBoost vs. Light GBM vs. XGBoost
https://towardsdatascience.com/catboost-vs-light-gbm-vs-xgboost-5f93620723db
And another article comparing even more boosting algos
https://medium.com/@divyagera2402/boosting-algorithms-adaboost-gradient-boosting-xgb-light-gbm-and-catboost-e7d2dbc4e4ca


np.shape[::-1] reversal of dims and np.moveaxis splitting array I cannot find anything in the documentation but it was used in some starter code for a class I am taking at school.
Upon testing per below, it seems to reverse the order of the dimensions of an numpy array.

pic = np.ones((3,4,5))
print(pic.shape,"\n", pic)
#new_shape is the reverse of pic.shape
new_shape = pic.shape[::-1]
print(new_shape)

This outputs:

(3, 4, 5)
 [[[1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]]

 [[1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]]

 [[1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]]]
(5, 4, 3)

In the starter code it was used in this way:

res_img = cv2.resize(img, sub_img.shape[::-1])

There are some other tricks that are planted in the starter code (to show us that they are there I'm guessing).  Here is another example:

channels = blue, green, red = np.moveaxis(color_img, 2, 0) #move channels to pos 0

The np.moveaxis() moved channels to the first dim position but it's assigned to blue, green and red at the same time via Python multiple assignment?  When I looked up multiple assignment, I only saw stuff like var1, var2 = 1, 2.  Also np.moveaxis() returns an array with the new shape so I have to stretch my imagination to assume the first axis can be assigned to variables.  Is this correct?

My instructor just replied that np.split() does the splitting but the documentation does not say anything about this being implicitly called when np.moveaxis() is called.

Does anyone know what is really happening?

Note that when you do:

pic.shape[::-1]

You only reverse the elements of the shape are of pic. So pic.shape is (3,4,5] and reversed, it's (5, 4, 3).

What happens with the multiple assignment is just that you can assign the elements along the first index to individual variables. For example:

pic = np.array( [['x1','x2','x3'],['y1','y2','y3']])

The first index can be 0 or 1, the second index can be 0, 1 or 2. I can assign the x-vector to a and the y-vector to be like this:

a, b = pic

In [18]: a
Out[18]: array(['x1', 'x2', 'x3'], dtype='<U2')
In [19]: b
Out[19]: array(['y1', 'y2', 'y3'], dtype='<U2')

The call to moveaxis reformats the image, by moving axis 0 to position 2 in the array of axes. My pic doesn't have a position 2 (since it only has 2 dimensions), but I can swap axis 0 with axis 1:

In [11]: np.moveaxis(pic, 1, 0)
Out[11]:
array([['x1', 'y1'],
   ['x2', 'y2'],
   ['x3', 'y3']], dtype='<U2')

And now I can assign (x1, y1) to a, (x2, y2) to b and (x3, y3) to c:

a, b, c = np.moveaxis(pic, 1, 0)

In [14]: a
Out[14]: array(['x1', 'y1'], dtype='<U2')

There is no call to split anywhere here. I think what confused you is just the multiple assignment from a numpy array, which allows the direct assignment of columns to individual variables. In combination with moveaxis this would allow you to split the red, green and blue images in a single command.


Unable to reduce loss when doing sentiment analysis with Keras I created a model in Keras that predicts 4 sentiments/emotions based on text input. Size of my data:
label_1 : 100.000
label_2 : 100.000
label_3 : 100.000
label_4 : 50.000

Validation data: 45.000

I have set the class weights to: class_weight = {'label_1':1, 'label_2':1, 'label_3':1, 'label_1':2}
I have used scikit-learn for vectorising with CountVectorizer. For preprocessing, I have converted all to lowercase, removed emails, urls/links and removed stopwords.
This is my model:

model = Sequential()
model.add(Dense(50, input_dim = features.shape[1], activation = 'relu')) # input layer requires input_dim param
model.add(Dense(100, activation = 'relu'))
model.add(Dense(50, activation = 'relu'))
model.add(Dropout(0.5))
model.add(Dense(4, activation='softmax'))

opt = SGD(lr = 0.001, momentum = 0.01)
model.compile(loss="categorical_crossentropy", optimizer = opt, metrics=['accuracy'])

es = EarlyStopping(monitor='val_loss', min_delta = 0.03, patience = 120, verbose=1, mode='auto')
history = model.fit(features, results, validation_split = 0.25, shuffle = True, class_weight = class_weight, epochs = 600, batch_size=512, verbose=2, callbacks=[es])

score = model.evaluate(x_test, y_test, batch_size=512)
print()
print(history.history.keys())
print()
print(score)
print('Test loss:', score[0],  'Test accuracy:', score[1])

Test loss: 0.215 Test accuracy: 0.900

These are the accuracy and loss graphs:

This is my prediction:
validation_features = transformerVectoriser.transform(validation_features)
prediction = model.predict_classes(validation_features , batch_size=512) # making prediction

These are confusion_matrix and accuracy_report:
[[11678   256  1181    23]
 [  477 12023   432    13]
 [ 1538   322 10947    18]
 [   16     9    17  6050]]

precision    recall  f1-score   support

0       0.85      0.89      0.87     13138
1       0.95      0.93      0.94     12945
2       0.87      0.85      0.86     12825
3       0.99      0.99      0.99      6092

accuracy                    0.90     45000
macro avg. 0.92   0.92      0.92     45000
weighted avg  0.91  0.90    0.90     45000

My question is, how can I reduce loss? I have tried to change number of layers, nodes and dropout layers, to change optimisers, learning rate and momentum, to change number of epochs and batch size, and to change max number of words (I have tried with 4.000, 5.000, 6.000, 8.000, 10.000, 12.000)
It does not matter what I change, my accuracy is always around 87%-90% (and I think thats good), but my loss is always around 0.21-0.24 (and I do not like that). On some data science related blogs I have read that loss should be less than 0.1. Is that true? What is acceptable loss value for multiclass classification?
Do you have any advices?

There is no universal loss value result that you can or should achieve. Beside your model it will heavily depend on the data that you have.
What is good or bad result is subjective and depends on your use-case.
It's a very broad question how to improve the result of a neural network and you should be more specific but some ideas that can work generally:

Preprocess your data differently (i.e. use TF-IDF or use text as a sequence)
Collect more data (neural networks usually prefer that, also your test set seems small)
Use different optimizer (i.e. Adam)
Use different architecture (i.e. some RNN)
Use different algorithm (i.e. XGBoost)
Use BatchNormalization
Use some callback that modifies your learning rate
Tune your parameters


How to set class_weight parameter for cost sensitive learning? I'm dealing with a binary classification problem with a balanced data set, however false positives are much more costly than false negatives. Let's just say that an FP is in general 3 times more costly than an FN and the response variable = 1 means a positive identification. How should I set the class_weight parameter in sklearn's RandomForest to reflect this ? From my understanding, I would say:
class_weight = {0:1.0,1:3.0}

I'm not sure if I understand this parameter correctly or should it be the inverse ?
Thanks.

\begin{array} {|r|r|}
\hline
- &(1) &(0) \\
\hline
(1) &- &FP\\
\hline
(0) &FN &-\\
\hline
\end{array}
Rows elements are Y_true and column is Y_pred.
FP means, we predicted Positive and it came out False i.e. Class was Negative(0 here).
It means we don't want the Model to mis-classify the Negative class.
So, it implies, we will put a bigger penalty for the Negative class. Hence,
class_weight = {0:3.0.0,1:1.0}


Is hyperparameter tuning more affected by the input data, or by the task? I'm working on optimizing the hyperparameters for several ML models (FFN, CNN, LSTM, BiLSTM, CNN-LSTM) at the moment, and running this alongside another experiment examining which word embeddings are best to use on the task of binary text classification.

My question is: should I decide on which embeddings to use before I tune the hyperparameters, or can I decide on the best hyperparameters and then experiment with the embeddings? The task remains the same in both cases.

In other words, is hyperparameter tuning more affected by the task (which is constant) or by the input data?

In other words, is hyperparameter tuning more affected by the task (which is constant) or by the input data?

It's correct that the task is constant, but hyper-parameters are usually considered specific to a particular learning algorithm, or to a method in general. In a broad sense the method may include what type of algorithm, its hyper-parameters, which features are used (in your case which embeddings), etc.

The performance depends on both the data and the method (in a broad sense), and since hyper-parameters are parts of the method, there's no guarantee that the optimal parameters stay the same when any part of the method is changed even if the data doesn't change.

So for optimal results it's better to tune the hyper-parameters for every possible pair of ML model and words embeddings. You can confirm this experimentally: it's very likely that the selected hyper-parameters will be different when you change any part of the method.


How to detect Anomalies in raw level data? I have a dataset that is growing at about 30 million rows per day.

The data schema is follow:

day_hour_timestamp timestamp,
campaign_id        varchar,
creative_id        int,
uuid               varchar,
device             varchar,
country            varchar,
os                 varchar,
browser            varchar

Each record in the dataset is information about an advertisement impression.

For example, if I visited a particular advertisement at a website on 10/11/2017 13:30. I will be one datapoint in the dataset for 10/11/2017 hour 13:00 along with my device, country, os, browser, etc.

My question is how do detect anomalies within this dataset? Anomalies such as data not normal to the columns, for example if the I receive an OS that is NULL or browser is 'Apple'.

Should I create a heuristic that checks every column or is there a machine learning algorithm I can apply?

Since this dataset is pretty big would checking each data point be expensive?

Should I create a heuristic that checks every column or is there a machine learning algorithm I can apply?

I would go with unsupervised or semi-supervised learning. Use 'normal' data as training data and then predict what is normal and what is abnormal. Depending on your needs you can fine-tune the cut-off.

One-class SVM should work. Or a combination of one-class SVM and SV Data-Description for possibly better results.


Is providing class weight to neural network enough for imbalanced binary classification? I have a highly imbalanced binary classification problem, probably 95:5 for two classes. I don't want to perform resampling as the data is already huge and training it would just take more time. (I'm also aware of down sampling)
But my question is , is providing class weights (let's say computed by scikit-learn's compute class weight) enough?
or there is any other method ?
model.fit(X,y,class_weight=class_weight)

Referring to an answer to a similar question, you don't have any reason to handle unbalance from the beginning. An imbalance of 95:5 isn't that big, I'd start with the regular training and if that doesn't work try more sophisticated things.


(Serious) Dataset of paedophilic Youtube comments (or similar)? I think it would be useful to create a model that tries to predict whether a youtube comment is paedophilic - maybe the model should also take into account the channel name/description/front image.
It's not an easy task but at the moment I'm just looking for data.
I know it's a sensitive topic - but does anyone know of a dataset out there with the characteristics I need?
Thanks

So your question is asking how to go about making a model which predicts whether the YouTube comment is paedophilic given the text provided in the YouTube comment itself.
Before I provide an answer, it is worth considering the ethical applications of this model and what this will be used for because models do implicitly cause bias in all stages of model production, from data collection to model implementation. So before taking this task on, I urge to read this paper on model ethics (https://arxiv.org/abs/1901.10002)
Firstly, you obviously need data/YouTube comments. The most important part to consider is labelling. Now, this stage is particularly important as you want to get multiple people involved with labelling these comments as (non-)paedophilic.
In this step, ensure that you use comments which are have a higher inter-labeller agreement.
For the model, something like a RNN/LSTM with a many-to-one configuration with a final 2 node softmax layer after the sequential architecture has been fed all of the text input. Again, before you start such an endeavour, do be aware of the ethical consequences of this model.


How to use live data to improve a existing model? I am using logistic regression to train a model to predict 'click/non-click' using ['browser info', 'publisher info', , 'location', 'time', 'day'].

I wanted to know the ways in which I can use the new live data to improve the improve the already trained model.

Does a solution exist which takes into account - change in feature set?

Suppose you have a model that has been trained on $N$ data over $E$ epochs. This means that the model has seen each of the $N$ examples, $E$ times.

Now say you got $M$ more training data. Normally you would want to train the new ones for $E$ epochs as well.

However if $N$ and $M$ don't come from the same underlying distribution (or don't represent it adequately), this would result in the model "forgetting" the first $N$ examples and "paying more attention" to the latter $M$ ones.

You could try training your model for $<E$ epochs, so that it learns the latter but doesn't forget the former, but that is purely empirical and very hard to achieve in practice.

You can a few things to avoid this:

Retrain your whole model using both the $N+M$ examples (which you would shuffle). This would require a new complete training of the model on regular occasions and would be increasingly difficult to train (due to the ever increasing size of the training data). This is a very inefficient solution and wouldn't work for any on-line training application
Make use of a model that supports on-line training. Some algorithms support incremental (on-line) training, without you needing to retrain the whole thing. A scikit-learn comparison is available.
Customize an algorithm so that is has the desired effect. For example, you could train a linear SVM incrementally, with a large regularization penalty and an SGD classifier. This is discussed in more detail for scikit-learn here.


How to find slope of curve at certain points how to find slope at certain points circled in blue in below curve ? Are these below 2 approaches valid ? though they give different results .
How to automatically find the points where the slope changes drastically in curve  like around at point 5,6 in below graph
x=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, ]

y=[512, 256, 128, 64 , 32 , 16 , 8  , 7  , 6  , 5  , 4  , 3  , 2  , 1  ]

Numpy gradient give below result

np.gradient(y)
[-256. , -192. ,  -96. ,  -48. ,  -24. ,  -12. ,   -4.5,   -1. ,-1. ,   -1. ,   -1. ,   -1. ,   -1. ,   -1. ]

can we use numpy.gradient to find the slope of curve ? since finding slope of line and curve is bit different Shown in this link
2.Using custom slope function
def slope(x1, y1, x2, y2):
    m = (y2-y1)/(x2-x1)
    return m

slope_value=[]
for i in range(len(y)):
    i += 1
    v=slope(y[i], x[i], y[i-1], x[i-1])
    print(i,v)
    slope_value.append(v)

result: [-0.00390625,  -0.0078125,  -0.015625,  -0.03125,  -0.0625,  -0.125,  -1.0,  -1.0,  -1.0,  -1.0,  -1.0,  -1.0,  -1.0]

The numpy calculation is the correct one to use, but may be a bit tricky to understand how it is calculated
Your custom calculation is accidentally returning the inverse slope, the x and y values are reversed in the slope function (x1 -> y[i], etc). The slope should be delta_y/delta_x
def slope(x1, y1, x2, y2):
v=slope(y[i], x[i], y[i-1], x[i-1])

Also, you are calculating the slope at x = 1.5, 2.5, etc but numpy is calculating the slope at x = 1, 2, 3
In the gradient calculation, numpy is calculating the gradient at each x value, by using the x-1 and x+1 values and dividing by the difference in x which is 2. You are calculating the inverse of the x + .5 values.
x=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, ]

y=[512, 256, 128, 64 , 32 , 16 , 8  , 7  , 6  , 5  , 4  , 3  , 2  , 1  ]

numpy answer:
[-256. , -192. ,  -96. ,  -48. ,  -24. ,  -12. ,   -4.5,   -1. ,-1. ,   -1. ,   -1. ,   -1. ,   -1. ,   -1. ]

each calculation in numpy function:
-256 = (-256)/1 there is no x-1 value so the default is to use x
-192 = (128-512)/2
-96 =  (64-256)/2
...


Does it make sense to use a tfidf matrix for a model which expects to see new text? I'm training a model to classify tweets right now. Most of the text classification examples I have seen convert the tweets into tf-idf document term matrices as input for the model. However, this model should be able to identify newly collected tweets without retraining. Does it make sense to use tf-idf in this context? What is the correct way to turn tweets into feature vectors in this task?

The problem is not really "new text", since by definition any classification model for text is meant to be applied to some new text. The problem is out of vocabulary words (OOV): the model will not be able to represent words that it didn't see in the training data.

The most simple way (and probably the most standard way) to deal with OOV in the test data is to completely remove them before representing the text as features.

Naturally OOV words can be a serious problem, especially in data such as Twitter where vocabulary evolves fast. But this issue is not related to using TF-IDF or not: any model trained at a certain point in time can only take into account the vocabulary in the training data, it cannot guess how future words are going to behave with respect to the class. The only solution for that is to use some form of re-training, for instance semi-supervised learning.


Stratified Sampling for XGboost I have a multiclass-classification dataset with the target (dependent) variable highly imbalanced. While using the randomForest package in R, I usually use the parameters sampsize & strata to account for the imbalance in training data. Are there any similar options in xgboost package also?

Summary of the number of datapoints available in each class.

Factor 1     : 667
 Factor 2     : 676
Factor 3    :7807
 Factor 4       : 850

In R, it's an option of the cross validation function : xgb.cv
See the documentation here : https://www.rdocumentation.org/packages/xgboost/versions/0.4-4/topics/xgb.cv


model.fit indefinite execution in Google Colab I am working on image classification model. My model is build using CNN, tensorflow and Keras  libraries.

I am trying to execute my CNN multi-classification model in google Colab, and I am loading the data (image data) from my google drive (mount drive command). Everything is going perfectly fine, up to model.fit, at this point the model execution is indefinite (never-ending) in Google Colab. However the same model is working perfectly in Jupiter Notebook.

So, can anybody tell me what is happening?

I have solved the problem.

In case anybody else deals with similar situation, the solution is to assign a batch_size for training and validation images. Then add the steps_per_epoch and validation_set as parameters within model.fit.

steps_per_epochs= number_of_training_images/batch_size

validation_set = number_of_test_images/batch_size


Overall AUC higher than all "stratified" AUCs For one of my binary classification models, I have observed this (Simpson's Rule-esque) paradox.  The AUC on the test set as a whole is 0.8.
Gender is one of the model's features.  So I decided to produce a "bias" report, for which I calculated AUCs for each of the Male and Female subgroups.  But I noticed that each of these AUCs is around 0.7.  How is this possible, given that the overall test AUC is 0.8?  (In my dataset, every data point belongs to either the Male or the Female subgroup.)  I don't expect the overall AUC to simply be a (weighted) linear combination of AUCs for the individual strata.
I'm hoping to get both a technical/mathematical answer and a high-level explanation.  Please let me know if any further information is needed (if you think I should plot the overall, Male, and Female ROC curves, for instance).  Thank you!

AUC can be defined as $P(X_1 > X_0)$ where $X_1$ is the score of a randomly chosen positive instance and $X_0$ is the score of a randomly chosen negative instance.
Like in Simpson's "paradox", what you see could happen because the group has a relatively large effect on your target. For example, one group could be mostly positive, another could be mostly negative. The classifier might not work at all within each of the groups, but if it is merely able to predict the group we are in (or as in your case it is simply one of the inputs), it can have a high AUC.


Customer Targeting for CRM Marketing Campaign I need help or ideas to solve the below business challenge. Sample questions has been provided. A snapshot of the sample data has been attached below:

Can you share all available columns in the data set? its hard to tell what data is there to use.
Given the data you show, I would have started with some explaratory analysis:
Group data per shops, check user activity ( how much purchases they do per month/week, what is AOV, LTV, etc).
Idealy, its good to use email marketing, because its cheaper that paid ads and you have list of users that can be mailed - given the low costs of email marketing campaigns, you can basicaly target all users that stopped shopping in given places and allowed you to send them mails.

Additional approach: you can use paid ads for remarketing and geo-based ads to recover lost clients - given the fact that you want to cut the costs, its might be beneficial to retain only loyal customers ( which have more that 2 or 3 transactions, this number should be based on your average sales per client).

So, answering the questions:

What are your recommendations for the campaign? Should we do it? How should we target and why?
Should we do it?

Select all users that were lost after opening new shop and calculate lost income per month (or week, based on your data) - you want to know how much money you can expect from recovered users, this can be your marketing spend baseline ( Recoverd Income - Costs - Margin ~ Marketing Budget). Marketing Budget + Recoverd Income will indicate if its logical to run the campaign.

What are your recommendations for the campaign?

I would start with email - cheap and effective, we can use targeting in emails based on what items users buy, their gender and age.  If you have segment of loyal users with high income - you can develop dedicated email campaign based on their needs, give bigger discounts ( its good to know why they left in the first place).

Is there any additional customer information that would help you? What is it? How would it help?

Any data that will help to better segment users will be usefull - age, gender, social status, all this info can influence email campaigs.
Also it would be good to have data about costs and effectivness of Paid Remarketing Campaigns - based of this data we can try to use remarketing for our loyal customers.

Have both stores been impacted similarly? How has customer shopping behavior been affected?

Split data per shops, calculate statistics, do some good-looking and easy to undestand graphs, cant say more without actual data.

Hope this help!

EDIT

Thanks for sharing data, here is my opinion about this task:

You can split data before-after 25 of may, and target users that have losses in sales comparing to period before 25 of may.

Based on Customer_Data dataset, it looks like you have user that can be mailed and emailed, so you can launch 2 campaigns for different users based on Mailable_flag / Emailable_flag flags.

Talking about losses: Based on given data is looks like your shops lost around 3k in sales and ~60 transcations for both shops after 25 of may and the losses are equal amont users that can and connot be emailed.

This leads us to a point of "additional" data - ideally you want to know the past performance of mailed and emailed campaigns to adjust the targeting.

Also the thing I havent done - you should calculate LTV ( basicaly how much money the users brings you per month) - if some users have very low LTV, you dont want to spend money advertasing them.

I`ve attached ipython notebook with brief workflow for you to check . Please, be aware that its just my ideas and they cant be totally wrong :)


Pre-trained CNN for one-shot learning I'm currently trying to learn one-shot learning using convolutional neural networks. According to this video, the CNN that I use should have been pre-trained on the MNIST. Why must the CNN be pre-trained??

Because the layers in the CNNs must be already able to extract features from images.

Normally this procedure takes thousands of iterations to be completed, so if they were not trained we could't have the concept of one-shot learning.


Is it necessary to tune the step size, when using Adam? The Adam optimizer has four main hyperparameters. For example, looking at the Keras interface, we have:

keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False)

The first hyperparameter is called step size or learning rate. In theory, an adaptive optimization method should automatically modify the learning rate during optimization. Thus, I would expect lr to be a not very influential hyperparameter, which I can remove from the list of hyperparameters I have to tune on the validation set, thus saving some time.

Does this hold true in practice? I.e., is it true that, at least for a class of problems (say, image recognition), lr doesn't affect the optimization a lot, and thus we can just leave it at the default value of 0.001? Or is it still as extremely influential, as it is for SGD with momentum?

EDIT: to clear a misunderstanding brought up in an answer, learning rate and step size are synonymous. See the algorithm definition in the Adam paper: $\alpha$, whose default value is 0.001, is clearly named the step size. I reckon it's probably not a great name (the actual size of the step in the parameter space depends on the accumulated first order and second order momentums, as well as on the gradient, of course) but unfortunately such misleading terminology is the norm in optimization (at least in Deep Learning papers).

the lr (learning rate) parameter is the MOST influencial parameter for Adam (or any other optimization algorithm). I It does not mean step size... as that is not controlled by Adam but by the neural network and how you feed batches into it.

So yes, lr is very, very important, and it is the learning rate, not the step size.

UPDATE

Actually as mentioned by OP, the learning rate and step size are the same thing (my bad, I confused it with the batch size).

Generally, my recommendation is to try with learning rates of 0.0001, 0.001, 0.01 and 0.1.

The rule of thumb is that a smaller learning rate provides a much stable learning curve... but a much slower one while a larger learning rate will converge quicker... but it will become unstable. So you will have to do a bit of exploration. I normally use 0.001 and 0.0001 but that depends a lot on the problem you are dealing with.

Also, as compared with SGD, Adam includes momentum, essentially taking into account past behaviour (the further away such behaviour is in the past, the least relevant it is) that is controlled by the beta params, but I do not recommend to modify them, as normally it is not required. So, although Adam is a more stable learning method (and normally a quicker one) than SGD, the learning rate it is still the most fundamental parameter to play with.


Multiple regression (using machine learning - how plot data) I wonder how I can use machine learning to plot multiple linear regression in a figure. I have one independent variable (prices of apartments) and five independent (floor, builtyear, roomnumber, square meter, kr/sqm).

The task is first to use machine learning which gives the predicted values and the actual values. Then you have to plot those values in a figure.

I have used this code:

x_train, x_test, y_train, y_test = tts(xx1, y, test_size=3)

Outcome: LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None,
         normalize=False)

regr.fit(x_train, y_train)

Outcome:nothing

regr.predict(x_test)

Outcome: array([2.37671029, 3.91651234, 2.98472475])

np.mean((regr.predict(x_test) - y_test) ** 2)

Outcome: 2.976924398032532e-26

How can I plot the actual values of the dependent variable and the predicted ones in the same figure?

There might possibly be a better way but one way of doing this is to map the variable to different aesthetics of the graph. I used python and used library plotninne which is as I understand tries to mimic ggplot2 from R.

from plotnine import *
   # df is the data frame that contains all the vairables as columns
   (ggplot(df, aes('actual_value', 'predicted_value',
   color='(dependent_var2)',
   size='dependent_var3',alpha='dependent_var4',shape='factor(dependent_var5)'))
    + geom_point()
    +theme(legend_title=element_text(size=8),
    legend_text=element_text(size=4)
    )

This will give you following graph:


Classification methods using one overlapping feature I'm currently working on an assignment related to house prices in the UK. I have a list of prices that depend on the size of the property (small, medium, large) and the property location in terms of how close the property is to urban areas (Yes = close to Urban areas | No = far away).

The data that I have is from two extremes, so all the properties are either close to urban areas or far away. As you can assume, the prices are high for large properties that are close to urban areas. In contrast, the prices are low for small properties that are far from urban areas.

However, the data is quite overlapping and a small house in urban area might have the same price (or close) to a large property outside the urban zone.

Now, my task is to determine whether the property is located in urban areas just by using house prices (the only available feature). But to do so I need to determine the size of the property as this has a significant affect on the price as well.

In reality, I would like to test my algorithm(s) on a range of property prices even those that are somewhere in between that is, not far to urban areas but no so close either. Can someone share their ideas on how one would tackle such a problem using only the house prices as my features? Thanks in advance for your input!

Could you plot something like heatmap or 3D plot (x=size,y=location,z=prices) ? I would try something like a simple decision tree to fit your data.

Using only the houses prices as feature, I would first plot prices versus location. If there are two distinct clusters slightly overlaping, maybe fixing just one threshold is good enough.


Issue with Spark SVD I have the following dataset with the dimensions:

Rows: 41174

Columns: 439316

The matrix is very sparse and on this, I want to perform Dimensionality Reduction. I am using Spark's computeSVD function to perform the dimensionality reduction.

However, I get an error saying that

  Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: k = 41174 and/or n = 439314 are too large to compute an eigendecomposition

But I ran the same computeSVD on the following dataset and it ran perfectly fine.

Rows: 3502

Columns: 103301

In both the cases, I am passing the value of "k" to be the Minimum of Rows, Columns. I am not able to understand what I am doing wrong here. As per the error, the issue is with K. How to resolve the above error. Also, any ideas on how to determine the K?

After reading the source code, I find out that the problem is nothing related to your memory, but the limit of integer.

In the source code, it shows both of $n\min(2k,n)$ and $\min(2k,n)*(\min(2k,n)+8)$ should be less than Integer.MAX_VALUE, which is $2^{31}-1=2147483647$. In your case, $n\min(2k,n)=36176793968>2147483647$, and $\min(2k,n)*(\min(2k,n)+8)=6781851888>2147483647$.

I'm no spark expert nor java export so maybe I can't give you a solution. But personally speaking your dataset is truly big!

References

https://docs.oracle.com/javase/7/docs/api/java/lang/Integer.html


Do we need to add the sigmoid derivative term in the final layer's error value? I have been studying professor Andrew Ng's Machine Learning course on Coursera. Currently, I am trying to prove the formulas for backpropagation, which is mentioned in Week 5 (in this document). Clearly, it mentions that
δ(L) = a(L) − y.

However, I have also came across this video on Youtube. In 4:30, it states that δ(3) = -(y - y^) * f'(z(3)), where f is the sigmoid function. Since L = 3 and y^ = a(L), this means that δ(L) = (a(L) - y) * f'(z(L)). I've also derived the equations and got exactly like in the video. But this has an additional "f'(z(L))" term, which is not the same as in Coursera. I have even implemented this equation in the programming assignment for Week 5, but it only works without the term.

So my question is why Coursera's formula didn't have the last term "f'(z(L))" and why the formula works without it? It seems quite irrational to not have the term in the formula. Can anyone help me to explain this, or correct me if I'm wrong? Any help would be greatly appreciated, thanks.

Both courses are correct.

For the output layer only, and depending on which activation function and which loss function are in use, sometimes the effects of $f'(z^{(L)})$ are perfectly cancelled out by the calculation of $\frac{\partial J}{\partial \hat{y}}$

This is often arranged deliberately to simplify calculations. So:

MSE loss $(\hat{y} - y)^2$ is paired to linear activation function $f(x) = x$
Binary cross entropy loss $-y\text{log}(\hat{y}) - (1-y)\text{log}(1-\hat{y})$ is paired to sigmoid activation function $f(x) = \frac{1}{1+e^{-x}}$
Multi-class cross entropy loss $-\mathbf{y}\text{log}(\hat{\mathbf{y}})$ is paired to softmax activation function $f(x_i) = \frac{e^{x_i}}{\sum_j e^{x_j}}$

In all these cases, the end result is that, for the output layer logits only:

$$\frac{\partial J}{\partial z^{(L)}} = \hat{y} - y$$

Each of these results can be verified with some differentiation. E.g. it is possible to show that $\frac{d}{dx}(-y\text{log}(\hat{y}) - (1-y)\text{log}(1-\hat{y})) = \frac{1}{\hat{y}(1-\hat{y})}$

For all the other layers, you absolutely need to include the factor of $f'(z^{(l)})$ for whatever activation function is in use on the layer you just back propagated through.


Error with Jupyter Notebook and Google Cloud so I integrated my google cloud with the Jupyter Notebook and it seemed to be working as follows:

[I 12:13:51.633 NotebookApp] Serving notebooks from local directory: /home/kerenb1

[I 12:13:51.633 NotebookApp] The Jupyter Notebook is running at:

[I 12:13:51.633 NotebookApp] http://localhost:5000/?
token=63dedf4645de55a797dff864c38721f01740048d93557092

[I 12:13:51.633 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

[C 12:13:51.636 NotebookApp]

To access the notebook, open this file in a browser:
    file:///run/user/1001/jupyter/nbserver-3124-open.html
Or copy and paste one of these URLs:
    http://localhost:5000/?token=63dedf4645de55a797dff864c38721f01740048d93557092

But when I did that, it told me that I had to activate my CloudShell and do Web Preview with the port that was given, but I wanted it on my Jupyter Notebook port so I did that and it said:

Error: Could not connect to Cloud Shell on port 5000.
Ensure your server is listening on port 5000 and try again.

Anybody know how to solve this issue?

Same issue here. Oddly enough I had another instance regirested before in the same way and it all worked. Unsure what is going on, but suspecting the VM is rejecting to connect for some reason.


Is it acceptable to use label encoding for nominal categorical data when one hot encoding would create too many features? I'm working on a short data science project to compare the accuracy of different classification methods. The groups decided to use and compare Random Forest, Naive Bayes and SVM.
The dataset we are using has four categorical features. Each of which has a large number of unique values.

There are 16537 unique combinations of 17370 unique values in
FeaureA.
There are 13860 unique combinations of 13852 unique  values
in FeaureB.
There are 3295 unique combinations of 29 unique  values
in FeaureC.
There are 1518 unique combinations of 29 unique  values
in FeaureD.

From what I've read the RF and NB algorithms should work fine with label encoding but SVM requires one hot encoding. However that would increase the number of features by ~35K. The performance cost seems like it would be significant. Ideally we would use the same encoding for all three algorithms. Would it be better to take the performance hit and try something like PCA for feature reduction?

Tree Based models like RF and XGBoost can handle the feature space created by either of the methods you suggested above. Additionally, you can try to do feature selection (use those statistical tests) to eliminate 0 variance features and then feed the filtered dataset to your models.


Data prediction using scikit-learn and a list I have a group of lists detailing temperatures over differing amounts of time. My goal is to use machine learning to identify periods in which a machine is turned on and off, where turning on the machine drastically increases temperature, and turning off the machine returns it to idle temperature.

The rub is that there is a level of temperature inertia- take the following example.

Among my control data is the following list (truncated for simplicity's sake):

At minute intervals, the temperature starts at 73 degrees. It is turned off after 12 minutes, when the temperature is 83 degrees. However, the peak is reached 3 minutes after the stopping point, at 86 degrees.

Given a control group of a list with labelled starts, ends, and peaks, how would I go about using supervised learning to create an algorithm that could predict stops using a list with only starts and peaks?

There are two popular approaches:

Model it with traditional supervised binary classification algorithms (e.g., logistic regression or Random Forest) and generate features (e.g., look back 1 steps, look back 2 steps, …)

Model it with a Hidden Markov Model (HMM). The hidden state is the machine on or off. The observed state is temperature. Given the observed state, what is the most likely hidden state that would generate it?


Word2Vec benefit in LSTM if Word2Vec is nothing but a transformation of one-hot into a dense vector, why can't I just feed one-hot into LSTM (or for that matter sacrifice first dense layer, in any network that will end up using the embedding) and call it a day?

Why would I actually spend time pre-computing Word2Vec embeddings?
Yes, the resulting embedings are vectors that are clustered together if the words have similar meaning. But I presume a feed-forward classifier would figure out a good internal model anyway?

You don't need word embeddings. Actually, in neural machine translation is frequent not to use them and simply train the embeddings along with the task.

Nevertheless, word embeddings work as a data augmentation technique, as you normally use a different (and much larger) dataset to train them, so they can be useful when you don't have much training data.

Therefore, the decision of using pre-trained word embeddings or not should be driven by the available data.


Encog neural network multiple outputs I am a little confused with using encog to create a neural network.

I am trying text classification with a basic feed forward network. For the input data I have 200 unique words (features/inputs) to input into the network and 100 different pieces of text. so I have a matrix of 100x230, essentially 100 different training items each with the frequency of 230 words. I then have an output which classifies the data as A, B, C, D or E, so one column and 5 different outputs, therefore this is an ordinal column and is part of the CSV data set I read in thus I actually have a matrix of 100X231 where the 231 column is the output I desire to classify.

I can train and run this neural network no problem but I am confused about the number of output neurons to have. I would have thought there are 5 different classifications therefore I should have 5 different neurons, however I can't setup the code like that, as it complains:

IMLDataSet trainingSet = new BasicMLDataSet(input, ideal);

My input variable is a double[100][230] and the ideal is just the column of expected classifications so double[100][1]. Because I only have one column and the data is normalised those A,B,C... values are turned into numbers between -1 and 1 and therefore when I run the neural net with 5 output neruons it complains that it should be only 1.

When I run the neural net with one neuron output it gives me what looks like correct and accurate answers but the value it gives is between -1 and 1. I am assuming the activation level is being matched to the output?

My question therefore is either how to denormalise this output to get back to my actual letter classification or how to use 5 neurons on the output so the network chooses the appropriate classification.

My neural network is as follows:

 BasicNetwork network = new BasicNetwork();
 network.AddLayer(new BasicLayer(null, true, 230));
 network.AddLayer(new BasicLayer(new ActivationTANH(), true, 16));
 network.AddLayer(new BasicLayer(new ActivationTANH(), false, 1));
 network.Structure.FinalizeStructure();
 network.Reset();

Needed to normalise the data which encodes the column and then denormalise on the output which gives the correct classification.


Create continous plot using matplotlib/seaborn with specific hue df.head()=

day SBP     exercise
0   146.0   False
1   149.0   False
2   146.0   True
3   148.0   False
4   149.0   False
5   149.0   False

i am looking to plot day vs sbp with each data point coloured as per exercise.
i used following command but couldn't get each point colored as per exercise:
plt.plot('user','SBP', color='green', linestyle='dashed', marker='o',data=df)

can anyone help how i should go about this..?

You mean something like this plot?

The code for doing it is:

import matplotlib.pyplot as plt

day=[0,1,2,3,4,5]
SBP=[146.0, 149.0, 146.0, 148.0, 149.0, 149.0]
exercise=[False, False, True, False, False, False]

plt.scatter(day, SBP, c=exercise, s=50)
plt.xlabel("Day")
plt.ylabel("SBP")
plt.show()

The color of each point is already specified in the array "exercise", so you have only to let matplotlib point to it.


Is data augmentation changing the train/test sets distribution? I am thinking a simple question which just came to my mind.
There are many people using data augmentation on their image data to train deep CNN.
When I learn from Andrew Ng's DL courses, he mentioned that to train a better model, you generally need to have your train/test data come from the same distribution. Like, if you train you CNN with a lot of images of cars and human, it's (usually) not a good idea to use it to classify a cat or a dog.
After the augmentation, my training data is kind of boosted, then can I still say my train/test data set comes from the same distribution? Am I violating what he said? Why? Or, why not?

I would argue that augmentation does not alter your train/test sets distribution.

I say this because I consider data augmentation to be part of your training pipeline and not part of your train set.

The pipeline below is one that I use often when someone asks about data augmentation. At the moment the data is split into train/test(/validation) sets, each set should have the same, or at least very similar distributions. This is the moment you need to worry about by picking a good splitting strategy.

Then, whatever you do with the train set, including data augmentation is part of your training pipeline. You will do data augmentation, like mentioned by others, do improve you model robustness. The validation and test set, do not need to be passed through your augmentation process.

          +-> training set ---> data augmentation --+
          |                                         |
          |                                         +-> model training --+
          |                                         |                    |
all data -+-> validation set -----------------------+                    |
          |                                                              +-> model testing
          |                                                              |
          |                                                              |
          +-> test set --------------------------------------------------+


How to methodologically show that a given 'time-series/sequential' data is not really sequential? I have an apparent time-series/sequencial (supervised: multi-class classification problem) dataset with each data-point time-stamped. However, some domain intuition tells me that the data is static and no need to treat them as time-series data. I have also verified that a non-timeseries method fits (even with rigorous cross-validation and other good practices) the data well. However, I want to prove this in more rigorous/methodological way, preferably in python but R is also fine if no alternative. Is there any method/test that validate my hypothesis?

Thanks.

TL;DR:
There is no off-the-shelf formula to determine if you should go for time-series or not. Hypothesis testing is the most formal approach, AFAIK.

Just to find a common ground, let's first distinguish between using timestamp vs. dealing with time-series.

Timestamp data itself may contain lots of information that could be quite useful in many domains. For instance, hour, day of week and month are widely used in
demand forecasting applications. However, using the timestamp in this fashion does not make it a time-series approach. Here, time plays only a role of just another feature.

When dealing with time-series, one would expect to benefit from temporal dependencies present in the data (as well as from spatial dependencies). BTW, most of the methods are limited to evenly spaced time-series. If you have timestamps of some arbitrary events, you would have to do pre-precessing/aggregation before being able to treat is as a time-series.

Should you treat your data as a time-series?

I would suggest considering two basic hypotheses (i.e., try proving the opposite if you believe your non-time-series model is just right):

(1) there is a time-domain relationship between observations;
(2) it is indeed relevant to your problem.

First, you may try to investigate (1) by checking if there is a time-series model that explains well your observations. As suggested in comments, autocorrelation test is a way to get a clue if there is a linear relationship between observations through time.

You may wish to go beyond linearity, and try to build a non-linear predictor by using a time delay (TDNN) or a recurrent (RNN) neural model.

Before deep diving in pure statistics for a formal answer, I would suggest a simple test: train your best predictor model on a real time-series and also do it on randomly shuffled (broken) time-series. If there is no significant difference in performance - there probably nothing in time-domain but noise or some sort of complex chaos.

For instance, a time-series of the chaotic process like x(t+1) = 4 * (1-x(t))*x(t) looks quite meaningless, but a very simple non-linear model will be able to predict it perfectly. On the other hand, good luck trying to predict an randn() generator, which is also a chaotic time-series :-).

If you cannot reject (1), i.e. there are strong evidence of time-domain correlations, then you should verify if (2) is true. Here it could be as easy/hard as it gets, depending on your domain. Speaking more formal, I'd go for trying to find out if the estimates of

P[Y(t)| X(t), X(t-1)..., X(t-n)] # predictions of the target variable given past observations

are as good as

P[Y(t)| X(t)].  # predictions of the target variable given only current observation

You could try to answer it empirically by comparing the performance of a time-series vs. non-time-series models in a sufficient number of trials, or by using some specific knowledge of your domain.

E.g., P[cat | pixels of current picture] = P[cat | pixels of the current picture, pixels of the previous picture], which may be true or not, depending whether the pictures were taken in the same or different households.


Reduce serving time complexity for real-time recommender systems I am working on a real-time recommender system predicting a product to a user using deep learning techniques (like wide & deep learning, deep & cross-network etc). Product catalogue can be huge (1000s to 1 million) and for a given user, the model needs to be evaluated against each product in real-time. As scalability is an important concern, is there any way to reduce the serving time complexity by tuning model architecture?

You write ... the model needs to be evaluated against each product in real-time., which gets me thinking that you use a binary classification (sigmoid in the final layer) architecture with negative sampling for the user/item interactions when training your model.
Have you considered using multi-class classification instead? Thus, for the user only predict once for the entire product catalogue, and selecting the top-k candidates from the softmax-layer. This way, you only need to feed-forward once through your neural net during inference.


Weird overfitting in linear regression This is really weird, I have a real simple test dataset and built a really simple linear model on it:

smallData = data.head(1000)
# smallData = data

y = smallData['points'].values
X = smallData.drop(['points','country', 'description'], axis=1)

X = pd.get_dummies(X, columns=['designation', 'province', 'region_1', 'region_2', 'variety', 'winery', 'taster_name', 'taster_twitter_handle', 'title'])

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

regressor = LinearRegression()
regressor.fit(X_train, y_train)

y_pred = regressor.predict(X_test)

It actually works really fine, as long as I only train it with ~200 rows of data. Once I have 300 rows and more, it returns COMPLETELY wrong predictions for some rows. Like, my scale is 80 to 100, and the prediction says it should be five million. That seems slightly off, especially there is no value of five million in the training data, 100 is the highest.

My data has quite a lot of columns due to the dummy encoding. It seems like it's overfitting?

What can I do to only get reasonable predictions between 80 and 100?

This is called (perfect) collinearity and happens e.g. when the reference category is included in the set of dummy variables. Use pd.get_dummies(..., drop_first=True) to avoid this.


ADAM algorithm for multilayer neural network I’m trying to touch neural networks without using “in box” algorithms. And so I found out that nowhere is written how to calculate square of gradient for hidden layers in ADAM optimizer.
I took the description from original article and the problem is that Vt is some kind of normalizing coefficient, so it should be scalar. But for hidden layers gradient is not really a vector byt a matrix. So how should we calculate gt^2 in this case?

It is written right in the algorithm description that $g_t^2$ is elementwise square.


What is the best file format to store an uncompressed 2D matrix? For what it's worth my particular case is a symmetrical matrix, but this question should be answered more generally.

I would go with .csv as it's universally accepted and can be read in different programming languages easily. Moreover you can simply open it with an office software. If you are using your matrix only in Python I also recommend Pickle library which writes your matrix in a .p format and can be easily read in Python with a simple load function.


Doc2vec(gensim) - How can I infer unseen sentences’ label? https://radimrehurek.com/gensim/models/doc2vec.html

For example, if we have trained doc2vec with

"aaaaaAAAAAaaaaaa" - "label 1"

“bbbbbbBBBBBbbbb" - "label 2"

can we infer “aaaaAAAAaaaaAA” is label 1 using Doc2vec?

I know Doc2vec can train word vectors and label vectors.
Using this vectors, can we infer unseen sentences(combination of trained words) in which label?

The title of this question is a separate question to its text so I will answer both separately (given that one leads into the other).

How can I infer unseen sentences:

# ... trained model stored in var model
list_of_words = ["this", "is", "a", "new","unseen", "sentence"]
inferred_embedding = model.infer_vector(list_of_words)

How does this work ? As per the original paper (https://cs.stanford.edu/~quocle/paragraph_vector.pdf) we have two weight matrices initialized at random $W \in \mathbb{R}^{N\times P}$ which is the same matrix from Word2Vec in which every column or row maps to a word vector and $D \in \mathbb{R}^{M \times R}$ which is the document matrix where each column or row maps to a sentence/document. During training a softmax classifier of fixed window $k$ size moves (in a moving window like fashion) minimizing the following log likelihood (multi-class cross-entropy):

$$
\frac{1}{M}\sum_{i=1}^{M}\frac{1}{|D_{i}|}\sum_{t=k}^{|D_{i-1}|-k}log(p(w_{t}^{i} | w_{t-k}^{i}, ..., w_{t+k}^{i},D_{i} ))
$$

Where $D_{i}$ corresponds to the vector representing the $i^{th}$ sentence, $|D_{i}|$ its number of words in that document and $w_{t}^{i}$ is the $t^{th}$ word in the $i^{th}$ document. All back-propagation remembers is the document we are currently moving our windowed softmax over and only updates that row in matrix $D$ along with the words in that window.

Furthermore when we want to infer something not in the training set we fix $W$ so that it is not updated and augment matrix $D$ with the new randomly initialised row and just train for several iterations (with the new row holding the embedding for the inferred vector). This leads into question 2.

Can we infer that a possibly unseen sentence exactly corresponds to a sentence in the training set ?

The short answer is no and this is not what Doc2Vec is for. Because of the random initialization + the complexity of convergence and training your inferred vector will never be exactly the same as its corresponding vector in $D$ this is why Gensim has not built in a function to support this, how ever given that the model has been well trained these two vectors should be arbitrarily close to each other  thus you can conclude that they are extremely similar.

Even fixing the random seed may not work, there are so many other variables that can affect its convergence please see first answer on https://github.com/RaRe-Technologies/gensim/issues/374 .

In any case you can find the most similar label in your data set to an inferred sentence just by iterating over your training set and comparing the similarity to the inferred vector. But why would you want to do an exact match to something in the training set ? that's what regular expressions are for the purpose of these document embedding's is for supervised or unsupervised learning tasks (i.e. classification and clustering).


k-means clustering data with large number of meaningless values I am looking to perform k-means on my dataset which contains a large number of 0 values.
The last value you see is different to the others, that is simply the sum of transactions, not related to the categorical frequency count.
Example:
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 189200.579626]
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.06556796]
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 5.46e-06]
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 5.46e-06]

Each feature is a frequency count of how many times the associated id is seen within a category.
When I run k-means, I see that most of the data is clustered in one cluster.
+------------+--------+----------------------+
| cluster_id |  size  | sum_squared_distance |
+------------+--------+----------------------+
|     0      | 659187 |    0.999997057952    |
|     1      |   3    |    1.33333326876     |
|     2      |   1    |         0.0          |
|     3      |   3    |    0.666666716337    |
|     4      |   1    |         0.0          |
|     5      |   1    |         0.0          |
|     6      |   1    |         0.0          |
|     7      |   1    |         0.0          |
|     8      |   1    |         0.0          |
|     9      |   11   |    2.72727286816     |
+------------+--------+----------------------+

I am assuming that this is because the majority of the dataset has not been seen in a feature category and therefore has a value of 0.
What is the best way to overcome this, drop rows where a 0 is seen across each category? Are these rows meaningless to clustering?

One thing you could do is apply some dimensional reduction algorithm (such as PCA) so you can get the columns with high variance, then run k-means on that data set.

However, I suggest against using k-means in sparse matrices like yours. Anony-Mousse's answer here explains it well.