Why can't we feed datetime to Linear Regression and how does toordinal() different from any other integer datatype? I'm working with dates for the first time. First I knew I had to convert it to timestamps which gave me the values in "datetime64" values. But then I came to know that Linear Regression from sklearn does not accept datetime as dtype for regression. Why is that?
Also, after executing df.date = df.date.map(datetime.toordinal) I saw that the datatype for the column is "int64". How does the sklearn LinearRegression know it is not any random int64 dataand a datetime data (considering the fact that I got the needed model after plotting)?

Linear Regression is associating any numerical (or binary, which is a particular numerical) value to a coefficient. Multiplying those values by those coefficients gives you an output, and setting the threshold, you know if the model predicts 1 or 0. (This is a brief summary, you'll find plenty of people explaining in details how it works).
If your variable is a date, you have a format, like "Year/Month/Day", and the regression doesn't know how to interpret it, since it need numerical data.
So as to use the date in a regression, you can basically take the year, as a variable.
If you want to be more precise, you'll have to create your own variable, with the meaning you want (Ex : a binary variable taking 1 if the day is a Saturday or Sunday, 0 either).
Side Note : Watch out when you use dates variables, if you want to use your model to predict new, future data. For ex, if you have a variable Year, and you test your model in a Cross-Validation, it will have an impact since the trend can change with years. Moreover, if you want to predict new values input, Year will always be '2020', so this won't be that useful. Utilizing it to evaluate your model might biaise your results.


Can a recommendation system be built without any user ratings? I was planning to make an artwork recommendation system as a project by using the WikiArt open source dataset available on kaggle, I'm still looking for datasets which might already have user ratings but is it possible to make a recommender without any user ratings? The dataset currently has these features. Any help or insight is appreciated. Thanks.

This is a common issue for recommender system refered to as cold start. You can look it up there are some solutions suggested out there but I have never tested them out. Have a look at this paper:

http://www.cs.toronto.edu/~mvolkovs/nips2017_deepcf.pdf


When does Adam update its weights? I have a dataset with at least 70% of labels incorrect.
I'd expect that incorrect labels would compensate each other while true labels will be taught properly (given a very large dataset).
For example, if I have 300 samples saying a => -1 and 300 samples saying a => 1, the result for the input "a" eventually will be 0 (for a regression problem).
If I use Adam for the example above, won't it affect the results for the inputs with noisy labels due to its adaptive nature? Won't it be better to use SGD instead and decay the learning rate, or does Adam change its weights only at the end of every epoch?

Adam works in the same way as SGD does in this regard, it updates the weights at the end of each iteration, so at the end of an epoch multiple weight updates have been applied.
Inherently neither Adam nor SGD do anything to counteract the noisy labels, they just try to find the best parameters that minimize a loss function. I don't think anyone can answer apriori if it will be better to use Adam or SGD for your problem.


Distance metric for graph layout of correlation matrix? I have a handful of N "things" for which I have a NxN symmetric correlation matrix.  The correlations are a based on a custom scheme for quantifying similarity between complex things, with a value range 0% to 100%, though the values for the N things are all high (95.4%-97.4%). I want to visualize the similarities and differences using force directed graph visualization [1].
I was going to use 1 minus the correlations as a distance metric.  Is this an appropriate for correlations, particularly when values are all high?
Googling hasn't been my friend when searching for the intersection between correlation matrices and graph visualization.  My thought is that 1 minus the correlation plots the N things with distances somewhat proportional to their very small differences.  An alternative might be to use the reciprocal of the correlation as the distance, but since the values are close to one, the Taylor expansion would show such a distance would be similar to 1 minus the correlation.  The two schemes would differ much more for near-zero correlations, where the reciprocal would yield a distance approaching infinity (not practical for graph visualization).  I can tweak the 1 minus distance scheme by raising to various powers, greater or less than 1 depending on whether I want to emphasize the small distances or the great distances.
[1] I would be using Matlab's layout tool, which uses Fruchterman & Reingold's "Graph Drawing by Force-directed Placement." Software — Practice & Experience. Vol. 21 (11), 1991, pp.  1129–1164.

What I have found is that it is not worth sweating the nuances of how distances are represented because they only partially determine the layout.  Other factors are geared to generating a pleasing layout so that analysts can recognize gross patterns based on the absence presence of interconnections rather than purely based on distance.


Normal distribution instead of Logistic distribution for classification Logistic regression, based on the logistic function $\sigma(x) =
\frac{1}{1 + \exp(-x)}$, can be seen as a hypothesis testing problem. Where the reference distribution is the standard Logistic distribution where the p.m.f is

  $f(x) = \frac{\exp(-x)}{[1 + \exp(-x)]^2}$

and the c.d.f is

  $F(x) = \sigma(x) = \frac{1}{1 + \exp(-x)}$

The hypothesis to test is

  $H_0: x \text{ isn't positive} \hspace{2.0cm} H_1: x \text{ is positive}$

The test statistic is $F(x)$. We reject $H_0$ if $F(x) \geq \alpha$ where $\alpha$ is the level of significance (in terms of hypothesis testing) or classification threshold (in terms of classification problem)

My question is that why they don't come up with the Standard normal distribution, which truly reflects the "distribution of nature", instead of Logistic distribution ?

Nice comparison.

Generally, we are allowed to experiment with as many distributions as we want, and find the one that suits our purpose. However, the normality assumption leads to an intractable derivation consisting of the notorious erf function.

Let's first pinpoint what is $x$ in the context of logistic regression. Logistic regression model can be written as:
$$P(y=1|\boldsymbol{x})=\frac{1}{1+e^{-\boldsymbol{w}^t\boldsymbol{x}}}=F(\boldsymbol{w}^t\boldsymbol{x})$$
So your $x$ is actually $z=\boldsymbol{w}^t\boldsymbol{x}$. This means, although it is reasonable to assume that predicate $\boldsymbol{x}$ comes from a normal distribution, the same argument does not hold for a linear combination of its dimensions, i.e. $z$. In other words, the normal assumption is not as natural for $z$ as for $\boldsymbol{x}$.

But still, let's see what happens with normal assumption. The problem that we face here is analytical intractability. More specifically, to fit a similar model to observations using Maximum Likelihood, we need (1) derivative of cumulative distribution function (CDF) with respect to each parameter $w_i$, and (2) value  of CDF for a given $z$ (see this lecture section 12.2.1 for more details).

For logistic distribution, the required gradient would be:
$$\begin{align*}
\frac{\partial F(\boldsymbol{x};\boldsymbol{w})}{\partial w_i}&=\frac{\partial (1+e^{-\boldsymbol{w}^t\boldsymbol{x}})^{-1}}{\partial w_i}= x_i e^{-\boldsymbol{w}^t\boldsymbol{x}}(1+e^{-\boldsymbol{w}^t\boldsymbol{x}})^{-2} =x_if(\boldsymbol{x};\boldsymbol{w})
\end{align*}$$

However for normal distribution, CDF is the erf function which does not have an exact formula, though, its gradient is tractable. Assuming $z \sim \mathcal{N}(0, 1)$, the gradient would be:
$$\begin{align*}
\frac{\partial F(\boldsymbol{x};\boldsymbol{w})}{\partial w_i}&=\frac{\partial \left(\frac{1}{2}+\frac{1}{2}\text{erf}\left(\frac{z}{\sqrt{2}}\right)\right)}{\partial w_i}=\frac{x_i}{\sqrt{2 \pi}} e^{-\frac{(\boldsymbol{w}^t\boldsymbol{x})^2}{2}}=x_if(\boldsymbol{x};\boldsymbol{w})
\end{align*}$$

In summary, the normality assumption is not as justified for $z=\boldsymbol{w}^t\boldsymbol{x}$ as for $\boldsymbol{x}$, and it leads to an intractable CDF. Therefore, we continue using the good old logistic regression!

Here is a visual comparison of normal and logistic CDFs:

taken from a post by Enrique Pinzon, which implies a large analytical cost for a small difference!


Interpreting Decision Tree in context of feature importances I'm trying to understand how to fully understand the decision process of a decision tree classification model built with sklearn. The 2 main aspect I'm looking at are a graphviz representation of the tree and the list of feature importances. What I don't understand is how the feature importance is determined in the context of the tree. For example, here is my list of feature importances:

Feature ranking:
1. FeatureA (0.300237)

FeatureB (0.166800)
FeatureC (0.092472)
FeatureD (0.075009)
FeatureE (0.068310)
FeatureF (0.067118)
FeatureG (0.066510)
FeatureH (0.043502)
FeatureI (0.040281)
FeatureJ (0.039006)
FeatureK (0.032618)
FeatureL (0.008136)
FeatureM (0.000000)

However, when I look at the top of the tree, it looks like this:

In fact, some of the features that are ranked "most important" don't appear until much further down the tree, and the top of the tree is FeatureJ which is one of the lowest ranked features. My naive assumption would be that the most important features would be ranked near the top of the tree to have the greatest impact. If that's incorrect, then what is it that makes a feature "important"?

It is not necessary that the more important a feature is then the higher its node is at the decision tree.

This is simply because different criteria (e.g. Gini Impurity, Entropy-Information Gain, MSE etc) may be used at each of two these cases (splitting vs importance).

For example, at SkLearn you may choose to do the splitting of the nodes at the decision tree  according to the Entropy-Information Gain criterion (see criterion & 'entropy' at SkLearn) while the importance of the features is given by Gini Importance which is the mean decrease of the Gini Impurity for a given variable across all the trees of the random forest (see feature_importances_ at SkLearn and here).

If I am right, at SkLearn the same applies even if you choose to do the splitting of the nodes at the decision tree according to the Gini Impurity criterion while the importance of the features is given by Gini Importance because Gini Impurity and Gini Importance are not identical (see also this and this on Stackoverflow about Gini Importance).


Object detection is not improving although loss is decreasing I am working on a project for detecting buildings from satellite images (using OpenStreetMap data as labels) using the Tensorflow Object Detection API. I recently upgraded to Tensorflow 2 and chose EfficientDet D4 from the Model ZOO.
I train (fine-tune) this pre-trained model on my data. Throughout the training, I keep making predictions using always the current state of the model. The point is to see how the performance of the model gets better with training.
Problem: The ability to predict buildings in validation dataset is not improving at all. In fact, it is getting worse! However, the loss during training is constantly decreasing.
Here is an example - in blue are predictions of the model after less than 1 epoch while in red are predictions by a model trained much more:

To me, it doesn't make any sense. I've spent loads of time trying different variations of this, using another dataset for evaluation, etc. But still I get worse results (in terms of precision, recall, f1) when I train the model more. And this is not due to overfitting, the model's performance starts to decrease almost immediately.
Earlier, I was using Tensorflow 1 and SSD Inception V2 as a model. That worked as expected, i.e. the model's ability to detect buildings was improving as the training progressed.
Question: What could be the cause of this and what can I try to solve this problem?I have run out of ideas.
I am new to machine learning so I am likely missing some important information here. Please let me know if more details are needed to answer this.

So the question asks why you are seeing a decrease in the loss function (for both training and validation?), but you are also observing decreasing generalisation performance over iterations.
One first thought could be due to the loss function that you have chosen might not be appropriate for your task.


Tensorflow Fine-tuning a model from my own checkpoint Assume that I am going to do more training with a similar data set in the future, is there any benefit to me using a fine tune checkpoint from a model that I created from my own training as opposed to the original SSD_Mobilenet_V1 version (for example 5000 images and 50000 steps).  Does it improve any future training or am I just better off using the original one every time I train?  I'm probably searching for the wrong thing, but I cannot find anything that suggests improvement.  In my reading, I thought that the fine tune only helped with the final layer which leads me to believe its a pointless exercise no matter how good the resulting model was.  Can anyone confirm this point and any useful reading for reference?

Yes, you can. After the below lines everything else will be same as what you have done previously to save the checkpoint.

with tf.Session() as sess:
  saver = tf.train.Saver()
  saver.restore(sess, tf.train.latest_checkpoint('./')) # checkpoint file path

Please refer to the below links:

https://nathanbrixius.wordpress.com/2016/05/24/checkpointing-and-reusing-tensorflow-models/

https://github.com/tensorflow/nmt/issues/51

https://blog.metaflow.fr/tensorflow-saving-restoring-and-mixing-multiple-models-c4c94d5d7125


Interpreting the C-Index I have some problems understanding/interpreting the C-Index cluster quality measure. So, if we have

$c(x_i, x_j) = 1 $ if $ x_i, x_j $ in the same cluster; $0$ else

$\Gamma = \sum_ {i=1}^{n-1}\sum_ {j=i+1}^n d(x_i,x_j)*c(x_i,x_j)$

$\alpha = \sum_ {i=1}^{n-1}\sum_ {j=i+1}^n c(x_i,x_j)$

$min=$ sum of the $\alpha$ smallest $d(x_i,x_j)$ of distinct pairs $x_i,x_j$ where $x_i \neq x_j$

$max=$ sum of the $\alpha$ largest $d(x_i,x_j)$ of distinct pairs $x_i,x_j$ where $x_i \neq x_j$

then the C-Index is defined as $C=\frac{\Gamma - min}{max - min}$

The result is a value in $[0, 1]$, where lower values indicate a better cluster quality.

So, here are some things I get from this value:

if all elements in a cluster are close together and all clusters are far apart, we can get  $\Gamma=min$, which means $C=0$
Analogously, in a worst case scenario, all the observations that are furthers apart might be in the same cluster, so we would get $\Gamma=max$, which means $C=1$

Now, these are the things I'm unsure about:

First: If we only have a single cluster in our clustering (e.g. k-Means for $k=1$), then $\alpha$ is equal to the number of distinct pairs of observations, so $max=min$, which means $C=\frac{\Gamma - min}{max - min} = \frac{\Gamma - min}{min - min} = \frac{\Gamma - min}{0}$ and we get a division by 0. A similar problem occurs if we have $N$ observations in $N$ different clusters, since $c$ is always $0$ in that case. So, is it correct to say that the C-Index can only be used for clusterings with $k$ many clusters where $1 < k < N$, for $N$ observations?

Second: Is it reasonable to say that the C-Index is agnostic to the number of clusters (e.g. the value of $k$ in k-Means)? For instance, we might have 5 observations $x_1...x_5$ close to each other, but each put into a separate cluster $C_1...C_5$. Then, we might have a clusters $C_6=\{x_6, x_7\}$ where $x_6, x_7$ are very close to each other but far apart from all other observations. In that case, $\Gamma=d(x_6,x_7)$, $\alpha=1$, $\min=d(x_6,x_7)$, so $\Gamma=min$, which means $C=0$. That is, we have the best possible C-Index-Value, even though, intuitively, it might have been better to put $x_1...x_5$ into a single cluster.

Lastly, this is more about k-Means: if we use normal k-Means (not global k-Means), are we always guaranteed to reach $C=0$, for an unbounded number of iterations? I can't seem to find an example that wouldn't result in this.

For the first Q you already give a counterexample:

It is biased to k, it prefers k = N. And it will also overrate N-1, N-2, ... So it is not agnostic to k.

If k-means would always find the best C index, then the C index would just be redundant to SSQ, which is much cheaper to compute... But you probably have just been looking at way too simple toy datasets. Use real data.


Is Overfitting a problem in Unsupervised learning? I come to this question as I read the use of PCA to reduce overfitting is a bad practice. That is because PCA does not consider labels/output classes and so Regularization is always preferred.

That seems purely valid in Supervised Learning.

What about the case for Unsupervised Learning?
We don't have any labels whatsoever.
So 2 questions.

Is overfitting a problem in Unsupervised learning?
If yes, Can we use PCA to prevent overfitting?Is that a good practice?

Overfitting happens when the model fits the training dataset more than it fits the underlying distribution. In a way, it models the specific sample rather than producing a more general model of the phenomena or underlying process.

It can be presented using Bayesian methods. If I use Naive Bayes then I have a simple model that might not fit either the dataset or the distribution too well but of low complexity.

Now suppose that we use a very large Bayesian network. It might end up not being able to gain more insight and use the complexity to model the dataset (or even just trash).

So, overfitting is possible in unsupervised learning.

In PCA we start with a model in the size of the dataset. We have assumptions about the way the data behaves and use them to reduce the model size by removing parts which don't explain the main factors of variation. Since we reduce the model size one could expect to benefit always.

However, we face problems.
First, a model in the size of the dataset is extremely large (given such a large size you can model any dataset). Compressing it a bit is not enough.

Another problem is that it is possible that our assumptions are not correct. Then we will have a smaller model be it won't be aligned with the distribution. Again, it this case it might overfit or just not fit the model.

Though that, PCA is aimed to reduce the dimensionality, what lead to a smaller model and possibly reduce the chance of overfitting. So, in case that the distribution fits the PCA assumptions, it should help.

To summarize, overfitting is possible in unsupervised learning too. PCA might help with it, on a suitable data.


Varying Image sizes in Tensorflow Malaria dataset | Dealing with unclean tensorflow data I am trying to build a CNN based image recognition system for the Tensorflow malaria dataset. I loaded the dataset (~27k RGB images) using conventional tensorflow_datasets syntax.
After some data exploration, I found that all the images are not of the same size. A print statement for few instances is shown in below snippet:
import tensorflow_datasets as tfds

ds_train, ds_info = tfds.load('malaria', split='train', as_supervised=True,with_info=True)

ds = ds_train.take(5)  #selecting 5 images

for image, label in tfds.as_numpy(ds):
  print(type(image),image.shape, type(label), label)

OUTPUT
<class 'numpy.ndarray'> (103, 103, 3) <class 'numpy.int64'> 1
<class 'numpy.ndarray'> (106, 121, 3) <class 'numpy.int64'> 1
<class 'numpy.ndarray'> (139, 142, 3) <class 'numpy.int64'> 0
<class 'numpy.ndarray'> (130, 118, 3) <class 'numpy.int64'> 1

The varying sizes of images across the dataset affect the initial CNN layer as flattening each image tensor yields a different sized array.
I understand that all the images need to be converted to a common aspect ratio before the modelling step and we can achieve that by using padding or some other preprocessing techniques of keras.preprocessing.image but I am not sure about the steps to efficiently implement it.
I will be grateful if someone could provide an elegant way around this.
Thank you in advance!

#Here image is your batch.
# Add "batch" and "channels" dimensions
image = image[tf.newaxis, ..., tf.newaxis]
image.shape.as_list()  # [batch, height, width, channels]

tf.image.resize(image, [height,width])[0,...,0].numpy()


How to add incorporate meta data into text classification? I have a collection of statements which I need to classify into 5 classes. Each statement have meta data in different columns:

Author|Editor| date of release| statement | Class

How can one use the meta data to improve the text classification task?

Some models cannot really handle this, while others lend themselves for it easily. I'll explain two approaches that you could use:

Naive bayes

With Naive Bayes you can use other categorical values as well as your normal n-grams or sparse bag of words vectors. Just add them one-hot encoded to your features and it is also incorporated. With numerical features you would need to use something like Gaussian Naive Bayes, to fit a distribution to your features per target class, then you can use the likelihoods of these features per class to compute the probabilities.

Neural network

If you use a neural network approach like CNNs or RNNs, you can add any type of feature representation network and concatenate it somewhere in your original network. In your case you would have a softmax at the end of your RNN. Before this, concatenate the output of your 'normal-feature' neural network, add some dense layers and feed this to your softmax output layer. This way you can train your model end-to-end and it will learn important interactions as well.


Looping 2 DataFrame Pandas I am learning about pandas and find an interesting case
i try to find total sales for every genre that exist in the data frame,I try looping a few time and get a dead end,maybe you can give me a few tips about this
here is my code
import pandas as pd

df = pd.read_csv('vgsales.csv')
dcheck = df.dtypes

#check value count in data frame
genre_values = df['Genre'].value_counts()

#check what value in dataframe
genre2 = df['Genre'].unique()

zero = 0

def multiply():
    while genre2['Racing'].isin(df['Genre']):
        df['Global_Sales'][0] + zero

result = multiply()

A simple group-by sum will solve your query:
df.groupby(['Genre'])['Global_Sales'].sum()


Analysis of Time Series data The below graph is a scatterplot of daily stock price. My aim is to predict future stock price of the company.

From the scatterplot it seems that it is a multiplicative model, so I tried to "decompose" it in R. However it says that "time series has no or less than 2 periods". I also obtained a periodogram, which has only one peak at frequency close to 0.

However, my teacher told me that this time series cannot have a trend therefore to eliminate the seasonality I have to consider its period as 7 and then eliminate it choosing an appropriate model.
Can anyone tell me what could be an appropriate model along with a proper justification? Also is it true that the series cannot have a trend?

Speaking as someone from a finance background, the `usual' model for a stock price process is

$\frac{dS}{S}=r dt + \sigma dW_t$

i.e. we assume the returns (not the absolute price changes, i.e. dS/S is approximately the daily percentage change in price) as having a 'drift' equal to the risk free rate (the interest rate r) and a random shock $dW_t$ with standard deviation $\sigma$

So it's basically the opposite of what you've been asked, I would expect a trend (but it's very hard to estimate with confidence) and I would not expect any seasonality, you could fit an arithmetic model by taking the daily differences though.


What is a Dichotomy? I am currently reading:

  Stephen Jose Hanson: Meiosis Networks, 1990.

and I stumbled about this:

  It is possible to precisely characterize the search problem in terms of the resources or degress of freedom in the learning model. If the task the learning system is to perform is classification then the system can be analyzed in terms of its ability to dichotomize stimulus points in feature space.

  Dichotomization Capability: Network Capacity Using a linear fan-in or hyperplane type neuron we can characterize the degrees of freedom inherent in a network of units with thresholded output. For example, with linear boundaries, consider 4 points, well distributed in a 2-dimensional feature space. There are exactly 14 linearly separable dichotomies that can be formed with the 4 target points. However, there are actually 16 ($2^4$) possible dichotomies of 4 points in 2 dimensions consequently, the number of possible dichotomies or arbitrary categories that are linearly implementable can be thought of as a capacity of the linear network in $k$ dimensions with $n$ examples.

What is a "dichonomy" in this case?

(Side questions: what is a fan-in type neuron?)

In a machine learning context, a dichotomy is simply a split of a set into two mutually exclusive subsets whose union is the original set. The point being made in your quoted text is that for four points, a linear boundary can not form all possible dichotomies (i.e., it does not shatter the set). For example, if the four points are arranged on the corners of a square, a linear boundary can be used to create all possible dichotomies except it cannot produce a boundary that splits the two points lying along one diagonal from the other two points (and vice versa), as you indicated in your own answer.


Multiclass classification on imbalanced dataset : Accuracy or micro F1 or macro F1 I have a multiclass classification problem. Further, an instance can be assigned to exactly one class. My dataset is highly imbalanced. I know that accuracy is not a good metric to use in this case because one can simply predict the high frequent class and get a good score. I understand that micro F1 is better than macro F1 for multiclass classification problem but it turns out that the micro F1 score is the same as accuracy score. So, the whole idea of looking into alternative metric (i.e. micro F1) instead of accuracy has gone full circle.

Should I be using macro F1 instead?

There are two not so widely known in the data science community metrics that work well for imbalanced data and can be used for multi-class data: Cohen's kappa and Matthews Correlation Coefficient (MCC).

Cohen's kappa is a statistic that was designed to measure inter-annotator agreement, but it can be used to measure agreement between the ground truth and a prediction. There are number of explanations online e.g. on Wikipedia or here and it is implemented in scikit-learn.

MMC was initially designed for a binary classification but then generalized for multi-class data. There are also multiple online sources for MCC, e.g.  Wikipedia and here, and it is implemented in scikit-learn.

Hope this helps.


Gradient descent parameter estimation Package for R I am looking for a package that does gradient descent parameter estimation in R, maybe with some bootstrapping to get confidence intervals. I wonder if people call it something different here as I get almost nothing on my searches, and the one article I found was from someone who rolled their own.

It is not that hard to implement, but I would prefer to use something standard.

Ok, after a lot of looking I found the "optim" routine which is in "stats", one of the packages that is always loaded. It has quite a few methods including conjugate gradients and BGGS and a few others and worked well on the first few examples I tried. It doesn't seem to get a lot of attention strangely. I guess optimization people tend to use Matlab.

I knew there had to be something.


How to fix spelling mistakes in data? I have an input data file which contains list of drug names.

I have more than 1000 unique drug names. However, the drug names has spelling mistakes and space character issues.

For ex: we have ISONIAZID 300MG TAB , ISONAZID300MG TAB, ISNIAZID 300MG

You can see how the above 3 terms are different in representation (due to spelling mistake) but actually indicate the same drug ISONIAZID 300MG TAB (which is the right spelling)

But the problem is there are several other drugs with such spelling mistakes and I am not sure how can I group all of them into one (meaning rename it with right spelling)? ex: All the 3 terms above should be renamed to ISONIAZID 300MG TAB (which is the right spelling)

I am posting it here to seek your opinion on is there any medical dictionary or automated approach which can take my raw csv file as input and output proper drug names?

There are several general approaches to this but almost all of them compare the values to some baseline and make a decision whether the individual value is close enough.

You can compare the similarity of strings using different methods e.g. I often use the Levenshtein distance which basically measures how many characters you would have to change to convert word a into word b. By grouping all words with a low enough Levenshtein distance you already identify all words that should be the same.

It is even easier if you have a dictionary of "correct values" in which case you would compare each value to all entries in the dictionary and assign it the value with the lowest levenshtein distance.

Using some basic text analysis you could also identify the most common spelling mistakes in your data and eliminate them based on replacement rules.


Resampling train and test data in R I need to try out few different machine learning methods (SVM, Logistic regression etc.), predict a value either true or false, and write down their AUC and Accuracy of these predictions.
I have allready successfully done that, now i have a two matrixes one for AUC and one for Accuracy, and they are filled with data from SVM and logistic regression (one row).
Now i need to create models for SVM and Logistic regression 10 more times (i should use bootstrapping sampling) and with that i should have 10 rows of my AUC and accuracy data. I have read multiple articles and guides/tutorials, however i can't figure out how to achieve this. I also found and tried few libraries ( one is ROSE and the other one is boot) and none worked for me. Because if i understand the  assigment correctly i need to get 10 different samples from my dataset, and then seperate the data in train and test sets so i can compare the models AUC and accuracy and see how good those models actually are.
Like i said i found multiple sources and the best thing i came up with is this:
 for (i in 1:10){
      set.seed(123)
      ##########################
      ##########################
      boot.sample = sample(n, 1000, replace = TRUE)
      bootSample = dataset[boot.sample, ]
      bootSample

      split = sample.split(bootSample$blueWins, SplitRatio= 0.80)
      training = subset(bootSample, split == TRUE,  replace=TRUE)
      test = subset(bootSample, split == FALSE,  replace=TRUE)
      print(training)
}

But with this approach i think set.seed messes up everything, because it works with the same data every time. However i think the assingment wants me to use the same seed for every machine learning model.
I maybe overcomplicated the whole thing, i am new to R.
Hope someone can clear these things up.
Thanks

Try using a different seed for each loop. You can do it like this.
my_seeds <- c(1:10) # These are 10 seeds, 1, 2, 3...10. Change to whatever.
for (i in 1:10){
          set.seed(my_seeds[i])
          ##########################
          ##########################
          boot.sample = sample(n, 1000, replace = TRUE)
          bootSample = dataset[boot.sample, ]
          bootSample

          split = sample.split(bootSample$blueWins, SplitRatio= 0.80)
          training = subset(bootSample, split == TRUE,  replace=TRUE)
          test = subset(bootSample, split == FALSE,  replace=TRUE)
          print(training)
    }


Determining threshold in an area with very few samples of positive label I have a binary classification task where I want to either keep or discard samples. I have about a million samples, and about 1% should be kept. I want to discard as much as possible, but discarding the wrong sample carries a heavy penalty. I have concluded I want to optimize something like the following:

n_discards - n_false_discards * penalty

Where I expect penalty to be around 5000.

Now, it's easy enough to look at my validation data (around 100k samples) and look at where this value is maximized. However, the threshold that this returns is quite unstable. The high penalty means my threshold should be in an area with very few false discards, but the absence of these makes it very difficult for me to accurately estimate where the threshold should be, exactly. This leads to a lot of threshold variance between folds, and more importantly, between what the validation would suggest and what's right for the test set.

I feel like there must be a better way than going on a sample-by-sample basis, but I haven't found the right way. I've considered trying to fit distributions from scipy.stats and using weighted cdf's to optimize the above function, but I haven't been able to get these fits to work very well. I've looked into sklearn.isotonic, and I've tried some heuristics myself (linear probability-based interpolation in between the false discards to approximate expected number of false discards at that probability). So far I'm not really satisfied with any outcome.

In case it's relevant: the model I use is xgboost.

I feel like this should be a rather common dilemma, but I haven't been able to find any robust methodology to deal with this. What is a good approach for dealing with this issue?

You need a cost sensitive learning mechanism for these type of tasks. You basically create a cost matrix that determines the cost of getting predictions wrong. For example while predicting whether a patient has cancer or not, the cost of predicting he doesn't have a cancer while he does is much more than predicting than he has a cancer and actually doesn't.
Here's a good article : https://mlr.mlr-org.com/articles/tutorial/cost_sensitive_classif.html

Here's one for python :
https://towardsdatascience.com/fraud-detection-with-cost-sensitive-machine-learning-24b8760d35d9


Is it OK to use the testing sample to compare algorithms? I'm working on a little project where my dataset have 6k lines and around 300 features, with a simple binary outcome.

Since I'm still learning ML, I want to try all the algorithms I can manage to find and compare the results.

As I've read in tutorials, I split my dataset into a training sample (80%) and a testing sample (20%), and then trained my algorithms on the training sample with cross-validation (5 folds).

My plan is to train all my models this way, and then measure their performance on the testing sample to chose the best algorithm.

Could this cause overfitting? If so, since I cannot compare several models inside model_selection.GridSearchCV, how can I prevent it to overfit?

Basically, every time you use the results of a train/test split to make decisions about a model- whether that's tuning the hyperparameters of a single model, or choosing the most effective of a number of different models, you cannot infer anything about the performance of the model after making those decisions until you have "frozen" your model and evaluated it on a portion of data that has not been touched.

The general concept addressing this issue is called nested cross validation. If you use a train/test split to choose the best parameters for a model, that's fine. But if you want to estimate the performance of that, you need to then evaluate on a second held out set.

If you then repeat process for multiple models and choose the best performing one, again, that's fine, but by choosing the best result the value of your performance metric is inherently biased, and you need to validate the entire procedure on yet another held out set to get an unbiased estimate of how your model will perform on unseen data.


Fine tuning CNN with imbalance data gives good results - not sure why I'm fine tuning with caffenet for a 3 classes classification problem. I have 100 instances of class A, 90 instances of class B and 30 instances of class C. I thought that my net would be biased toward classes A and B but I'm actually getting quite good results. I know that caffenet doesn't take care of imbalance data for me. Maybe it has to do with the fact that my entire train set fits into one batch? Or maybe it has to do with the fact that I'm not really training from scratch but mostly using caffenet's already learned weights? Thanks

Are you getting good results on your training or your test set? If it's only the first then you are overfitting, but if it's the second it will not have to do with the fact that your whole training set fits in your batch, the gradients still have a strong pull towards the dominating class(es). But having the pretrained weights will help a lot if the classes are relatively easy to seperate. It will also help if your dataset is similar to the one it was pretrained on, because the intermediate layers that do feature extractions have learned valuable filters for your problem instance.


How to integrate google cloud with dropbox and jupyter notebook using tensorflow So I opened up a google cloud account and have access to global and local (us east 1) resources (Compute Engine API , NVIDIA K80 GPUs) and connected it to my dropbox.

Next, I followed this youtube video to try to connect it to my jupyter notebook.

The code to be entered into the google cloud platform is as follows:

sudo apt-get update
sudo apt-get - -assume-yes upgrade
sudo apt-get - -assume-yes install software-properties-common
sudo apt-get install python-setuptools python-dev build-essentials
sudo easy_install pip
sudo pip install jupyter
jupyter notebook - - generate-config
sudo nano ~/.jupyter/jupyter_notebook_config.py

Insert these following lines under #configuration file for jupyter notebook
c=get_config()

c.NotebookApp.ip = ‘*’
c.NotebookApp.open_browser = False (or True)
c.NotebookApp.port=1000

ctrl+alt? enter ctrl+x?
from 3.5.0.tar.gz you write tmux0.tar.gz
ctrl+b shift+double_course (to get two diff windows) or exit to go out of the double

jupter notebook password
jupyter notebook –no-browser –port=1000

And then you go to your browser and write external ip:port (ex. 35.197.7.72:1000)
At the top window, upload your environment

BUT, before running the following line:

sudo nano ~/.jupyter/jupter_notebook_config.py

It asks me if I want to overwrite .../jupyter/jupyte_notebook_config.py with default config, and with both scenarios (when entering yes and when entering no) after I run the line it says

  sudo: nano: command not found

I even tried to pip install nano and it worked (the installation) but still the same response - anyone know what I should do?

The python package nano is not the text editor you need. Instead, you need the Ubuntu package nano. Try:

$ sudo apt-get install nano


Finding sequence combinations that impact target variable the most One can create a time series model to predict a target variable. What I need to do is find the input combinations and sequences that impact the target variable the most. In this case, the input data is a series of time steps, each of which has many features. The desired model would give clarity to not only the best features but the combinations and sequences of combinations that impact the target the most. This is desired because it models a system that I'd like to optimize based on this analysis.

Hope that makes sense. Ideas?

With machine learning, I could think that process analytics is what you are looking for:

I imagine you may have a dataset like this:

[step1, step2, step3]  -> [No Target]
[step1, step2]         -> [Target]
...

Business Process Analytics is the family of methods and tools that can be applied to these event streams in order to support decision-making in organizations. The analysis of process events can focus on the behavior of completed processes, evaluate currently running process instances, or focus on predicting the behavior of process instances in the future. Process Analytics

Is the most similar process (to your question) in Analytics I know


Random Forest Classifier - KFold CV Tunes Very Deep Trees --> Overfitting? I'm tuning a random forest in python and am wondering if/why my model is overfit. The dataset is described below:

1700 Positive Cases / 54000 total cases ~ 3.2% (unbalanced)
50 Numerical Features,~450 label/hot encoded features (post data
reduction)
10Fold CV using 85% of data, with 15% hold out for final test
Classification Metrics = AUC or F1 (as data is imbalanced)

The results I get tend to suggest using very deep trees i.e depth 18 with no restriction on number of samples per split = 2(default). In this case, Train AUC was 99.9% , Max Test AUC was 84%. My scores are also almost monotonically increasing in max depth of trees. Given the results and how deep the trees are - I suspect the model is overfit? If this is the case then why would I not observe some sort of out of sample reduction in AUC as depth and min_samples_split typically constrain the random forest? Or have I overlooked anything in tuning?

My ranges in CV Grid Search are more or less:

n_estimates : range(100,1000,by=100)
max_features : {sqrt(p),0.3,0.4,0.5}
max_depth : range(2,20,by=1)
min_samples_split : range(2,50,by=1)
class_weights : {balanced,None}

Thanks

There are signs of overfitting the data when the model's performance drops on the test set considerably.

Good feature engineering can also improves your model's performance. Considering that you have an imbalanced data set, you can look at resampling techniques for your data set such as random over/under-sampling and SMOTE. You can also extract feature importances from ensemble learners or regularized regression models like LASSO to see which features are contributing the most weight in your model's predictions. You could also apply various statistical tests such as ANOVA or chi-square square tests to gain further insights of your data. There is lots to do here.

You can also reconsider using the ROC AUC metric and choose something better tuned to a class imbalance in your data set. The ROC curve plots the True Positive Rates (TPR) on the x-axis and the False Positive Rates(FPR) on the y-axis whose formulas respectively correspond to $$TPR=\frac{TP}{TP+FN}$$ and $$FPR=\frac{FP}{FP+TP}$$ Since TPR is based only on the TP instances, the ROC curve won't measure the effects of negative instances. The AUC does does not consider the class imbalance when evaluating overall model performance, so we cannot trust it to weigh on the underrepresented class.

One way to resolve this is by using the Matthews Correlation Coefficient since it takes into account true and false positives and negatives. This helps represent the minority class a lot more during your model evaluation.

You could also look into random grid searched for your hyperparameter tuning; this method uses a fixed number of parameter settings sampled from your specified distribution. This allows you to get a general neighborhood of your parameter space where you can use a brute force grid search in a shorter parameter space. Then you can use cross validation to evaluate your model as you normally would.

This is just a handful of things to consider, there are many other things that you may find are suitable for your problem.


Standardization/Normalization test data in R I understand that one should standardize and normalize the test data (or any "unlabeled" data) with the training mean and sd. How can I implement this in R language? Is there a kind of "fitting" to the training set and a kind of applying to the test data?

Check out the preProcess function from the caret library.  You can choose the parameters you want to scale/center the training data, and it also saves the transformations it makes so then you can normalize the test set with the same specifications that you normalized the training set with.  Could go something like this:
library(caret)
trainData <- data.frame(v1 = rnorm(15,3,1), v2 = rnorm(15,2,2))
testData <- data.frame(v1 = rnorm(5,3,1), v2 = rnorm(5,2,2))
normParam <- preProcess(trainData)
norm.testData <- predict(normParam, testData)

now your norm.testData is scaled and centered according to the training data set parameters.
Another way to do this without using caret:
## set up data
trainData <- data.frame(v1 = rnorm(15,3,1), v2 = rnorm(15,2,2))
testData <- data.frame(v1 = rnorm(5,3,1), v2 = rnorm(5,2,2))

## find mean and sd column-wise of training data
trainMean <- apply(trainData,2,mean)
trainSd <- apply(trainData,2,sd)

## centered
sweep(trainData, 2L, trainMean) # using the default "-" to subtract mean column-wise
## centered AND scaled
norm2.testData <- sweep(sweep(testData, 2L, trainMean), 2, trainSd, "/")


Named entity disambiguation contests I am interested in the field of named entity disambiguation and want to learn more about it. I have heard that there are contests organised by various associations on these kind of research topics. These contests are very helpful as they give a practical experience in these fields.

I found one such contest organised by Microsoft research here though the dates have already passed. Can anyone point me to any other such contests ? Also, is there a site which catalogues these contests so that one can just go there and know about all upcoming contests ?

Thanks in advance.

Some of the GREC shared task challenges included a named entity recognition & coreference resolution component (i.e., disambiguation), but I don't think they've run GREC since 2010...?

https://sites.google.com/site/genchalrepository/reg-in-context/grec-ner


Multiple categories within a variable in decision tree This is more a conceptual question than related to the implementation on decision trees.

I've a feature vector say V1,V2,V3,target_variable

If the vector is a,b,c,true then, using normal decision trees, we can classify the data set.

But if the variable V1 is a set say {x,y,z},b,c,true how can I implement it?

I think of methods by using x,y,z as dimensions rather than categories but the problem is the number of categories are very huge in the order if millions. This solution won't scale.

Are there any efficient ways to deal with this problem?

Your intuition that a categorical feature is treated as multiple dimensions (i.e., multiple features) is correct. Millions of categories will be problematic for tree-baed algorithms that search through all features/categories for each split. A random forest would be a good model to try for your project, since it samples the features. Most software implementations let you set the sample proportion, so you could set yours really low if need be to speed up the split searches.

Alternatively you could feed your dataset into a Lasso or Elastic Net model to remove noisy variables and substantially reduce the quantity of categories under consideration. Lasso and Elastic Net have both been shown to do well on datasets where the number of features far exceeds the number of observations (e.g., dealing with genome data).


Does batch normalization mean that sigmoids work better than ReLUs? Batch normalization and ReLUs are both solutions to the vanishing gradient problem.  If we're using batch normalization, should we then use sigmoids?  Or are there features of ReLUs that make them worthwhile even when using batchnorm?

I suppose that the normalization done in batchnorm will send zero activations negative.  Does that mean that batchnorm solves the "dead ReLU" problem?

But the continuous nature of tanh and logistic remain appealing.  If I'm using batchnorm, will tanh work better than ReLU?

I'm sure that the answer depends.  So, what has worked in your experience, and what are the salient features of your application?

See, the basic concept behind the batch-normalization is that (excerpt from a Medium article)-

  We normalize our input layer by adjusting and scaling the activations. For example, when we have features from 0 to 1 and some from 1 to 1000, we should normalize them to speed up learning. If the input layer is benefiting from it, why not do the same thing also for the values in the hidden layers, that are changing all the time, and get 10 times or more improvement in the training speed.

Read the article here.

This is the reason why we use Batch-normalization. Now coming to your question, see as the output of sigmoid is constraints between 0 and 1, and that what is the motto of Batch-normalization.
If we use Bach-normalization with sigmoid activation, then it will be constrained between sigmoid(0) to sigmoid(1), that is between 0.5 to 0.73 ~ $frac{1}/{(1+1/e)}$.
But if we use ReLU with Batch-normalization then the output will be spread over 0 to 1 which is the good thing for us as finally, we want the output as much diverse as it can be.
So I think ReLU will be the best choice among other activations.


General Process for new project Often when i get a new project in machine learning the client always ask me either to do a particular task like a prediction for one thing or
give me data and ask me to find what i can do with it.

i've read the book hands-on machine learning with scikit-learn & tensorflow where you can see a full process for starting a project , basically drawing plots and search in correlation matrices what is interesting.

Do you guys have a special way to do this ? what is the step by step process you use ?

Thank you.

One of the standardized process for doing data analysis / data mining is the CRISP DM (Cross-industry standard process for data mining)
 which is one of the most adopted norms in the industry.

The process is very straightforward and starts by an important task which is the Business understanding of the issue or data you are dealing with.

This means that a prior research should be done if the analyst is not very familiar with the industry or the problem.

What makes this standard very popular is the fact that you can iterate various times between the step of formulating the problem ( writing in simple terms), data exploration and data visualization.

Here is a a very rich document that you can follow.

It would take a bit of time to go through all of the steps, however what could be quit interesting is the practice part, where you adapt your own client's problem into a simple question and start your exploration phase right away.


why do we operate with graphical models in VAE, if there are no probabilites involved? In the variational autoencoder, I often see graphical models e.g. $P(X|Z)$ for the decoder, but when I looked at code, I don't see any random variables, I see just deterministic network, with composite functions.

I am very confused by that, can someone explain where the magic happens? I am not interested in KL divergence and other stuff. I just don't understand how graphical model > $P(X|Z)$ == composite functions.

Not sure which type of code you were looking at but e.g. here they do sampling of random variables. The encoder deterministically maps the input X to mean and standard deviation vectors. Using these vectors you sample Z from factorizing Gaussians with these means and standard deviations. The samples of Z are the again mapped deterministically back to X by the decoder. The reparametrization trick is used to propagate the gradients through the network while only having to sample from standard Gaussians.


Sentence embeddings with LSTM to classify the sentences is not working I am trying to build LSTM NN to classify the sentences. I have seen many examples where sentences are converted to word vectors using glove, word2Vec and so on here is an example of it. This solution works, on the similar lines I wrote the below code which uses Universal Sentence encoder to generate the embedding of the entire sentence and use that with LSTM NN to classify the sentences but it is not working even after 200 epochs the model doesn't converge.
Please find the code below
import tensorflow as tf
import keras
from keras.layers import Input, LSTM, Dense, Activation, Dropout,Embedding
from keras.models import Model
import pandas as pd
import tensorflow_hub as hub

encoder=hub.load("https://tfhub.dev/google/universal-sentence-encoder/4")

word = "Elephant"
sentence = "I am a sentence for which I would like to get its embedding."
paragraph = (
    "Universal Sentence Encoder embeddings also support short paragraphs. "
    "There is no hard limit on how long the paragraph is. Roughly, the longer "
    "the more 'diluted' the embedding will be.")
messages = [word, sentence, paragraph,word, sentence, paragraph]
labels = [0, 1, 2,0, 1, 2]
reviews = tf.one_hot(labels, depth=3)
train_embed=encoder(messages)

embedding_layer = Embedding(6,512,trainable=False)
embedding_layer.build((None,))
embedding_layer.set_weights([train_embed])

input=Input(shape=(512,),dtype="float64")
X=embedding_layer(input)
X=LSTM(units=128,return_sequences=True)(X)
X=Dropout(rate=0.5)(X)
X=LSTM(units=128)(X)
X=Dense(units=3)(X)
X=Activation("softmax")(X)
model=Model(inputs=input,outputs=X)
model.compile(loss="categorical_crossentropy",optimizer="adam",metrics=["accuracy"])
model.fit(train_embed,reviews,epochs = 200,shuffle=True)

The out put looks like below
Epoch 196/200
1/1 [==============================] - 1s 534ms/step - loss: 1.1557 - accuracy: 0.3333
Epoch 197/200
1/1 [==============================] - 1s 501ms/step - loss: 1.0919 - accuracy: 0.5000
Epoch 198/200
1/1 [==============================] - 1s 518ms/step - loss: 1.2014 - accuracy: 0.0000e+00
Epoch 199/200
1/1 [==============================] - 1s 501ms/step - loss: 1.1008 - accuracy: 0.3333
Epoch 200/200
1/1 [==============================] - 1s 512ms/step - loss: 1.0509 - accuracy: 0.5000

Why is the model not converging? does sentence encoding doesn't work with LSTM?

The problem is that you are encoding the pieces of text as vectors and feeding those vectors to the model, but then the first layer of the model is again an embedding layer.
You should only use the embedding once: either you embed the text out of the model (train_embed=encoder(messages)) or you pass integer inputs to the model and them inside the model (X=embedding_layer(input)), but not both.
The problem is masked by the fact that the embedding layer does not raise an error when it receives an input of type float. Instead, it silently casts the input to integer.
Apart from the problem itself, there are some extra things that you should note:

depending on what you do, you may need to adjust the input to the LSTM to make it have the 3 dimensions it expects.
Here, the universal sentence encoder creates a fixed size vector for each sequence, and then you are feeding such fixed-size vector to an LSTM. It doesn't make much sense to have LSTMs process non-sequential data. I think you could just apply an MLP (i.e. a couple of dense layers with ReLU activations plus a final dense with the softmax activation) to the vector.
You are applying too much dropout for this case where there is almost no data.


Neural network options for simple data classification I want to clear about keras neural network options for classification of simple data where there are a number of features and one target column, as in iris flower dataset (Species is target):

SL,  SW,  PL,  PW,  Species
5.1, 3.5, 1.4, 0.2, setosa
4.9, 3.0, 1.4, 0.2, setosa
4.7, 3.2, 1.3, 0.2, setosa
...
...

I am finding that in almost all examples various combinations of Dense layers Dropout are the only options:

model = Sequential()
model.add(Dense(12, input_dim=4, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(8, activation='relu'))
model.add(Dense(1, activation='softmax'))

What other keras layers can be used in such situations, especially if the data is large, say with 50K rows and 100 features?

Edit:  My specific question is whether Dense and Dropout are the only kind of layers for this purpose and such data?

You may not be very familiar with deep learning. Each kind of network is used for a special kind of task, you cannot just stack LSTMs, GRUs, dense layers and other stuff without supervision. If you have a task that your patterns are local and they may be in multiple locations in an input pattern, you can employ convolutional layers for feature extraction and you can employ dense layers for classifying those extracted features. If you want to classify data which there is a kind of sequence in it, temporal data, you can employ LSTMs and GRUs and you can stack them for better accuracy and you can use their output and feed them to other networks based on your need. MLPs are good for learning non-linear mappings.

Dropout is used for avoiding overfitting.


MSE and time delay relation I have time series for which I used autocorrelation property and made autoregression model for prediction, with different time delays 0-30. I got diagram which is dependency MSE on time delay 0-30 http://imgur.com/a/7l2je and I don't know how to explain this discontinuity with the first dot for which delay is 0 and MSE is equal 0?

The zero time delay seems wrong there. Suppose you have an autoregressive model. A general formula could be:
$$y_t = \alpha + \sum_{i}\beta_i y_{t-i} + w_t$$
Now, if $i$ is zero then you will have
$$y_t = \alpha +\beta_0 y_t + w_t$$
which is really weird. It looks like you predict values at time $t$ from the values at time $t$. In plain English is like you predict the future values from the real values. In that case the error is zero, since it is not very hard to predict something from the real values. So, I think the offsets should start at $1$.


Why are the ANN training and validation accuracy graphs not smooth? I am currently training an ANN using Keras (Python3), and I am slowly optimizing the model's architecture and came across something I have not seen before.
The graph of the training and validation accuracy seems a bit odd. The graph appears 'step-like' in the sense that it is not a smooth curve, but different.
Model:
model = tf.keras.models.Sequential()

model.add(Dense(7,activation='relu'))
model.add(Dense(10,activation='relu'))
model.add(Dense(10,activation='relu'))
model.add(Dense(1,activation='sigmoid'))

model.compile(optimizer=tf.keras.optimizers.Adam(0.0001),
              loss='binary_crossentropy',
             metrics=['accuracy', 'mse', 'mae'])

history = model.fit(
    X_train.values,
    y_train,
    epochs=1000,
    validation_split = 0.2,
    verbose=1,
    )

My two questions are:
(1) what is this indicative of, and
(2) how do i fix this assuming it is problematic?

Considering that your validation accuracy has bigger steps than your training accuracy, this may simply be an issue of data size. Accuracy counts correct/not correct, so if the model switches its opinion on a sample the accuracy increases suddenly. Because your validation data is likely smaller than the training data the steps are bigger there.
There is nothing to worry about, this looks normal. On the other hand your validation loss is increasing, so you are overfitting.


How to choose the best algorithm this might be simple  but I need to know how to choose a best algorithm based on a scenario.

I have a dataset. The target class is, let's say color, this color attribute can have upto 5 values (red, yellow, blue, green, pink)

So I have:

1.) 1 target class - color

2.) Multiple values for that target class - (red, yellow, blue, green, pink)

As this is a labelled dataset with known classes, I can use a Supervised Learning Algorithm

But I don't know which algorithm would be the best. It should be a multi-class classigication? Or multi-labeled ? Can someone help me to find the best algorithm?

note: I know choosing an algorithm depends on many factors (such as its acuracy), but at a glance, what type of algorithms should I try?

Thank you

Okay, so without going in to details,

Your target values are nominal, discrete values, hence the classification model would solve your problem.

Under classification there are variety of methods available,for example, there is decision tree, Naive Bayes etc.. . you may try all of these and check which gives the better confusion metrics.

Multi-Class means, there are multiple classes , such as in you colors.

If I have books features as input and i want to predict its genre, and the available categories i can put this book in are- fictional, romance, historical. all these labels makes it multi class because multiple genre.A book can be of any of the genre.

Multi-Label means, an instance can belong to more than 1 class.

Now, if this one book is fictional romance , then this makes it multi label. Because out of 3 available categories, this one book belongs to 2(it can be more than 2 also).

Now, This was just a short answer to get you started,please read on these key words to further details.

  1.) 1 target class - color

This is target attribute, color

  2.) Multiple values for that target class - (red, yellow, blue, green, pink)

Multiple class/labels for target attribute, red, yellow, blue... are classes/labels. Since there are more than 2 classes its multi class.

If you input map to 2 or more colors than 1 then it makes it multi label.


Meaning of Perceptron optimal weights Im studying the perceptron algorithm.
I know that we can use the weights w as the coefficients of the hyperplane that will separate the vectors that need a classifications

In every web page i read up for a detailed explanation, it's generally said that the perceptron algorithm will find the optimal weights w. But optimal in which sense ?

Media's explanation is true for regression problems. These are problems where you predict a continuous target variable.

Your image shows a classification problem. Here, the target variable takes only two values (typically -1 and 1 in the Perceptron algorithm).
In that case, an optimal solution $w^*$ is a vector of weights that perfectly separates both classes.
If such a solution exists, the Perceptron algorithm will find it. But: If there is one optimal solution, there are usually infinitely many other optimal solutions. You can easily see this in your image: You can move the line a little to the left or to the right, and you can rotate it a little, and it still perfectly separates the classes.

So while the Perceptron algorithm will find an optimal solution if there is one, you cannot know which one it will find. That depends on the random starting parameters.

This is different e.g. for support vector machines. Here, there is either no optimal solution or exactly one optimal solution.


Want suggestions on choosing open source embedded BI tool I need some suggestions from all group members regarding the open source reporting/dash-boarding tool which fulfills below specific requirements apart from some basic BI functionalities:
1. Can be embedded inside a web
2. Interactive (filters and one chart selection affecting other charts)
3. Able to give excel downloadable option for plotted data
4. Role based user access

You can suggest tools supporting all requirements or partial also.

You can try Power BI - it can do all mentioned stuff ( and even more, you can plug in R/python code), but its not open-sourced, though you can go for a free 1-user account.

I guess tableau can do the same and also has "community" version, but im a fan of Power BI, so cant tell more about it.

If you need some custom, build-from-scratch solution, you can go for Shiny - fully build on R, fully open sourced and has free hosting option, can be embedded as an iframe ( will prolly look horrible), but fully customizable and can be tuned the way you want it to be.

Similar to Shiny, but with Python: Dash


Multivariate Time series I have a dataset which have two features,"feature_1",and "feature_2" and time in timestamp of 10 second intervals.
While plotting,there is  "trend" of both features with respect to time,but don't have seasonality .And both features are also highly correlated.
The task is to predict "feature_2" values.
And importantly,in test set "feature_1" values are also given.
Which model and approach should I go with

Most of the commonly used techniques are based on time series data (predict tomorrows value based on the values from yesterday, last week, last month, etc)
Since you have some features in your dataset that are not time series (feature_1), you may want to look at Causal Forecasting along with classical time series forecasting. One example of Causal Forecasting the Vector Auto Regressive model (VAR), combined with Granger Causality to look at the relationship over time between different time series and build a model to predict future events.
Definition of causal forecasting:
"Causal forecasting is the technique that assumes that the variable to be forecast has a cause-effect relationship with one or more other independent variables. Causal techniques usually take into consideration all possible factors that can impact the dependent variable"
reference:
https://www.brillio.com/insights/choosing-the-right-forecasting-technique/
other references:
causal:
https://towardsdatascience.com/granger-causality-and-vector-auto-regressive-model-for-time-series-forecasting-3226a64889a6
https://courses.edx.org/asset-v1:MITx+CTL.SC1x_1+2T2015+type@asset+block/w4l1_causalmodels_ANNOTATED_FINAL.pdf
classical:
https://machinelearningmastery.com/time-series-forecasting-methods-in-python-cheat-sheet/


How to separate overlapping curves more effectively? So, I am trying to create a Neural Network which will effectively separate 2 Gaussian curves with somewhat different means and standard deviations. My basic aim is, for some given input vector the Neural Network will select 2 points from the 2 Gaussian curves (selection based on the input vector) and compare them, and output a value depending on the Gaussian curve which gives a greater value.

Now, if I implement a simple Neural Network with 2 hidden layers the accuracy of my predictions is about 68.75% for 2 Gaussian curves with N(0,1) and N(0,2000) respectively and 100% for N(0,1) and N(500,1)  (my NN outputs 0, 1 and hence I can easily measure accuracy by true_positives/total_samples). Clearly more the overlapping curves, less the accuracy as the NN cannot distinguish between the 2 curves.

I have tried increasing the number of hidden layers, changing learning rate but still there is no improvement in accuracy. I am using back-propagation with momentum.

So my question is what structure, learning algorithm and modifications in the Neural Network will help me to effectively separate 2 curves which is kind of overlapping for a better accuracy?

Clarification 1: I have generated the data myself for testing purposes, in real situation I will only have an input/output pair from a black-box, and all I know the black-box follows some Gaussian distribution. Also the input vector has to be arranged in some sort of way, and then if we fit the curve to it, it turns out to be Gaussian.

Clarification 2: I am looking for possible hints. Any experienced person in this field might make a guess for some better methods. I know you cannot have a definite method, so suggestion of methods which might work are welcome.

The problem here isn't your chosen neural network architecture, it's that you're working with literally unseparatable data.

There's isn't any practical way to distinguish observations near the mode of multiple distributions that share a mode. The best you can do is approximate the distance from the mode at which the high variance distribution is more likely than the low variance distribution, and classify observations relative to that. You can do that analytically, and your neural network isn't going to do any better.

Have you tried visualizing your data to better understand what you're tasking your model with here? This is a bit more extreme than "kind of overlapping" distributions.


Regression: how to interpret different linear relations? I have three datasets, let's call them X and Y1 and Y2. A scatterplot is produced out of them, with Y1 and Y2 sharing them same X dataset (or support).

My question: if the two regression lines are different in both slope and intercept, is there a way to evaluate if the X dataset has more influence on Y1 or Y2?

Based on the image below, this is to say - which Y dataset is more influenced by the X dataset?

Blue slope (Y1): -112
Red slope (Y2): -90

EDIT

It is visible that an increase in X produces a decrease in both Y1 and Y2. My question could be interpreted as follows: which Y dataset decreases the most, given the increase in X? Is the slope everything I need?

Image:

What you are looking for is the Analysis of Covariance (ANCOVA) analysis, which is used to compare two or more regression lines by testing the effect of a categorical factor on a dependent variable (y-var) while controlling for the effect of a continuous co-variable (x-var).

Here is an example for carrying out the ANCOVA analysis using R.


What are some good methods to forecast future revenue on categorical and value based data? I have monthly snapshots (3 years) of all the contract data. It includes following information:

Contract status [Categorical]: Proposed, tracked, submitted, won, lost, etc
Contract stages [Categorical]: Prospecting, engaged, tracking, submitted, etc.
Duration of contract [Date/Time] : months and years
Bid Start date [Date/Time]: Date (But this changes when the contracts are delayed)
Contract value [Numerical] : Value of the contract in local currency
Future revenue projection [Numerical]: Currency value breakdown of revenue for next 5 years (this value is available for all the contracts, no matter if it's won or lost)

I also have other information about the contracts like id, name, description, etc.
Answers I am trying to get:

Total value of contracts that are changing status from month to month
Total value of contracts that are changing stages from month to month
Average delay of the start date of the contracts
Future revenue projection (5 years) based on change of status and average delay

Problems I am having with this data:

It's not time series data, it's monthly snapshot, so I can either turn it into monthly time series dataset and accumulate revenues based on each status and stages or count of all the contracts.

Do I accumulate the contracts data or leave it as individual contracts? In the later case, how do I feed it to any model? It won't be a time series data then.

Main problem with finding the right approach:

I am not sure what approaches to use to answer very different questions. Some values are categorical and some are numerical. I am not sure if it is a forecasting problem or 'change in event' prediction problem. Or mix of both?

How do I incorporate, these very different categorical variables with numerical revenue value, into any model.

Methods I looked into:

I have read about forecasting models like ARIMA (mostly sales data). It takes time series data to forecast revenues based on historical data. I am not sure if it is valid here because I have contracts that changes status, and I am not sure how to use it in ARIMA model. Or if it is necessary to do so. I am also not sure if there is a seasonality to the data. Contracts winning or losing is not a seasonal event.

I also looked into  Simple Exponential Smoothing (SES) and Holt Winter's Exponential Smoothing (HWES) examples and found the same issue while calculating average delays or forecasting future revenue. The current data is not univariate.

I looked at following answers: https://stats.stackexchange.com/questions/246151/difference-between-time-series-prediction-vs-point-process-prediction and it made me think that maybe my problem is not time series prediction.

Best Approach to Forecasting Numerical Value Based on time series and categorical data? : This made me think maybe I should look into RNN and LSTM.

How do you predict a continuous variable when all your independent variables are categorical : Or my problem is similar to this one.

I am sorry for the long post. I am trying to make the problem as clear as possible. I have no idea what would be the best approach to solve this problem and what data to feed to the model. I am also lost at how to structure the data to get the best use out of all the variables.
I would be grateful if you can help me suggest any good methods or reading resources, so I can answer the questions. Thank you for your time!

If you want to use neural networks this post on Kaggle might help: https://www.kaggle.com/c/m5-forecasting-accuracy/discussion/159052
It has a short list of resources for categorical embeddings and LSTM (I think).
If you think your dataset has periodic patterns, and you only need to answer your questions (not deploy a model). I would take a look a FB Prophet: https://facebook.github.io/prophet/docs/quick_start.html#python-api
It extracts the periodic components and fits them with sine and cosine waves. You can also add additional regressors, i.e., one-hot encoded categorical variables.


What scale does LightGBM use for output? Let's assume I'm modeling a process like:

$$y=log(x1+x2+x3+N)$$

Where $x_i$ are features and $N$ is some error/noise value.

With the way that decision trees work and the way that LightGBM works, will this be modeled correctly? Or will it be necessary for me to train using $e^y$?

The underlying model will be a stepwise function. I don't see any garanty that it will work better (or worse) with the transformation, in the general case. This may be different depending on you variable (for binary variables you may want to work directly on the linear predictor).

However, in practice, if you know there is an underlying link transformation, I would suggest you to use it. This will often ease the model intepretation.


How to adjust/smooth a certain number using constants or rules Hi, I am handling a dataset with a customer purchase history.
The field ord_cnt represents the purchase without coupon usage, and cpn_ord_cnt represents the purchase with coupon usage.

There are two things I'm trying to figure out:

How to penalize the small purchase count when it comes to a coupon purchase proportion.

I added a field 'mod_cpn_ratio' because when i calculate the simple proportion: (cpn_ord_cnt/total_ord_cnt), it did not do a good job for the customers with the smaller number of purchases, eg. (1 coupon purchase / 2 entire purchases) = 0.50% vs (50 / 100) = 0.50%

So for the 'mod_cpn_ratio' I added a constant '10' to the denominator to penalize those who bought less. I wonder if this was a fair way to handle this problem, or if there are better ways to handle it(possibly not just constant but dynamic number)

Rank customers with total_ord_cnt, but give some extra credit for those who made more purchases without coupons.

When two customers have the same number of 'total_ord_cnt', I would like to rank higher the one purchased more without coupons.

Any advice would be appreciated.

Thanks!

This is a good question. The best method will be whatever gives you the best result, so the method you suggest of adding a constant to the denominator might be the right idea. However, I don't think it is ideal. The reason is that you will be systematically underestimating the percentage of coupon uses, which is not ideal. It also might be difficult to interpret or explain.

What I suggest instead is to borrow a play from the Bayesian handbook and make use of a prior. A prior represents your beliefs about the coupon spending of the average person. For example, you can represent your belief about the distribution of percent coupon usage as a beta distribution. All you need to know about this distribution is that it has parameters $\alpha$ and $\beta$ and that $\alpha$ corresponds to the number of products that used coupons while $\beta$ corresponds to the number of products that didn't use coupons. Let's say you believe the average person uses coupons for 10% of items. You might choose $\alpha=1$ and $\beta=9$ for your prior which has a mean of $\tfrac{\alpha}{\alpha+\beta}=0.1$.
 For each person in your dataset, you can estimate how often they use coupons by updating the prior with the new information. For example, if the person used one coupon on two items, you update the beta distribution by adding 1 to $\alpha$ and 2 to $\beta$ which gives a new estimated mean of $\tfrac{2}{2+10}=0.17$. You can see that when the number of purchases for an individual is small, you have little information, so your estimate is going to be close to your prior. Without much information, you assume a person is close to the average. However, when you have a lot of information, the effect the prior will be minimal because you have a lot of information and no longer need to make assumptions.

The simple way to think about this is that you are adding a constant to the numerator and the denominator: (# coupon purchases + $\alpha$) / (# entire purchases + $\alpha$ + $\beta$)

Note: even for a constant proportion like 10%, you can choose different values for $\alpha$ and $\beta$ that represent how confident you are in the prior. For example, you can choose $(\alpha,\beta) = (0.1, 0.9)$ or $(\alpha,\beta)=(10,90)$. This controls how much "smoothing" you apply.


Why the VC dimension to this linear hypothesis equal to 3? I am trying hard to understand this. Here is the scenario:

X = R^2
H = { h(x) = x + 10 }

I need to calculate the VC dimension for the above linear separator. Somehow, the VC dimension for this linear separator is 3. I just cannot understand how.

According to what I understand, VC dimension is the size of the largest finite subset of X which can be shattered by h. So if there exists a subset of size n where all points are shattered by h and there exists a subset of size n+1 where at least one point is not shattered by h, then the VC dimension will be n.

Is the VC dimension >= 1?

  Yes. We just need to pretend there is a single point on the line and by keeping the line (X-axis) steady we can flip which side is positive/negative

Is the VC dimension >= 2?

  Yes, because we could separate all 4 combinations { ++, --, +-, -+ } using a single line

Is the VC dimension >= 3?

  This should be NO, according to what I understand. How could we separate this case + - +?

But I was going through a video, that explains VC dimensions which says it is three and I do not understand how.

The VC dimension for the classifier depends on the dimension of space that your data points belong to.

In your problem, if your mean $x\in \mathbb{R^2}$, then the VC dimension is 3. For $\mathbb{R^2}$, you can always shatter any three general position points ("general position" means they do not coincidentally lie on the same line). For instance, consider three points (0,0), (0,1), (1,0). No matter how you assign labels to them, you can always seperate them by a line. If you consider point points with label (0,0), (0,1), (1,0), (1,1) with label [+,-,-,+], this is the "xor" case which cannot be seperated by a line.

For $x\in \mathbb{R}$, then the VC dimension is 2 because you cannot separate +-+.

In general for $x\in \mathbb{R^d}$, the VC dimension for a linear classifier is $d+1$.


CNN - How does backpropagation with weight-sharing work exactly? Consider a Convolutional Neural Network (CNN) for image classification. In order to detect local features, weight-sharing is used among units in the same convolutional layer.
In such a network, the kernel weights are updated via the backpropagation algorithm.

An update for the kernel weight $h_j$ in layer $l$ would be as follows:

$h_j^l = h_j^l - \eta \cdot \frac{\delta R}{\delta h_j^l} = h_j^l - \eta \cdot \frac{\delta R}{\delta x_j^{L}} \cdot \frac{\delta x_j^{L}}{\delta x_j^{L - 1}} \cdot ... \cdot \frac{\delta x_j^{l}}{\delta h_j^l}$

How can the kernel weights be updated and still be the same (=shared)?

I have 2 possible explanations:

Weights of the same layer, which are initialized to the same value, will stay the same (independently of the input). This would imply that the expression $\frac{\delta R}{\delta h_j^l}$ is the same for all of these weights $h_1^l$ to $h_J^l$. This does not make sense, since $x_j^l$ is different for different j's. Or am I missing something here?
There is a trick, e.g. after the back-propagation update, the shared weights are set to their mean.

EDIT
The confusion I had was that I didn't consider that if a weight is shared, its parameter $h_j^l$ appears several times in the loss function. When differentiating for $h_j^l$, several terms (considering the according inputs) will "survive". Therefore the updates will be the same.

I think you're misunderstanding what "weight sharing" means here. A convolutional layer is generally comprised of many "filters", which are usually 2x2 or 3x3. These filters are applied in a "sliding window" across the entire layer's input. The "weight sharing" is using fixed weights for this filter across the entire input. It does not mean that all of the filters are equivalent.

To be concrete, let's imagine a 2x2 filter $F$ striding a 3x3 input $X$ with padding, so the filter gets applied 4 times. Let's denote the unrolled filter $\beta$.

$$X = \begin{bmatrix}
    x_{11}       & x_{12} & x_{13} \\
    x_{21}       & x_{22} & x_{23} \\
    x_{31}       & x_{32} & x_{33}
\end{bmatrix}$$

$$F = \begin{bmatrix}
    w_{11}       & w_{12} \\
    w_{21}       & w_{22}
\end{bmatrix}$$

$$\beta= [w_{11}, w_{12}, w_{21}, w_{22} ]$$

$$F*X = \begin{bmatrix}
\beta \cdot [x_{11}, x_{12}, x_{21}, x_{22}] & \beta \cdot [x_{12}, x_{13}, x_{22}, x_{23}] \\
\beta \cdot [x_{21}, x_{22}, x_{31}, x_{32}] & \beta \cdot [x_{22}, x_{23}, x_{32}, x_{33}]
\end{bmatrix}
$$

"Weight sharing" means when we apply this 2x2 filter to our 3x3 input, we reuse the same four weights given by the filter across the entire input. The alternative would be each filter application having its own set of inputs (which would really be a separate filter for each region of the image), giving a total of 16 weights, or a dense layer with 4 nodes giving 36 weights.

Sharing weights in this way significantly reduces the number of weights we have to learn, making it easier to learn very deep architectures, and additionally allows us to learn features that are agnostic to what region of the input is being considered.

EDIT: To further motivate this, here's an animation of a 3x3 filter applied to a 5x5 input


How to speedup K-Means used in 'for loop' I'm trying to solve an interesting problem. One solution which seems to work well, involves using K-Means in a 'for loop'. The dataset per loop is independent and fairly small (Minibatch not required).

e.g.

for i in range(100):
    y = Kmeans(x) # x is new data per loop

Results are good but execution is quite slow. Debugging shows that most of the delay is in the K-Means algorithm (from scikit-learn).

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    40960   13.402    0.000   92.785    0.002 k_means_.py:426(_kmeans_single_elkan)
   243337    8.506    0.000   51.421    0.000 pairwise.py:165(euclidean_distances)
   931529    6.950    0.000    6.950    0.000 {method 'reduce' of 'numpy.ufunc' objects}
   341357    5.178    0.000   32.424    0.000 validation.py:332(check_array)
2157218/1952418    5.010    0.000   28.877    0.000 {built-in method numpy.core._multiarray_umath.implement_array_function}
   341357    4.617    0.000   15.927    0.000 validation.py:36(_assert_all_finite)
   117456    3.356    0.000    3.849    0.000 _k_means.pyx:258(__pyx_fuse_1_centers_dense)
   789264    2.617    0.000    9.859    0.000 fromnumeric.py:73(_wrapreduction)
    40960    2.584    0.000   37.103    0.001 k_means_.py:43(_k_init)
  1265786    2.053    0.000    2.053    0.000 {built-in method numpy.array}
   253435    2.015    0.000    2.015    0.000 {built-in method numpy.core._multiarray_umath.c_einsum}

I've already reduced the number of iterations and initialisations. Is there a way to speed up this execution?

Using the multiprocessing library I was able to parallelize the loop and speedup execution.


multiclass-multioutput classification I am new in ML field and I got this question when classifying the DEAP data with sklearn. There is my data set description.

The whole data is a (1280*322) matrix，which the last two columns are the labels and the rest of it is the feature（1280*320).  Because the output of the labels has three categorical values, I try to predict this problem with RandomForestClassifier. However, the mean accuracy of the result is just 0.222656250. Would you give me any clues about this problem? Thanks for your patience. the detail code as follows:

RandomForestClassifier

import scipy.io
from sklearn.datasets import make_classification
from sklearn.multioutput import MultiOutputClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.utils import shuffle
import numpy as np

def mydata():
    data = scipy.io.loadmat('/home/jdang/Documents/RenKaimeng/EEGfile/Classification-for-DEAP/whole.mat')

    # initialization

    x_vals = data['all_data'][:,0:320]

    y_vals_Valence = np.array([0 if each=='Neg'  else 1 if each =='Neu' else 2 for each in data['all_data'][:,320]])
    y_vals_Arousal = np.array([0 if each=='Pas'  else 1 if each =='Neu' else 2 for each in data['all_data'][:,321]])

    Y = np.vstack((y_vals_Valence, y_vals_Arousal)).T

    DEAP_x_train = x_vals[:-256] #using 80% of whole data for training
    DEAP_x_test = x_vals[-256:]  #using 20% of whole data for testing
    DEAP_y_train = Y[:-256]
    DEAP_y_test = Y[-256:]

    forest = RandomForestClassifier(n_estimators=100, max_depth=2,random_state=1)
    multi_target_forest = MultiOutputClassifier(forest, n_jobs=-1)
    multi_target_forest.fit(DEAP_x_train, DEAP_y_train)

    print("the result of MutliOutput-classification-using-RandomForest ")
    print('The mean square error %10.9f ' % np.mean((multi_target_forest.predict(DEAP_x_test)-DEAP_y_test)**2))  # The mean square error

    print('the mean accuracy on the given test data and labels %10.9f'% multi_target_forest.score(DEAP_x_test, DEAP_y_test))

Increasing the max_depth and the n_estimators might help. Especially the max_depth (2 is way too low). Take a look at that.


Inputs required for Random Forest Regressor and ways to improve performance I am using Random Forest Regressor to predict inventory needs. The data I am using to train the model lists the total quantity picked for each product per date, but does not include rows where total quantity picked for a product on the specified date is 0. The model considers the features that allow it to take date into consideration. For details on the data being used see the example data below:

                 UPC  day_ID  month    day_of_year  day_of_week  quantity_picked
0      0000000002554  7500.0      5            141            1              4.0
1      0000000002554  7503.0      5            144            4              2.0
2      0000000002554  7512.0      6            153            6              2.0
3      0000000002554  7527.0      6            168            9              2.0
4      0000000003082  7494.0      5            135            2              2.0
5      0000000003082  7495.0      5            136            3              2.0
6      0000000003082  7496.0      5            137            4              8.0
7      0000000003082  7497.0      5            138            5              4.0
8      0000000003082  7498.0      5            139            6              4.0
9      0000000003082  7499.0      5            140            0              9.0
10     0000000003082  7500.0      5            141            1              3.0
11     0000000003082  7501.0      5            142            2              5.0
12     0000000003082  7502.0      5            143            3              3.0
13     0000000003082  7503.0      5            144            4              8.0
14     0000000003082  7505.0      5            146            6              2.0
15     0000000003082  7506.0      5            147            3              7.0

Will the model be less accurate at predicting inventory needs because it is missing dates for items where quantity picked is 0?
I have tried running the same model with the rows where quantity picked = 0 but the total number of rows changes from approximately 50k to 5 million and my computer literally can't handle it, it just freezes.
Without the rows where quantity picked = 0, the model reports mean squared log error level of .39448 and runs successfully within 4 minutes 37 seconds.

Any guidance on if that data is necessary or not would be very much appreciated and/or advice on how to improve performance/accuracy of such a model.

A random forest trained on data whose labels are all positive integers cannot produce any prediction less than 1. So yes, the model will be less accurate.

I would suggest including a random subset of the rows with 0 sold, so that the forest can learn a (hopefully representative) pattern to them.  I'd also probably weight them higher in the fitting algorithm, to reproduce a more faithful average of items sold (in each leaf).


R-square or adjusted R-square for one variable model? I have model like y=mx. Since the adjusted R2 tells you the percentage of variation explained by only the independent variables that actually affect the dependent variable and I have only one independent variable, do I need to consider my adjusted r-square value? Or, r-square is good for this type of model?

They're going to be very similar (practically the same), for a model with only one independent variable. So I'd say it doesn't matter without understanding better your purpose in using R2 / Adj R.


How to split data into training, validation, test data sets if the data is non-stationary? When splitting data into training, validation, test data sets to be fed to machine learning model, the data is ideally expected to be stationary. However, in the real world, some data is non-stationary. For example, financial time-series data are non-stationary.
So, for this kind of non-stationary data, how do you split data into training, validation, test data sets?

In general, for time series data, you can split it in such a way that the temporal order is as is in all your splits (training, validation and testing sets).
Once you make your series stationary (via differentiation for instance), you can make use of the TimeSeriesSplit cross-validator by scikit-learn, with wich you get the time-ordered indices of each train-validation split, so that you can use them later on with the validation strategy you might want to use, something like:

As a possible validation strategy, you could use the walk-forward validation (nice info here)


Time-series forecasting Here is the data:

l <- data.frame(date = c(1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24))

k <- data.frame(cost = c(25,20,18,15,5,0,0,0,10,15,30,40,45,34,26,20,10,7,4,4,15,34,57,62))

m <- cbind(l,k)

ggplot(m, aes(m$date,m$cost)) + geom_line()

This is the graph I get:

What is a good prediction model? I think I can use polynomial regression  if I subset the max and min values. see image below for better understanding. (Red for max, blue for min, lines were created using paint to explain a point)

Another way, I don't know what it is called, but I think they use it to predict weather, Not sure what the formula to obtain the  blue line below would look like. (Look at graph below for better understanding)

What would be an appropriate formula to get the blue fitted line and predict the points highlighted in red??

Something simlar to the blue fitted line can be obtained using Holt-Winters model. Check HoltWinters() function from stats package in R.

You can find some examples here and here.


Is there a way in pandas to import NA fields as a string rather than NaN? I'm doing a Kaggle challenge, and a lot of entries in the data are NA. However, according to the data description, this doesn't actually mean "missing data", it means something like "Not applicable", in the sense of it just not having that quality (for example, not having a basement).

However, when I use pandas to import the data using read_csv(), and then use head() to look at it, it shows NaN for all those things that should be NA (comparing with the spreadsheet in LibreOffice).

I know how to just replace one value with another for a given column, but there's still a problem. NaN is usually what it should enter if there's just a missing value. So, if it's importing both "NA" and a blank cell as NaN, I won't know which it is.

It's probably not a huge issue, because I'll probably want to replace actual missing values with something common anyway, but it's good to know. I know I could also go into the csv with regular python and rename those actual "NA"s, but that's cumbersome. Is there a way I can just import certain columns as strings?

Use the pandas.read_csv() options:

d = pandas.read_csv('foo.csv', keep_default_na=False)

  na_values : scalar, str, list-like, or dict, default None

  Additional strings to recognize as NA/NaN. If dict passed, specific >per->column NA values. By default the following values are interpreted as >NaN: ‘’, ‘#N/A’, ‘#N/A N/A’, ‘#NA’, ‘-1.#IND’, ‘-1.#QNAN’, ‘-NaN’, ‘-nan’, >‘1.#IND’, ‘1.#QNAN’, ‘N/A’, ‘NA’, ‘NULL’, ‘NaN’, ‘n/a’, ‘nan’, ‘null’.

  keep_default_na : bool, default True

  If na_values are specified and keep_default_na is False the default NaN >values are overridden, otherwise they’re appended to.

  from here


How to load all images using image_dataset_from_directory function? I am working on a multi-label classification problem I faced memory issues so I want to use Keras image_dataset_from_directory method to load all images as batch. how to apply multi-label technique on this method.
I have these folders.
['Tomato_BacterialSpot', 'Tomato_EarlyBlight', 'Tomato_Healthy', 'Tomato_LateBlight']
I am generating class names using the below code.
Here the sample code tutorial for multi-label but they didn't use image_dataset_from_directory  technique.
https://www.pyimagesearch.com/2018/05/07/multi-label-classification-with-keras/
 label = imagePath.split(os.path.sep)[-2].split("_")

and I got the below result but I don't know how to use image_dataset_from_directory method to apply the multi-label?

BacterialSpot
EarlyBlight
Healthy
LateBlight
Tomato

You don't actually need to apply the class labels, these don't matter. Keras will detect these automatically for you. It does this by studying the directory your data is in. Make sure you point to the parent folder where all your data should be. Your data should be in the following format:
my_data/
...BacterialSpot/
...EarlyBlight/
...Healthy/
...LateBlight/
...Tomato/

where the data source you need to point to is my_data. Here is an implementation:
train = tf.keras.preprocessing.image_dataset_from_directory(
  'my_data',
  validation_split=0.2,
  subset="training",
  image_size=(128, 128),
  batch_size=128)

val = tf.keras.preprocessing.image_dataset_from_directory(
  'my_data',
  validation_split=0.2,
  subset="validation",
  image_size=(128, 128),
  batch_size=128)

Then when you run it:
Found 3647 files belonging to 1 classes.
Using 2918 files for training.
Found 3647 files belonging to 1 classes.
Using 729 files for validation.

Keras has detected the classes automatically for you. I have used only one class in my example so you should be able to see something relating to 5 classes for yours. See an example implementation here by Google:
https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/images/classification.ipynb#scrollTo=iscU3UoVJBXj


SAP DI: Ml Scenario Manager - Git How do you manage the code (and notebooks) in the ML Scenario Manager workspace in SAP Data Intelligence?

It seems that SAP DI does not support a direct integration of Git.

For every ML Scenario, there is a notebook section where you can upload/create files with code and/or Jupyter notebooks:


Implementing "full convolution" to find gradient w.r.t the convolution layer inputs I've been trying to implement "full convolution" w.r.t to convolution layer inputs. According to this article, it looks like this:

So, I wrote this function:

def full_convolve(filters, gradient):
    filters = np.ones((5,5))
    gradient = np.ones((8,8))
    result = list()
    output_shape = 12
    filter_r = filters.shape[0] - 1
    filter_c = filters.shape[1] - 1
    gradient_r = gradient.shape[0] - 1
    gradient_c = gradient.shape[1] - 1

    for i in range(0,output_shape):
        if (i <= filter_r):
            row_slice = (0, i + 1)
            filter_row_slice = ( 0 , i + 1)
        elif ( i > filter_r and i <= gradient_r):
            row_slice = (i - filter_r, i + 1)
            filter_row_slice = (0, i + 1)
        else:
            rest = ((output_shape - 1) -  i )
            row_slice = (gradient_r  - rest, i + 1 )
            filter_row_slice = (0 ,rest + 1)
        for b in range(0,output_shape):
            if (b <= filter_c):
                col_slice = (0, b + 1)
                filter_col_slice = (0, b+1)
            elif (b > filter_c and b <= gradient_c):
                col_slice = (b - filter_c, b + 1)
                filter_col_slice = (0,b+1)
            else:
                rest = (output_shape - 1 ) - b
                col_slice = (gradient_r - rest , b + 1)
                filter_col_slice = (0, rest + 1)
            r = np.sum(gradient[row_slice[0] : row_slice[1], col_slice[0] : col_slice[1]] * filters[filter_row_slice[0]: filter_row_slice[1], filter_col_slice[0]: filter_col_slice[1]])
            result.append(r)
    result = np.asarray(result).reshape(12,12)

I tested this with ones and the output seems correct (if I get "full convolution" right):

[[ 1.  2.  3.  4.  5.  5.  5.  5.  4.  3.  2.  1.]
 [ 2.  4.  6.  8. 10. 10. 10. 10.  8.  6.  4.  2.]
 [ 3.  6.  9. 12. 15. 15. 15. 15. 12.  9.  6.  3.]
 [ 4.  8. 12. 16. 20. 20. 20. 20. 16. 12.  8.  4.]
 [ 5. 10. 15. 20. 25. 25. 25. 25. 20. 15. 10.  5.]
 [ 5. 10. 15. 20. 25. 25. 25. 25. 20. 15. 10.  5.]
 [ 5. 10. 15. 20. 25. 25. 25. 25. 20. 15. 10.  5.]
 [ 5. 10. 15. 20. 25. 25. 25. 25. 20. 15. 10.  5.]
 [ 4.  8. 12. 16. 20. 20. 20. 20. 16. 12.  8.  4.]
 [ 3.  6.  9. 12. 15. 15. 15. 15. 12.  9.  6.  3.]
 [ 2.  4.  6.  8. 10. 10. 10. 10.  8.  6.  4.  2.]
 [ 1.  2.  3.  4.  5.  5.  5.  5.  4.  3.  2.  1.]]

However, I don't like all these manual checks and if/else statements. I feel there is a better way to implement this in NumPy (perhaps, using some zero paddings or something like this). Can anyone suggest a better approach? Thanks