How to compute conditional probability in code? In this paper http://www.aclweb.org/anthology/D13-1176, in "2. Framework" it says

  We begin by describing the modelling framework underlying RCTMs. An RCTM estimates the probability $P(f \ | \ e)$ of a target sentence $f = f_1, ..., f_m$ being a translation of a source sentence $ e = e_1, ..., e_k$. Let us denote by $f_{i\ : \ j}$ the substring of words $ f_i, ..., f_j $. Using the following identity,

$$
P(f \ | \ e) = \prod_{i=1}^m P(f_i \ | \ f_{1:i−1}, e)
$$

  an RCTM estimates $P(f|e)$ by directly computing for each target position $i$ the conditional probability $P (f_i|f_{1:i-1}, e)$ of the target word $f_i$ occurring in the translation at position $i$, given the preceding target words $f_{1:i-1}$ and the source sentence $e$.

I understand the identity that they used, but how is it implemented in terms of code? How can a module calculate such a probability?

For this specific paper, the conditional distribution is calculated as follows:

$ P(f_i = v |f_{1:i-1} ) = \frac{\exp(o_{i,v})}{\sum_{v=1}^V\exp(o_{i,v})} $

on page 1702


why is SVM cost function the norm of the hyperplane parameters, and not the mean square error? working through Dr. Ng's coursera course
cost function intro:
basically, you want to minimize the difference squared between your guess and the actual value
cost function for SVM: this or this
he says the cost function is now the norm of the vector representing the hyperparameters: sqrt(theta_1^2 + theta_2^2... )
Why is that the case?

Given a linearly separable data set, the intuition behind the SVM-algorithm is to find a hyperplane that seperates both classes and maximizes the margin (which is twice the distance of the hyperplane to the closest data point).

So from all hyperplanes that seperate both classes, we want to find the one with the biggest margin (to achieve the best generalization).

Let's imagine we have two parallel seperating hyperplanes defined by the vector $\theta$ and with no data points in between, their distance is then given by: $$m= \frac{2}{\|\theta\|}$$
To maximize this distance, the two hyperplanes must pass through data points of the opposing classes.
In this case we can define another hyperplane in the middle (with same distance to both hyperplanes and the same vector $\theta$) that now has the margin $m$.

As you can see from the formula: the smaller $\|\theta\|$ the greater the margin $m$.

So maximizing the margin $m$ is equivalent to minimizing $\|\theta\|$. Thus, a separating hyperplane is optimal if it minimizes $\|\theta\|$.

This is just the rough intuition - if you want to go deeper into the maths behind it, I could recommend: http://www.svm-tutorial.com/2015/06/svm-understanding-math-part-3/


How to convert a non gaussian distribution into a gaussian destribution? Suppose I have a dataset inwhich there are few dimensions that distribution over them is non gaussian and this means, skewness is nonzero (possitive or negative). This is caused by some outliers in my data, which are not possible to be excluded manually.

Now my question is, How is it possible to convert this dataset into a gaussian (zero skewness) so that a learning procedure could be run on it without being biased to outliers? How should one behave this kind of datasets?

I recommend order normalization from R package bestNormalize, good info at:
https://cran.r-project.org/web/packages/bestNormalize/vignettes/bestNormalize.html

Below a short comparison of a few methods:

Best Normalizing transformation with 6283 Observations
Estimated Normality Statistics (Pearson P / df, lower => more normal):
 - No transform: 21.8352
 - Box-Cox: 4.0105
 - Log_b(x+a): 4.5358
 - sqrt(x+a): 7.1273
 - arcsinh(x): 4.5268
 - Yeo-Johnson: 4.0185
 - orderNorm: 0.9744

It is not written anywhere, but logic says, that you should rbind train and test datasets before this normalization if you want to use all of its advantages.


How do I build a permutation invariance neural network in keras? My question is about the structure of the network required to solve my problem with fewer data.

I have a sensor device that simply reports the color of the thing it's seeing in front of it. One sensor reports me 4 numbers: Red, Green, Blue, and Alpha.
The intensity of the color changes depends on the distance and the thing it's seeing.
I have 6 such sensors attached on each side of a small cube. The cube can be moved and rotated using the hand.

I want to predict the position of the cube in space in real-time.

My problem:

Input: 6 identical sensors, each giving 4 numbers. Total=6*4=24 numbers.

Output: 3 numbers, X, Y, Z, (Position of the cube)

I have my data ready with label XYZ.

Now, train a simple Multi-layer perceptron which takes 24 numbers and output 3 numbers.
This is working quite fine but it needs hell lots of data in a cubic meter space to predict accurately.

The problem is about the rotation. I need to rotate and cover 360 degrees for every location for it to predict well.

But I know for the fact that each sensor is identical so I want to share the weight of each sensor. I know that when you rotate the cube 90 degrees, it should not affect the output position at all. So this should mean the order of the sensors is not important. This means I should somehow use add or average to merge my sensors' layer. If I use concatenate it will preserve the order which make the output position changes.

The way I'm doing it is that I feed 4 numbers into a sensor model that are shared among all sensors, get the encoding, add them up, then connect it to Dense layers. Following is the model prototype:

from keras.layers import Input, Dense
from keras.models import Model, Sequential

sensor1 = Input(shape=(4,))
sensor2 = Input(shape=(4,))
sensor3 = Input(shape=(4,))
sensor4 = Input(shape=(4,))
sensor5 = Input(shape=(4,))
sensor6 = Input(shape=(4,))
sensor_model = Sequential([
    Dense(64, activation='relu'),
    Dense(64, activation='relu'),
])
sensor1_encoding = sensor_model(sensor1)
sensor2_encoding = sensor_model(sensor2)
sensor3_encoding = sensor_model(sensor3)
sensor4_encoding = sensor_model(sensor4)
sensor5_encoding = sensor_model(sensor5)
sensor6_encoding = sensor_model(sensor6)
sensor_encoding = average([
    sensor1_encoding,
    sensor2_encoding,
    sensor3_encoding,
    sensor4_encoding,
    sensor5_encoding,
    sensor6_encoding,
])
h = sensor_encoding
h = Dense(128, activation='relu')(h)
h = Dense(128, activation='relu')(h)
h = Dense(3, activation='linear')
model = Model(inputs=[sensor1, sensor2, sensor3, sensor4, sensor5, sensor6], outputs=[h])

Right now, when I change the average function to concatenate, the model loss is lower on both training and validation set which contradicts with my intuition. What is wrong with my thinking?
What do you think? How do I adjust this model in order for it to predict the same position if I rotate 90 degrees while also not suffering from 45 degrees rotation. And also make it in such a way that it does not remove the useful relationship between the inputs.

Concatenate was fine.

By concatenating, you give to your network access to all information it needs to predict the output. The order does not matter.

By averaging, you crunch your data into something smaller : something that represent your data in a less informative way. That's why your loose accuracy : you are giving to your network less information to learn from.

What you need to do is get more data. Your initial approach is fine, you just need enough data for your network to learn that when the data represent a 90 degree-rotation, the output should be the same.


Data Under fitting and its handling Regularization is used to handle over-fitting problem.

Similarly can we have some methodology to overcome under fitting problem or merely adding new features or training data will help us in reducing under-fitting issue?

As such we don't have much ways to handle under-fitting issue.
Generally I follow following ways to handle it:

Add more features to training data. That includes deriving new features front the existing one.
Make a complex model.
Increase iterations.

Adding new training data only, I believe will not solve this problem.


Best or recommended R package for logit and probit regression Could somebody please recommend a good R package for doing logit and probit regression? I have tried to find an answer by searching on Google but all the links I find go into lengthy explanations about what logit regression is, which I already know, but nobody seems to recommend an R package.

Thanks in advance.

Jerome Smith

Unless you have some very specific or exotic requirements, in order to perform logistic (logit and probit) regression analysis in R, you can use standard (built-in and loaded by default) stats package. In particular, you can use glm() function, as shown in the following nice tutorials from UCLA: logit in R tutorial and probit in R tutorial.

If you are interested in multinomial logistic regression, this UCLA tutorial might be helpful (you can use glm() or packages, such as glmnet or mlogit). For the above-mentioned very specific or exotic requirements, many other R packages are available, for example logistf (http://cran.r-project.org/web/packages/logistf) or elrm (http://cran.r-project.org/web/packages/elrm).

I also recommend another nice tutorial on GLMs from Princeton University (by Germán Rodríguez), which discusses some modeling aspects, not addressed in the UCLA materials, in particular updating models and model selection.


something wrong with my Alexnet I made Alexnet NN but it had brought out an error

input : image of 32*32*3

output : 10 classes

below is my Alexnet code

#alexnet
model_3=Sequential()
#Add the Dense layers along with activation and batch normalization
model_3.add(InputLayer(input_shape=(32,32,3)))
# 1st Convolutional Layer
model_3.add(Conv2D(filters=96, kernel_size=(11,11), strides=(4,4), padding='valid'))
model_3.add(Activation('relu'))
# Max Pooling
model_3.add(MaxPool2D(pool_size=(2,2), strides=(2,2), padding='valid'))

# 2nd Convolutional Layer
model_3.add(Conv2D(filters=256, kernel_size=(11,11), strides=(1,1), padding='valid'))
model_3.add(Activation('relu'))
# Max Pooling
model_3.add(MaxPool2D(pool_size=(2,2), strides=(2,2), padding='valid'))

# 3rd Convolutional Layer
model_3.add(Conv2D(filters=384, kernel_size=(3,3), strides=(1,1), padding='valid'))
model_3.add(Activation('relu'))

# 4th Convolutional Layer
model_3.add(Conv2D(filters=384, kernel_size=(3,3), strides=(1,1), padding='valid'))
model_3.add(Activation('relu'))

# 5th Convolutional Layer
model_3.add(Conv2D(filters=256, kernel_size=(3,3), strides=(1,1), padding='valid'))
model_3.add(Activation('relu'))
# Max Pooling
model_3.add(MaxPool2D(pool_size=(2,2), strides=(2,2), padding='valid'))

# Passing it to a Fully Connected layer
model_3.add(Flatten())
# 1st Fully Connected Layer
model_3.add(Dense(4096, input_shape=(224*224*3,)))
model_3.add(Activation('relu'))
# Add Dropout to prevent overfitting
model_3.add(Dropout(0.4))

# 2nd Fully Connected Layer
model_3.add(Dense(4096))
model_3.add(Activation('relu'))
# Add Dropout
model_3.add(Dropout(0.4))

# 3rd Fully Connected Layer
model_3.add(Dense(1000))
model_3.add(Activation('relu'))
# Add Dropout
model_3.add(Dropout(0.4))

# Output Layer
model_3.add(Dense(500))
model_3.add(Activation('relu'))
model_3.add(Dropout(0.2))
model_3.add(Dense(10))
model_3.add(Activation('softmax'))

the error message say

Negative dimension size caused by subtracting 11 from 3 for 'conv2d_22/convolution' (op: 'Conv2D') with input shapes: [?,3,3,96], [11,11,96,256].

it may concerned to layer size

what is the error of the above code and how to resolve it?

After you pass your input through your first convolutional layer it's size decreases from 32x32 to 6x6. This is then further decreased by the max pooling layer after that, which decreases the size from 6x6 to 3x3. It is then not possible to try and apply a second convolution with a kernal size of 11. You should therefore decrease your kernel size in the second layer (and probably also in the first convolutional layer).


Input contains NaN, infinity or a value too large for dtype('float32'). Without ruining Dataset I have a datasheet that is all in Binary but sometimes there are missing cases. In one of the inputs, there is a blank, I don't want to completely remove the whole column because some of the rows have values that maybe could be useful.

Example: Column 9 has all data except for Row 15 (random row) which is blank. How can I make my decision Tree without having to remove the whole column or make the data bad (By Averaging).

I'm not sure if averaging in the situation is will strengthen the data because it might correlate to a different output.

If NaNs are not too frequent you can try to drop the rows containing it.

Otherwise if your algorithm can't handle NaNs (i.e. lightgbm that is also a tree based algorithm can) you have to fill those values somehow. Usually it's a good idea to use a value that is out of the range of normal values (like -99 for a variable with only positive values) so tree can learn to treat those values differently.

In your case with binary (0/1) data it's a little different. I would put 0.5 so model can learn with a single cut to put missing values to either group (0 or 1) depending on which one makes more sense. Don't worry if all 3 groups are very different model should be also able to learn it given that you have enough data for the tree to make another cut.


T-DBSCAN - Implementing STOP logic I am attempting to implement the T-DBSCAN algorithm described in T-DBSCAN: A Spatiotemporal Density Clustering for GPS Trajectory Segmentation.  I have been able to implement most of the logic between the definitions (Page 3) and the pseudo-code (page 4), but I have not been able to implement the logic to determine if a cluster is a stop as described in Definition 9 of the paper (Page 3).  Specifically, I am having trouble understanding Definition 7 "Temporally Continuous", which is referenced in Definition 9:

  Definition 7.  Temporally continuous (TC). Let the min
  and max time stamps of a cluster $C$ ($\subseteq D$) $ \text{mint} $ and
  $ \text{maxt} $, respectively. $ C  $ is known as being “temporally
  continuous” if, for $ \forall{p_t \in D} $ and $ mint < t < maxt $,
   $ p_t \in C $.

where $ D $ is the trajectory, and $ p_t $ is a point at time $ t$.

As I read Definition 7, it sounds like it means that a cluster is temporally continuous if the points in the cluster are in the trajectory, and if all the timestamps of the point in the cluster is between the minimum time and maximum timestamp in the cluster, which doesn't make sense, because all points in a cluster will always be between the minimum and maximum timestamps in the same cluster, making all clusters temporally continuous.

Would someone be willing to point me in the right direction of how to interpret this section?  Definition 9 logic is also not included in the pseudo-code, so I only have the definition itself to go off of.

I've read through an existing implementation on Github which does not have this logic implemented either, which leads me to believe either this section is trivial and does not need to be implemented, or that I am not the only one who has had issues implementing this last section of the algorithm.

Thanks for any help that you can provide.  I'm willing to provide anymore information that I have that will be helpful.

Just to point out a minor confusion that there seems to be in the wording: there is mixed use of the the words temporarily and temporally. [OP has since corrected this]

We really only care about the final word, temporally i.e. relating to time, or the timestamps of the points in a cluster. Temporarily means only for a short period of time, which is a slightly different meaning to temporally' being related to time.

The way I understand the definition, a cluster is defined as a stop cluster if the minimum and maximum timestamps included that cluster ($mint$ and $maxt$)

Also, it seems definition 9 implies the usage of definition 7, so I would assume you need to implement it. If would be used a criterion as to whether a candidate cluster $C_i$ is defined as a stop cluster. So the final pseudo code would be something like:

stop_clusters = []    # empty list
for candidate in candidate_cluster:    # iterate over each stop cluster candidate
    TC = cluster_is_tc(cluster)        # check cluster is Temporally Continuous
    TO = cluster_is_not_to(cluster)    # check cluster is Temporally Overlapping

    if (TC & TO):                      # if both criteria are met...
        stop_clusters.append(cluster)  # assign to stop clusters

How you implement those two check functions would be quite straight forward, according to the definition, assuming you have the timestamps of each point, you are trivially checking that all points are contained in the time bounds. Just skimming page 3, I am afraid the paper doesn't make these time bounds completely clear to me, but I hope it is clear to you as you are implementing it.

The same is true for the temporally overlapping criterion: you check that there is no overlap of timestamps in two clusters - if there is overlap (i.e. the intersection is not an empty set), then you cannot call the current candidate cluster a stop cluster.

Edit:

Looking at that implementation you linked, it could be that that are performing the check Temporally Overlapping check at this if line for the case of being non-overlapping and this else line for the other case that they do overlap in time (assuming there index values idx are the timestamps); they are checking that the largest timestamp is less than the minimum timestamp of the previous cluster.

I can't see evidence of the check for temporal continuity of a cluster.


Words as features of a neural networks I'm new in Machine learning and I'm working on a problem related to text. I know that in ML we can use features as numerical values as input to neural network, but I don't know how to use features as words. In some papers I read that we take features to be n words with some property. I really don't understand how is that possible. Please, if it is not a problem, just to tell me some good papers or textbooks or links where it is explained how to do that.

You need to make a dictionary of words. It means you have to make a dictionary which you assign each word a unique value. then you can use one-hot-encoding to represent each word uniquely. If this is what you need it will do what you want. But this has a big problem. When you think about cats and dogs, you may find similarities and differences between them. This is because you have more knowledge than the only representation of words in your brain. Consequently, you should use approaches to assign a unique number to each word, and put near concepts as neighbors. For the first part take a look at here and the second part, take a look at here.


Difference between output of probabilistic and ordinary least squares regressions If I execute the commands

my_reg = LinearRegression()
lin.reg.fit(X,Y)

I train my model. To my understanding training a model is calculating coefficient estimators.

I do not really understand the difference between this and e.g.

scipy.stats.linregress(X,Y)

calculating a 'normal' regression that also gives me the coefficient estimators and all the other statistics connected with it.

Could anyone tell me what is the difference here?

They both solve the exact same objective, which is minimizing the mean squared error. However the second method can answer "how confident it is that slope is not zero, i.e. $Y$ is correlated with $X$?" via p-value.

In detail

Lets denote the data as $(X, Y) = \{(x_n, y_n)|x_n \in \mathbb{R}^D, y_n \in \mathbb{R}\}$. And the regression as $\hat{y} = Ax+B$.

Extra quantities returned by scipy.stats.linregress(X,Y) are: rvalue ($r$), and pvalue ($p$).

In statistics, $r^2$ (known as r-squared) measures the "goodness-of-fit" . That is, as regression $\hat{y}=Ax+B$ gets closer to observation $y$, $r^2$ gets closer to $1$. Since it is a function of $y$ and $\hat{y}$, it can be calculated for the first method too. So no difference here.

However, $p$ is specific to second method. scipy.stats.linregress(X,Y) adds a normality assumption to noise, i.e. assumes $\epsilon \sim  N(0, \sigma^2)$ where $$\epsilon = y - \overbrace{Ax+B}^{\hat{y}}$$
On the basis of this assumption, it can answer an additional question: "how confident it is that the slope is not zero?". The first method cannot answer this question.

For example, suppose the estimated slope is $2.1$ for both methods, we still cannot tell whether this slope is significant or $Y$ is actually independent of $X$. Unless we look at the value of $p$. For example, for $p < 0.01$ we are confident (at significance level $0.01$) that $Y$ is correlated with $X$, but for $p > 0.1$ we cannot be confident, i.e. slope $2.1$ could be due to chance and $Y$ might be independent of $X$.

This link gives more details on how p-value is actually calculated in second method.


Doing a fine tuning after a transfer learning I read about fine tuning and transfer learning for CNNs and was wondering if we can do fine tuning after using transfer learning on the same CNN? If so, will this increase the performance of the model or decrease it?

Generally, it will increase the performance of the model, but you'll have to experiment with it and check for your case. Try it, till you get a better result. If after many trials also you are not able to increase the performance of the model then it means that you are limited by the architecture of the pre-trained model.
Refer to this http://cs231n.github.io/transfer-learning/ for more details.


How YOLO training and prediction works for an object fall in multiple grid? So far what I understood about YOLO, it expects training image should be divided in to fixed grid, where each grid has Label like P(object present or not), object bounding box, object classes. Similarly it will return the same output for each image prediction.

If it's correct, I'm not able to map those images for both cases training and prediction where some objects  are the part of multiple grids.
During training we provide bounding box information corresponding to particular(single) grid only, how it clubbed the bounding box info of multiple grids?

Note: Non-max suppression is again confusing, if it is related with it.

It seems after referring many documents, I found the answer of my question.

  First, likely to correct my understanding. I thought for labeling,
  bounding box size (width, height) will always be with in the range of
  particular grid dimension i.e. between 0 and 1. It's not correct and
  this is the only source of confusion atleast for me.

I believe that if I'm able to answer about the labeling process, I think it will clear most of the doubt raised in the form of question.

Steps are labeling:

Identify the center point of the image with respect to image boundary, refer RED dot.

Create a grid on top of the image.

Identify the grid where center point exist. In this case its 8th grid (Numbering start from 1).
Marked only that grid has an object(not others) and labeling should be something like this:

g8 is having object and bounding box information. bx and by are wrt to grid dimension, whereas width and height are wrt to its size, which is greater than one.

During YOLO model training, it learned this intelligence and can start predicting the height and width which can be greater than 1.
In a case if a car( or any object) falls in multiple gird and model training is perfect it should predict the height and width accordingly i.e > 1.


Can someone explain the following error in my python code? I am analyzing a dataset in Python for strictly learning purpose.
In the code below that I wrote, I am getting some errors which I cannot get rid off. Here is the code first:

plt.plot(decade_mean.index, decade_mean.values, 'o-',color='r',lw=3,label = 'Decade Average')
plt.scatter(movieDF.year, movieDF.rating, color='k', alpha = 0.3, lw=2)
plt.xlabel('Year')
plt.ylabel('Rating')
remove_border()

I am getting the following errors:

1. TypeError: 'str' object is not callable
2. NameError: name 'remove_border' is not defined

Also, the label='Decade Average' is not showing up in the plot.

What confuses me most is the fact that in a separate  code snippet for plots (see below), I didn't get the 1st error above, although remove_border was still a problem.

plt.hist(movieDF.rating, bins = 5, color = 'blue', alpha = 0.3)
plt.xlabel('Rating')

Any explanations of all or some of the errors would be greatly appreciated. Thanks

Following the comments, I am posting the data and the traceback below:
decade_mean is given below.

year
1970    8.925000
1980    8.650000
1990    8.615789
2000    8.378947
2010    8.233333
Name: rating, dtype: float64

traceback:

TypeError                                 Traceback (most recent call last)
<ipython-input-361-a6efc7e46c45> in <module>()
      1 plt.plot(decade_mean.index, decade_mean.values, 'o-',color='r',lw=3,label = 'Decade Average')
      2 plt.scatter(movieDF.year, movieDF.rating, color='k', alpha = 0.3, lw=2)
----> 3 plt.xlabel('Year')
      4 plt.ylabel('Rating')
      5 remove_border()

TypeError: 'str' object is not callable

I have solved remove_border problem. It was a stupid mistake I made. But I couldn't figure out the problem with the 'str'.

Seems that remove border is not defined. You have to define the function before used.

I do not know where the string error comes, is not clear to me. If you post the full traceback it will be clearer.

Finally your label is not show because you have to call the method plt.legend()


Better approach to assign values to determine potential fake sentences I am trying to assign different values for each sentences based on information about the presence of hashtags, upper case letters/words (e.g. HATE) and some others.
I created a data frame which includes some binary values (1 or 0):
Sentence           Upper case   Hashtags

I HATE migrants       1             0
I like cooking        0             0
#trump said he is ok  0             1
#blacklives SUPPORT   1             1

I would like to assign a value based on the binary values above, if they are satisfied or not, for example:
- if Upper case = 1 and Hashtags = 1 then assign -10;
- if Upper case = 1 and Hashtags = 0 then assign -5;
- if Upper case = 0 and Hashtags = 1 then assign -5;
- if Upper case = 0 and Hashtags = 0 then assign 0;

This would be ok for a small number of requests and combinations, but with three variables to check, it would be a greater number of combination to consider manually!
Do you know if there is a way to take into account all these in an easy (and feasible) way?
Someone told me about using regression, but I have never used it before for similar task. The context is about fake tweets.

I'll suggest to test the sentence or the tweet for polarity. This can be done using the textblob library. It can be installed as pip install -U textblob. Once the text data polarity is found, it can be assigned as a separate column in the dataframe. Subsequently, the sentence polarity can then be used for further analysis.
Polarity and Subjectivity are defined as;

Polarity is a float value within the range [-1.0 to 1.0] where 0
indicates neutral, +1 indicates a very positive sentiment and -1
represents a very negative sentiment.
Subjectivity is a float value within the range [0.0 to 1.0] where 0.0
is very objective and 1.0 is very subjective. Subjective sentence
expresses some personal feelings, views, beliefs, opinions,
allegations, desires, beliefs, suspicions, and speculations where as
Objective sentences are factual.

Data
import pandas as pd

# create a dictionary
data = {"Date":["1/1/2020","2/1/2020","3/2/2020","4/2/2020","5/2/2020"],
    "ID":[1,2,3,4,5],
    "Tweet":["I Hate Migrants",
             "#trump said he is ok", "the sky is blue",
             "the weather is bad","i love apples"]}
# convert data to dataframe
df = pd.DataFrame(data)

Notice, the sentiment column is a tuple. So we can split it into two columns like, df1=pd.DataFrame(df['sentiment'].tolist(), index= df.index). Now, we can create a new dataframe to which I'll append the split columns as shown;
df_new = df
df_new['polarity'] = df1['polarity']
df_new.polarity = df1.polarity.astype(float)
df_new['subjectivity'] = df1['subjectivity']
df_new.subjectivity = df1.polarity.astype(float)

Next, on basis of the sentence polarity found earlier, we can now add a label to the dataframe, which will indicate if the tweet/sentence is fake, not fake or neutral.
import numpy as np
conditionList = [
    df_new['polarity'] == 0,
    df_new['polarity'] > 0,
    df_new['polarity'] < 0]
choiceList = ['neutral', 'positive', 'negative']
df_new['label'] = np.select(conditionList, choiceList, default='no_label')
print(df_new)

The result will look like this;
Result
        Date  ID                 Tweet    sentiment  polarity  subjectivity  label
0  1/10/2020   1       I Hate Migrants    (-0.8, 0.9)  -0.8      -0.8        fake
1  2/10/2020   2  #trump said he is ok    (0.5, 0.5)    0.5       0.5        not_fake
2  3/10/2020   3       the sky is blue    (0.0, 0.1)    0.0       0.0        neutral
3  4/10/2020   4    the weather is bad    (-0.68, 0.66) -0.7      -0.7       fake
4  5/10/2020   5         i love apples    (0.5, 0.6)    0.5       0.5        not_fake

Complete Code
import pandas as pd
import numpy as np
from textblob import TextBlob
data = {"Date":["1/10/2020","2/10/2020","3/10/2020","4/10/2020","5/10/2020"],
        "ID":[1,2,3,4,5],
        "Tweet":["I Hate Migrants",
                 "#trump said he is ok", "the sky is blue",
                 "the weather is bad","i love apples"]}
    # convert data to dataframe
df = pd.DataFrame(data)
# print(df)
df['sentiment'] = df['Tweet'].apply(lambda Tweet: TextBlob(Tweet).sentiment)
# print(df)

# split the sentiment column into two
df1=pd.DataFrame(df['sentiment'].tolist(), index= df.index)

# append cols to original dataframe
df_new = df
df_new['polarity'] = df1['polarity']
df_new.polarity = df1.polarity.astype(float)
df_new['subjectivity'] = df1['subjectivity']
df_new.subjectivity = df1.polarity.astype(float)
# print(df_new)

# add label to dataframe based on condition
conditionList = [
    df_new['polarity'] == 0,
    df_new['polarity'] > 0,
    df_new['polarity'] < 0]
choiceList = ['neutral', 'not_fake', 'fake']
df_new['label'] = np.select(conditionList, choiceList, default='no_label')
print(df_new)


Final layer of neural network responsible for overfitting I am using a multi-layer perceptron with 2 hidden layers to solve a binary classification task on a noisy timeseries dataset with a class imbalance of 80/20. I have 30 million rows and 500 features in the training set. The dataset is structured, ie, not images. My original features were highly right skewed; I do my best to transform these into nicer distributions by either taking logs or categorising some of them. I use an architecture of 512->128->128->1, with relu activations in every layer except the last. My loss function is sigmoid cross entropy.

The validation set contains 10 million rows. Initially the validation error goes down, but then starts to go up again after a couple of epochs. On analysing the gradients and weights of each layer, I see that the overfitting coincides with the weights on the final layer only getting larger and larger. The final layer seems to go into overdrive while the rest of the network seems to do very little learning.

I can solve the overfitting problem by using l2 regularisation, but this hurts the validation error. I have yet to find a beta regularisation parameter which doesn't hurt the best validation error I've seen. Dropout makes things even worse.

Granted, the classification problem is very difficult, with probably a very weak signal, but I find that gradient boosted trees are able to generalize much better than a simple, say, 64x64 multi-layer perceptron (the log loss on the training set is the same for both network and gradient boosted tree).

Are there any words of wisdom on how to make this network generalize better given that I've already tried:

dropout of varying degrees
l1/l2/group lasso regularization
adding noise to inputs
adding noise to gradients and weights
feature-engineering so as to remove/re-represent highly skewed features
batch normalization
using a lower learning rate on the final layer
simply using a smaller network (this is the best solution I've found)

to some or all layers. All methods hurt the validation error so much that the performance is nowhere near how well the tree model does. I would have given up by now were it not for the fact that the tree model is able to do so much better out of sample, but the training log loss for both is the same.

Are you balancing your training dataset 50/50? It seems to me that you are making a lot of changes to account for 80/20 balance when it sounds like you have a deep enough dataset to balance your training to be 50/50 and still have a reasonable size for your training set. How about keeping it to a smaller network - as you and some notes above suggest - and with a 50/50 set on a smaller training sample?


Knn distance plot for determining eps of DBSCAN I would like to use the knn distance plot to be able to figure out which eps value should I choose for the DBSCAN algorithm.
Based on this page:

  The idea is to calculate, the average of the distances of every point
  to its k nearest neighbors.  The value of k will be specified by the
  user and corresponds to MinPts. Next, these k-distances are plotted in
  an ascending order. The aim is to determine the “knee”, which
  corresponds to the optimal eps parameter.

Using python with numpy/sklearn, I have the following points, with the following distance for 6-knn:

X = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])
nbrs = NearestNeighbors(n_neighbors=len(X)).fit(X)
distances, indices = nbrs.kneighbors(X)

# Indices

[[0 1 2 3 4 5]
 [1 0 2 3 4 5]
 [2 1 0 3 4 5]
 [3 4 5 0 1 2]
 [4 3 5 0 1 2]
 [5 4 3 0 1 2]]

# Distances
[[ 0.          1.          2.23606798  2.82842712  3.60555128  5.        ]
[ 0.          1.          1.41421356  3.60555128  4.47213595  5.83095189]
[ 0.          1.41421356  2.23606798  5.          5.83095189  7.21110255]
[ 0.          1.          2.23606798  2.82842712  3.60555128  5.        ]
[ 0.          1.          1.41421356  3.60555128  4.47213595  5.83095189]
[ 0.          1.41421356  2.23606798  5.          5.83095189  7.21110255]]

then I computed the average distance:

distances.mean()
2.9269575028354495

The problem is I don't understand how exactly could I represent the same plot as them with distances in y-axis and number of points according to the distances on the x-axis using python.

Thank for your help.

You

take the last column of that matrix
sort descending
plot index, distance
hope to see a knee (if the distance does not work well. there might be none)


Finding optimal weights for models I'm trying to implement an algorithm to find the minimal value of a function.

Before moving to sigmoid activation functions, i'm trying to understand linear regression.

Usually, a gradient descent algorithm is used to find an minimal value where the algorithm converges, but there are some other ways for linear models.

Say I have two vectors:

x=[1,2,3,4,5,6,7,8,9,10,11,12]

y=[2.3,2.33,2.29,2.3,2.36,2.4,2.46,2.5,2.48,2.43,2.38,2.35]

Between these points, I would like to add a linear separator with least squares.

Say I have some imperfect linear function:

$f(x)=0.026x+2.3$

As I know, there are two ways to find this:

$w = (X^TX)^{-1}X^Ty$

and a gradient descent algorithm:

$w^{new} = w^{old} - \nu \frac{dy}{dx}$

Although for linear models finding derivative is trivial, thus second method is not necessary.

Now i've used the first equation on the vectors, in Python:

w = ((np.transpose(x)*x)**-1)*np.transpose(x)*y

Unfortunately, the output was irrelevant:

[ 2.3, 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ]

Then i've tried using the second method for 500 iterations, in Python:

for i in range(1,5000):
    x_old = x_new
    x_new = x_old - v*dydx
    print("x_new = {0} - {1}({2}) = {3}".format(x_old, v, dydx, x_new))

However, i'm not sure how to know when it reaches a convergence point.

How can I use these methods properly for linear models? And if so, how can they be used for more complex models such as logistic regression?

Also you have misunderstood the -1..
It's not the exponent rather the symbol of inverse of a Matrix

Also use .T for Transpose...(a bit Pythonic Convenience)

Just to help you out,
Search np.linalg.inv

For your second Query,

Refer here

Just to add a little, you stop when you see that your loss isn't improving any more as such or is improving at the 5th decimal place..

Hope this helps..


Model for classifying time-series data with distinct features? I've heard about time-series classification being done with TCN's and CNN's combined with LSTM's very often, citing that CNN's would provide insight both forward and in the past since you already have all the information for that time period. For my application, there is a distinct shape and I'd like to classify whether it exists or not. For example, I want to detect whether the data looks like this  or this
Of course, there would be noise involved and the feature would be much less obvious making the problem worthy of using machine learning. Is there some way I can exploit this knowledge of there being a single important feature (this hump) to use a different architecture or do anything differently?

This specific problem looks at the pattern across the whole data I.e. pattern will not show up from time < -3 or time > 3 for a given curvature.

You can try two models :

Simple feed-forward Network with number of inputs = number of time steps (Maybe scale / shift the data so that it always has the same number of time steps )

This should be able to detect some patterns for classification (Like f(0) must be less that f(4))

Univariate LSTM with different sizes of time steps in each sample

This should be able to learn that f(x) should stay near constant, reduce and then increase and return to constant

Both networks will have a sigmoid in output layer since it is a binary classification problem.

Code exmaple for LSTM : https://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/


How do I start Orange I installed Orange via conda install orange3

I see the package installed in Anaconda3 / pkgs directory however there is not desktop icon nor can I find Orange via Window start.

Firstly, I am assuming that you are using Anaconda Navigator, and are trying to use Orange Canvas (the GUI method)

If the package has been installed correctly (from within Anaconda, using the Environments tab) you will see the following when opening Anaconda Navigator:

So, there is not a Windows desktop or start-menu icon; instead launch Orange from Anaconda Navigator.

If you have instead only installed the biolabas Orange python library, 'Orange3', you will only be able to access the functions via a python programming shell by calling:

  import Orange3


What are the cases where it is fine to initialize all weights to zero I've taken a few online courses in machine learning, and in general, the advice has been to choose random weights for a neural network to ensure that your neurons don't all learn the same thing, breaking symmetry.

However, there were other cases where I saw people initializing using zero weights. Unfortunately, I can't remember what those were. I think it might have been non-neural-network cases, like a simple linear or logistic regression model (simple weights only on the inputs, leading directly to an output).

Are those cases safe for zero initialization? Alternatively, could we use random initialization in those cases too, just to stay consistent?

Whenever you have a convex cost function you are allowed to initialize your weights to zeros. The cost function of logistic regression and linear regression have convex cost function if you use MSE for, also RSS, linear regression and cross-entropy for logistic regression. The main idea is that for convex cost function you'll have just a single optimal point and it does not matter where you start, the starting point just changes the number of epochs to reach to that optimal point whilst for neural networks the cost function does not have just one optimal point. Take a look at here. About random initialization, you have to consider that you are not allowed to choose random weights which are too small or too big although the former was a more significant problem. If you choose random small weights you may have vanishing gradient problem which may lead to a network that does not learn. Consequently, you have to use standard initialization methods like He or Glorot, take a look at here and Understanding the difficulty of training deep feedforward neural networks.

Also, take a look at the following question.

How should the bias be initialized and regularized
When to use (He or Glorot) normal initialization over uniform init? And what are its effects with Batch Normalization


Bag-of-words and Spam classifiers I implemented a spam classifier using Bernoulli Naive Bayes, Logistic Regression, and SVM. Algorithms are trained on the entire Enron spam emails dataset using the Bag-of-words (BoW) approach. Prediction is done on the UCI SMS Spam Collection dataset. I have 3 questions:

During test time, while creating the term-frequency matrix, what if none of the words from my training BoW are found in some of my test emails/smses. Then, wouldn't the document vectors be zero vectors for those datapoints. How should I tackle this?

What if a new word from my test email/sms doesn't exist in BoW?

How do I choose my BoW so as to improve my prediction accuracy?

Supervised ML works on the assumption that the test data follows the same distribution as the training data. This is not always the case in real-world use cases, but it's at least necessary that the test data is "mostly similar" to the training data.
As a consequence, a BoW model can only be applied to data which uses mostly the same vocabulary, with a mostly similar distribution over the words. It is true that out of vocabulary (OOV) words frequently appear in the test data, because words in natural languages follow a Zipf distribution so there are many words which occur rarely. The general assumption in BoW ML models is that since these words occur rarely, they can reasonably be ignored.

During test time, while creating the term-frequency matrix, what if none of the words from my training BoW are found in some of my test emails/smses. Then, wouldn't the document vectors be zero vectors for those datapoints. How should I tackle this?

This event is supposed to be unlikely: a sentence usually contains at least a few common words (again due to the Zipf distribution). However this could happen with a very short text message. Anyway there is nothing special to do about it: all the words are ignored, the vector contains only zeros indeed, the model gives a prediction for this vector of features.

What if a new word from my test email/sms doesn't exist in BoW?

This is the traditional case of OOV words mentioned above. The most simple (and probably most common) option is to completely ignore the unknown word. With some probabilistic models smoothing can be used to account for the existence of OOV words, but as far as I know this is used only with n-grams.

How do I choose my BoW so as to improve my prediction accuracy?

Experimentally: use a validation set in order to evaluate several methods, then select the one which performs best and apply only this one on the test set.


Is there any machine learning algorithm that can solve this problem? I have a data set of 100000 samples with binary output.

I would like to study the impact of Col_A (a continuous feature) on the output result.

Col_A has values from 0 to 7000000 and when I add this feature to my classifier it gives bad accuracy. I have tried z-score and scalar number but it doesn't change a thing.

I would like to try another method to solve this problem.

I made some plot visualization to my data and I found that there is some range in which the majority of the outputs are negatives. ie. when col_A is ranging between (0 - 200), between (2500 - 2800) and between (5200-5400) the majority of outputs are negatives.

I would like to create a new feature to specify which category my Col_A is in, instead of using Col_A.

PS: I can't use if--else rules because I don't know where to delimit these ranges, I have just analyzed the previous values from my plot. but I need to have a dynamic method for getting those categories.

Is there any type of clustering that can solve this problem?

Does neural network help for this case?

You want a supervised approach. Clustering will not care about your target variable and perform arbitrary splits that don't help.

Likely a decision tree can be helpful here, if you use a good implementation that can split the same feature multiple times (to break it into intervals). There are other approaches you could try, such as piecewise regression etc.


XGBClassifier error! ValueError: feature_names mismatch: I have a data set given as follows:
target shape (200000, 1)
train_data.shape (200000, 48)
test_data.shape(100000, 48)
I had used the data to predict_proba using RandomForestClassifier ExtraTreesClassifier RandomForestClassifier AdaBoostClassifier GradientBoostingClassifier and SVC
and got results without an error.
BUT
clf = XGBClassifier() is showing following errors (with the same code except clf):
ValueError: feature_names mismatch:

The problem occurs due to DMatrix..num_col() only returning the amount of non-zero columns in a sparse matrix. Hence, if both train & test data have the same amount of non-zero columns, everything works fine. Otherwise, you end up with different feature names lists. There're currently three solutions to work around this problem:

realign the columns names of the train dataframe and test dataframe using:

test_df = test_df[train_df.columns]

save the model first and then load the model
change the test data into array before feeding into the model, ie: use,

test_df.values instead of test_df


How to improve a regression prediction model for data with parameters which are out of sample range? I have been looking for some methods to improve my prediction model, but could not find any so far. I have data, including 10 numerical features, that I use for prediction. I used a random forest regression model and it works perfectly on the train set (some part of perfection comes from over fitting).

However, the model is not doing a good job for data points that some of their features are not within the range of the sample data. For example, parameter x is between 0 and 2000 in the sample but the new data point that we want to predict has a value of 3000 for x.

I know this is due to data limitation, but I wonder if there is any way that I can generate samples or improve the prediction?

I wanted to build a simple model excluding this parameter, but the problem is that this is the most significant parameter in my model.

Any hints are appreciated.

I have encountered case in which potent tree learners acted like Nearest Neighbor variants: They would learn to divide the search space so that only examples remained that where close on some meaningful axis (in my case lat and long :)). This could still generalize to examples in the test set that shared those features but not to genuinely unseen examples. Or to put it in another way: The method is potent enough to find rules that work well, but do not generalize in the way you want it to (aka over fitting, but the maybe in a specific way). What helped me were two things: First of all I tested this characteristic of the domain by just using KNN,only feeding it subsets of the feature space (and sure enough KNN over lat, lng worked like a charm). This helped me understand that examples that were in the same space time coords shared certain history, so the second action was to eliminate those examples in the train space and evaluate on an a set of examples totally outside of the coordinates (space and time) of the train set. It is hard to say where this behavior could lie in your case without having a peek at the code, but perhaps my anecdote helps.


Is GNU Octave a perfect place to code neural networks GNU Octave is used for its simplicity and compiling speed to write numerical algorithms (such as eg machine learning problems), but I wanted to know if I can also use it for faster coding of neural networks.

No.

The key to implementing neural networks easily and fast is having automatic differentiation. This is the key feature that allows programs using Python libraries like Pytorch or Tensorflow (and Keras), to implement only the forward pass of a neural network and have the backward pass computed automatically.

Octave does not have automatic differentiation. While Matlab's deep learning toolbox seems to include automatic differentiation, I think it is not included with Octave. And even if it were, the deep learning community is mainly focused on the Python ecosystem, and the rest of the environments like R or Matlab are paid almost no attention in comparison.


Specifying class or sample weights in Keras for one-hot encoded labels in a TF Dataset I am trying to train an image classifier on an unbalanced training set. In order to cope with the class imbalance, I want either to weight the classes or the individual samples. Weighting the classes does not seem to work. And somehow for my setup I was not able to find a way to specify the samples weights. Below you can read how I load and encode the training data and the two approaches that I tried.
Training data loading and encoding
My training data is stored in a directory structure where each image is place in the subfolder corresponding to its class (I have 32 classes in total). Since the training data is too big too all load at once into memory I make use of image_dataset_from_directory and by that describe the data in a TF Dataset:
train_ds = keras.preprocessing.image_dataset_from_directory (training_data_dir,
                                                             batch_size=batch_size,
                                                             image_size=img_size,
                                                             label_mode='categorical')

I use label_mode 'categorical', so that the labels are described as a one-hot encoded vector.
I then prefetch the data:
train_ds = train_ds.prefetch(buffer_size=buffer_size)

Approach 1: specifying class weights
In this approach I try to specify the class weights of the classes via the class_weight argument of fit:
model.fit(
    train_ds, epochs=epochs, callbacks=callbacks, validation_data=val_ds,
    class_weight=class_weights
)

For each class we compute weight which are inversely proportional to the number of training samples for that class. This is done as follows (this is done before the train_ds.prefetch() call described above):
class_num_training_samples = {}
for f in train_ds.file_paths:
    class_name = f.split('/')[-2]
    if class_name in class_num_training_samples:
        class_num_training_samples[class_name] += 1
    else:
        class_num_training_samples[class_name] = 1
max_class_samples = max(class_num_training_samples.values())
class_weights = {}
for i in range(0, len(train_ds.class_names)):
    class_weights[i] = max_class_samples/class_num_training_samples[train_ds.class_names[i]]

What I am not sure about is whether this solution works, because the keras documentation does not specify the keys for the class_weights dictionary in case the labels are one-hot encoded.
I tried training the network this way but found out that the weights did not have a real influence on the resulting network: when I looked at the distribution of predicted classes for each individual class then I could recognize the distribution of the overall training set, where for each class the prediction of the dominant classes is most likely.
Running the same training without any class weight specified led to similar results.
So I suspect that the weights don't seem to have an influence in my case.
Is this because specifying class weights does not work for one-hot encoded labels, or is this because I am probably doing something else wrong (in the code I did not show here)?
Approach 2: specifying sample weight
As an attempt to come up with a different (in my opinion less elegant) solution I wanted to specify the individual sample weights via the sample_weight argument of the fit method. However from the documentation I find:

[...] This argument is not supported when x is a dataset, generator, or keras.utils.Sequence instance, instead provide the sample_weights as the third element of x.

Which is indeed the case in my setup where train_ds is a dataset. Now I really having trouble finding documentation from which I can derive how I can modify train_ds, such that it has a third element with the weight. I thought using the map method of a dataset can be useful, but the solution I came up with is apparently not valid:
train_ds = train_ds.map(lambda img, label: (img, label, class_weights[np.argmax(label)]))

Does anyone have a solution that may work in combination with a dataset loaded by image_dataset_from_directory?

...Because the Keras documentation does not specify the keys for the class_weights..

You may get an idea with these two parameters,

labels: Either "inferred" (labels are generated from the directory structure), or a list/tuple of integer labels of the same size as the number of image files found in the directory. Labels should be sorted according to the alphanumeric order of the image file paths (obtained via os.walk(directory) in Python).

class_names: Only valid if "labels" is "inferred". This is the explicit list of class names (must match names of subdirectories). Used to control the order of the classes (otherwise alphanumerical order is used).

So, it will be mapped to alphabetical order if not specified explicitly.
traindata = image_dataset_from_directory("/content/Data") # Default
traindata = image_dataset_from_directory("/content/Data",class_names=['train_data','test_data']) #as per the values passed in class_names

Does anyone have a solution that may work in combination with a dataset loaded by image_dataset_from_directory?

You may try a wrapper of a custom Generator over the initial Keras generator. Below is a very basic code sample
def cust_gen():
    for image, label in traindata:
        yield image, label, np.ones((2,1)) # 3rd parm is the sample_weight

history = model.fit(cust_gen(), epochs=epochs)

You will have to write a custom code to create the sample_weight for each batch.


Which text clustering algorithms should I use to group free text fields in Python? I have a data set in which users have input free text (think "APPLE", "Apple", "apple", "appl") and I'm trying to group the input by response type (e.g. organize all of the preceding examples by "Apple"). What's the best way to do this in Python?

That is called "entity resolution" or "record linkage". It is a very hard problem.

If it all possible, I would try to add "type ahead" to the form. That would encourage users to select already used categories.

fuzzywuzzy is a Python package for approximate string matching.

dedupe is "a Python library that uses machine learning to perform fuzzy matching, deduplication, and entity resolution quickly on structured data."


.1 after column name while read csv file from pandas I am getting .1 after column name while read csv  file from pandas

I rechecked and reloaded the data. I manually removed duplicate columns which solved the problem. But yes , it was showing .1 with non duplicate columns as well


How can I use different loss functions for images with different labels in tensorflow? I have two types of images, meaning two different sets of possible labels. How can I use two different loss functions based on which image I am currently training on with a batch size of 1?
Thanks !

This is not possible, for 2 reasons:

It doesn't make mathematical sense. The Gradient must explore one loss function that is optimized to distinguish between more than one class. Using a loss function to classify a single class is equivalent to not learning anything.
In ML, the models you train are meant to generalize on data they have never seen before. Because of this, you can't use different loss functions because once you test its performance on test data nobody knows to what class they belong.


Implementation of Tree Kernels in Python I'm looking to use a tree kernel (e.g. Zelenko et al., 2003, Moschitti 2006, etc.) for text classification in Python. However, I'm still a little unclear on the implementation. Is there an existing implementation (in any language) that I could use as a reference, or is there a 3rd-party tree kernel library I could use?

I assume that the ones from here are Moschitti's version, as the author is associated with Moschitti in some papers. I'm unable to confirm, though.


matrix factorization? I have sparsely populated matrix of users as rows  with columns being categorical answers to various questions ( question are of various domain about  preferences / behaviors of the users ) . answers may be be either numerical ( for example an answer to the question "what's the number of children in your household ? " ), or categorical   ( "specify your education level  ? BS / PHd  / etc."). As mentioned the matrix is sparse , and the aim is to infer the missing entries.

do you think matrix factorization techniques ( for example  ALS ) could be suitable for solving this ? ( with proper normalization of the response ) and do you suggest another learning algorithm ?

I think ALS is very applicable here. I would imagine it would need to be fine tuned quite a bit though. Unless the questions are correlated, it might yield some weird results. Just because two people with PHD's have one child, does not mean the third would only have one child. Basically, the more the data the better. Recommending movies works so well because all the columns are related to movies, but this should work with enough data.


which outlier detection technique? I'm new to data science. I have a question on anomaly detection techniques.

There are several anomaly detection techniques such as statistical, density based, depth based, clustering, etc..

Given a dataset, what are the criteria or how should I choose which one of the techniques above (not the algorithms inside the techniques). In general why choose a particular technique.

You first need to specify if you are going to work on anomaly detection or novelty detection.

The difference between them is that in anomaly detection, your data contains anomalies, but in novelty detection, your data contains only valid data.

After that you need to choose the algorithms depending on what you have chosen before, there are some algorithms that work only with one type, and there are algorithms that work with both types.


Handling Imbalanced Datasets I work in the medical domain, so class imbalance is the rule and not the exception. While I know Python has packages for class imbalance, I don't see an option in Orange for e.g. a SMOTE widget. I have read other threads in Stack Exchange regarding this, but I have not found an answer to how to tackle class imbalance in Orange without resorting to Python programming. Thanks

You can add class_weights with dictionary containing class weights, e.g.:
class_weight = {0: 1., 1: 20.}

While SMOTE can be used to synthesize new examples for the minority class (the process is called oversampling) in order to get equal weights.
For Orange please check this link.
Please, provide more information so that we can help you.


Error while using decompose and stl functions in R I am new to time series analysis and I am trying out some codes seen in some tutorials. I am using stock prices as the time series data.

library(quantmod)
aapl=getSymbols("AAPL",from="2015-01-01",auto.assign=F)
is.ts(aapl)
ts.aapl=as.ts(aapl)
is.ts(ts.aapl)
x=Cl(ts.aapl)
head(x)
plot(decompose(s))
plot(stl(x))

But its showing these errors

> plot(decompose(x))
Error in decompose(x) : time series has no or less than 2 periods
> plot(stl(x))
Error in stl(x) : series is not periodic or has less than two periods

is the error is due to its lack of seasonality and trend?

The error is likely due to the fact that your x is an irregular time series: there are temporal gaps, e.g. no data on 2015-01-03, 2015-01-04. You will have to implement your own version of decompose, e.g. following this approach:

https://stackoverflow.com/questions/12623027/how-to-analyse-irregular-time-series-in-r


Truncating float/doubles for reproducibility I deploy machine learning models (typically GPU) to a variety of environments. I work sort of at the edge of ML R&D and devops, so I am really big into reproducibility, and one thing that drives me nuts is when models output similar but not byte-for-byte identical values, frustrating any hash-based automated testing. For example, here is a score from the same sample, inference model, code, container image, etc, but one is on a Titan, one is on an RTX 2080.

Titan X  = 0.9887396097183228
RTX 2080 = 0.9887396693229675

That's a relative error of 6.0e-08 or about 60 parts-per-billion. Obviously, this is the "same number", well within the weirdness of IEEE 754 and GPU processing.

I'd like to truncate the output of my algos, as this simplifies the job of automated testing, but the question then becomes, how to round to achieve this, and how much precision? Sounds simple, but take for example this note on numpy.around

  Results may also be surprising due to the inexact representation of
  decimal fractions in the IEEE floating point standard [R9] and errors
  introduced when scaling by powers of ten.

One part-per-million (for typically 0.0-1.0 confidence values) seems reasonable but I'm not sure if there are any subtle gotchas with this approach.

I'm not expert in this but as far as I know the proper way to test for equality modulo floating point imprecision is to compare the differences of the two values, i.e. instead of:

trunc(a) == trunc(b)

one would do:

abs(a-b) <= epsilon

where epsilon is the constant which represents an acceptable difference, e.g. $10^{-6}$. Of course this requires to read the two values and calculate the difference, instead of simply comparing one against the other.


Imbalanced class with same rows? In my dataset i have 3 classes-> 0,1,2.
0(72k),1(13k)and 2(13K) in brackets are there count.
So whenever i try to predict them with any algorithm ,i observed that almost all the "2"'s are predicted as "0".
On little exploration i found that there are some rows where attributes of 0 and 2 are exactly same.
Any technique to tackle this issue?

The further exploration of your data would help. Are there, for example, some clusters where relative frequencies of your classes are much different from average?

Now I see two ways to increase sensitivity of your algorithm toward class 2:

Use probabilistic prediction. Maybe what you really need is to estimate the probability that the current example belongs to class 2. If identical observations are sometimes marked as 0 and sometimes as 2, you cannot do much better then say "with probability X this is the second class".
Use class_weights and increase them for class 2. This will lead to more predictions of 2 and less predictions of 0 and 1. With imbalansed datasets it sometimes helps.


Looking for help calculating a probability formula How do I put this into a calculator or a excel spreadsheet formula.  I have never done this math before, but I want to figure it out.

Ok, I am not sure if I understood it correctly. But if it is only the right-side figure equation it would be:

=1/(1+EXP(3.1058))

(I have tested on LibreOffice only, but it should work on excel as well)

Now, if you want to do something more complex using the prior knowledge of event A, it should be more clear what is exactly this event and how it is related to the equation.

(I am relly not sure it this is what you want, so if it is not, I apologize)


Algorithm suggestion for anomaly detection in multivariate time series data I have time series data containing user actions at certain time intervals
eg

Date                 UserId   Directory  operation      Result
01/01/2017 99:00     user1    dir1       created_file   success
01/01/2017 99:00     user3    dir10      deleted_file   permission_denied

unique userIds > 10K
10 distinct operations

and 4 distinct Results

I need to perform anomaly detection on user behavior in real time. Any suggestions on which method I should use?

The anomaly needs to flag whether some user operations are outliers

A very small subset of input data will be labelled. But most of the data will be unlabelled.

The problem with your data set it that it does contain multiple categorical variables (as far as I can see). Another problem is that the users might do sequences with different lengths and different order (which makes it very difficult to detect suspicious patterns). I would create histograms for each variable and see which categories are common and which are not so common. If you have looked at the descriptives of each variable you should be able to see which variables allow you to discriminate.

A good metric is the entropy (dispersion) $H = -\sum_{l=1}^{L}p_l\ln p_l$ (is 0 if all manifestations of the categorical variable are concentrated at one label; is $\ln L$ if all manifestations are uniformly distributed). and the Gini-index $\text{G}=1-\sum_{l=1}^{L}p^2_l$ (tends to zero if one label is very dominant, becomes larger for uniformly distributed labels for a variable and is bounded by $1-1/L$). The variable $p_l$ is the relative frequency of the $l^{\text{th}}$ manifestation of the categorical variable that we are investigating and $L$ is the number of all possible manifestations of the categorical variable.

The problem with this procedure is that we are not considering the interactions between your variables. But it is the first approach that you could try. If the variables do not correlate that much this might be sufficient.

Without labeled data, it will be very difficult to use machine learning methods to solve this problem.


Searchable list of Kaggle challenges Is there a way to search Kaggle for a list of current and prior challenges, or an external site that does that?
When I search on Kaggle it will only bring up solution notebooks and datasets, it doesn't seem to have a filter on challenge page.

Have you checked the competition's stage?
In there you will see all the active and concluded contests
https://www.kaggle.com/competitions


Estimating effort of building machine learning model I know it depends on the problem and various other factors like data availability, the complexity of the use case, the workload of developer ..etc, but can someone suggest effort estimation of building a minimum viable product.

I have taken the following parameters into consideration while preparing effort estimation(close to 3months) for building a price prediction model based on two problem definitions(no sample data supplied by client), which would be integrated with web interface.

Data loading & pre-processing - 1 week
Development of model - 2.5 month
Deployment - 2 weeks.

not sure whether it's overestimated or underestimated.

Note: Please do not close with the comment "too broad" as I do not have much/can not provide more information. Requesting expert advice.

Data Loading & Pre-Processing : depending on your data-size, this is a highly variable measure. we can agree that this will be the second longest task.

Edit : This phase ,for me, was about collecting the relevant columns and structuring a csv file to be read and loaded in a python environment.

Model Development : again, this will highly depend on your data ( and i'm referring to complexity and structure of your data rather than the size) , if you have a lot of features at hand, and your problem is complex ( demands a lot of effort in feature engineering and feature selection ) , this will be the longest task at hand and will probably take up most of your time
I would say deployment would take something in weeks like you mentionned depending on the complexity ( yes again ) of your production environnement.

Please take my statement lightly as i don't have much experience in this field. I've had the opportunity to deploy some models as webservices, and it took me less than your projections. However, our problems are quite different.. so my numbers could be out of match.


Steps to fit a Machine learning model for prediction of up and down market movement I have around 5 years of data of an index containing many features on a daily basis. I want to classify whether the index will move up or down the next trading day (up or down movement is determined by next day open/close price). I am using an SVM classifier for this classification. What could be some essential steps which need to be followed? I suppose since I am using financial data, there would be some deviation from the traditional method of applying machine learning. I have the following steps in mind:

Prepare data containing all features at-hand and direction variable.

Feature engineering: That is creating various features using transformations like log, % change on various available features. I have not incorporated this step but will do it later when I have a decent running model.

Feature Selection: How to go about this? I first am using various features which I think have predictive power but how to go about it systematically.

Walk forward Modelling: After selecting features I have the data which I will be using for my model. I am using the first 500 days of data to train the model and testing it on the next day and then using data from day 2 to day 501 to train the model and then testing it for day 502, and so on until I reach the present day. I make sure that I scale my training data in the range 0-1 and using the same scaler for test data before running the model. I use the default parameters of SVM in sklearn right now.

I then check performance of my model by using various metrics obtained from confustion matrix like accurace, F1 score, etc.

For applying SVM I am using this reference which tells to use cross-validation to choose C and gamma. How can we use CV here if we have financial time series data? What else should I include in my steps?

Why do you want to use SVM+feature engineering in 2021? If you have a time series dataset, use LSTM/GRU/biLSTM for binary classification (last layer is fully connected). Loss function is binary cross-entropy, accuracy metrics overall accuracy or F1 or precision-recall.


Plots getting rid of whitespace So I'm doing the Udemy course for Data Science in Python, and there's something weird happening with the plots. The first one in the above picture is mine, the second is the instructors.

How do I get rid of the white space on the sides, they seem to be default when using the plotting function in pandas.

Edit:
This is all the code. Everything is basically default, so I'm guessing the matplotlib has some default settings changed in newer version of the package. The file df3 is here https://github.com/sxsheng/Udemy

Code:

import pandas as pd

import matplotlib.pyplot as plt

df3 = pd.read_csv('df3')

%matplotlib inline (This line is to be used in jupyter notebooks to display plots)

df3['a'].plot.hist()

Welcome to the site!

what you where thinking is right, the new version plot looks a bit different, to get exactly the same plot as above you can use the following code:

import pandas as pd
import matplotlib.pyplot as plt
df3 = pd.read_csv('df3')
#%matplotlib inline (This line is to be used in jupyter notebooks to display plots)

ax = plt.subplot(111)
ax.hist(df3['a'],ec='black')

#this is x axis limiter since your value ranges in between 0 to 1 I gave these values
plt.xlim([0,1])
plt.show()

Outcome:

Let me know if you need anything else.

If this is what you were looking for, then you can accept the answer by clicking on the green tick mark.


Conditional attributes in a prediction problem I am trying to implement an algorithm to predict fantasy points of a player regarding past events. However, I am having some doubts about some attributes.
Imagine that a player is going to play away in the next match, I believe it is useful to have the points/goals he did in the past away matches to evaluate how well he performs away from his stadium. However, that same attribute would not be useful if the next match is at home. The same with the % of occupation of the stadium - it is important when a player plays at home but not relevant when he plays away.
What is the best approach to this kind of scenario? Do I put the away goals anyway even when the match is at home?

Regards

Decision trees can handle conditions based on several features by design: if the model determines that the feature "at home" is important for the prediction, it will create a node based on this condition close to the root of the tree. By construction other features will appear as conjuncts, e.g. "if at-home==true and featureX==valueX and ... then predict ..." (that's just how a decision tree works).

In case you consider that this particular feature is so important that everything else should depend on it, another option is to train two distinct models: one for "at home" and one for "away". The disadvantage is that a model can only use the instances relevant for its case, so there is less training data for each of them.


How to do hidden variable learning in Bayesian Network with Python? I learned how to use libpgm in general for Bayesian inference and learning, but I do not understand if I can use it for learning with hidden variable.
More precisely, I am trying to implement approach for Social Network Analysing from this paper: Modeling Relationship Strength in Online Social Networks.
They suggest to use following architecture

Here

S(ij) represents vector of similarity between user i and j - Observed
z(ij) is a hidden variable - relationship strength (Normal distribution regularised by W - weights and similarity vector)- Hidden
yt(ij) is user interaction(1,2…n -> certain type of interaction e.g. 1=I retweeted j) (function of z and a that involves Theta parameter) - Observed
at(ij) is auxiliary variable which represents how often certain interaction occurs - Observed

Approach described in paper for training is quite difficult and involves coding of ascent optimisation. I wonder If I can use libpgm to learn W and Theta parameters. If yes, how to do it? If no, what libraries I can use to do it.

While this model could be implementable in the libpgm library (it seems to have quite rigid interface tailored to several special models, though), it will not allow you to reproduce the results of this paper. In the paper authors actually do something way simpler than Bayesian Inference – they only perform maximum-aposteriori (MAP) inference, which is way easier, but does tell you how confident the model is.

You don't have to code optimization algorithms for MAP inference by yourself, these days there are powerful and general packages that do almost everything for you. You can use automatic differentiation features of TensorFlow / PyTorch – they will compute the gradients and run optimization algorithm for you. You can also use numpy.autograd to get gradients for free, and then run any optimizer you like (check out scipy.optimize). The autograd + scipy.optimize pair is particularly neat if you wanna use more advanced optimization methods, e.g. 2nd order ones.

All these methods will give you values (point estimates) of $w, \theta, z$ that you can use to make inferences. However, if you want to have more than point estimates, and would like to do find all possible values (along with their probabilities) and maybe Bayesian Model Averaging, you'll need more powerful libraries enabling probabilistic programming. There are Edward (built upon TensorFlow), Pyro (built upon PyTorch), Stan and others.

If you feel overwhelmed by the choices, I recommend going with PyTorch.


Fully endogenous models for predicting multivariate time series I have a formal social science background but I am new to data science. My interest is in building predictive models for applications in the social sciences, mostly (but not only) in economics.

I am interested in the following kind of setups:

I have data that describe the evolution of a number of variables $j \in J$ for a number of "individuals" $i\in N$ across time periods $t \in \{1\dots, T\}$.
For example, "individuals" $i\in N$ could be countries, with $j=1$ being GDP, $j=2$ inflation, $j=3$ interest rate, $j=4$ unemployment rate, etc, although in practice I would look at situations where $\#N > 195$ (maybe $i\in N$ would be regions, or counties, rather than countries).
As the above example suggests, I am interested in situations where all my variables of interest are likely to be related to one another.
I am looking for models that, based on data for $t\in \{1,\dots, T\}$ can forecast the co-movement of all my variables of interest for $t > T$ (at least a couple of periods ahead). In particular, I don't want to have to assume some particular future scenario for all but one variable in order to get a prediction about the last one.

I am interested in prediction only, not in satistically interpreting my model.

Being trained as an economist, I know that Vector Autoregression (VAR) is one option for such "fully endogeneous" models. I have tried to learn a little more about forecasting and time-series (e.g., read through https://otexts.org/fpp2) but found little alternatives to VAR so far. Now, VAR might be fine and it might be the only option out there. But I would like to know if there are alternatives modeling techniques one may want to consider for these kinds of problems.

My questions:

What would be some alternative to VAR be to tackle these kinds of endogenous prediction problems (if any)?
Are there any resources where the application of these alternative techniques would be discussed specifically in this context?

I am still interested in alternatives that depart more fundamentally from VAR, but digging a little deeper, I found some interesting variations around VAR that I did not know about and could be very useful.

Of particular interest are regularized VAR models. Applications to macroeconomic forecasting are for example discussed here: https://arxiv.org/abs/1508.07497.

Even better, the paper comes with a fully funcitonal R-package: https://cran.r-project.org/web/packages/BigVAR/index.html, that is described in more details here: https://arxiv.org/abs/1702.07094.

Some of the regularized method in the package, like VARX-L, allows for exogenous variables. As I described in my answer, this is not what I am directly interested in. But VARX-L (and the BigVar package) only allow for exogenous variables and do not require the inclusion of such variables (the exogenous channel can be shut down with the model still being operational). Good to know I can start with a fully endogenous model and have to flexibility to add exogenous variables later if needed.


A doubt about the GridSearchCV function in Sklearn? When creating different hyperparameter combinations, does the function evaluate combination 1 on the same fold as combination 2? As in, are the folds the same across combinations? I understand that for example for cv = 3, a third of the dataset will be held out for evaluation.  There will be three such splits, each containing a test fold. Will the split be different for another combination? I also think that it is unnecessary computation to split again but does it happen?

I could not find strong reference to this fact in the documentation except for a part about cv_results_ or it is just assumed that, this is the way it is.

The probability that there's a mistake in SK-Learn is very low.

Reasons:

It's a mature library
Among the contributors is Sebastian Raschka. Those guys are very meticulous.

https://sebastianraschka.com/faq/docs/evaluate-a-model.html


How to determine feature importance while using xgboost in pipeline? How to determine feature importance while using xgboost (XGBclassifier or XGBregressor) in pipeline?

AttributeError: 'Pipeline' object has no attribute 'get_fscore'

The answer provided here is similar but I couldn't get the idea.


Terminology: SOMs, batch learning, online learning, and stochastic gradient descent I'm not sure which word to use to differentiate a self-organizing map (SOM) training procedure in which updates for the entire data set are aggregated before they are applied to the network from a training procedure in which the network is updated with each data point individually.

In the case of other algorithms I'd say stochastic gradient descent, but I'm not sure gradient descent is correct for SOM learning.  To my knowledge, SOM learning does not follow any energy function exactly, so that should mean it doesn't do gradient descent exactly, right?

Another term would be online learning, but `online' sort of implies I train my SOM in the real world with data points streaming in, eg. from a set of sensors.  It may also imply that I use every data point only once, which is not what I want to say.

SOMs

Self-organizing maps (SOMs) are one specific type of neural networks. In contrast to multilayer perceptrons (MLPs; they are used much more often) the SOMs neurons have a position on a regular grid.

Training of SOMs

SOMs are usually trained in a stochastic way (source: [1]). This means one training example is used at a time. After every single training example the network learns something.

In contrast, batch learning means the network is presented the complete batch of all training-examples (or a big part of usually 128 examples; then you call it mini-batch). The parameters are only updated after all of the examples were presented to the network.

In contrast to MLPs, you don't apply gradient descent (as far as I know; I've never used them, but only read about SOMs). I wouldn't even know to which function you would apply the gradient descent algorithm, because there is no error function in SOMs. (At least no standard error function).

For MLPs, I know that the advantage of batch training compared to stochastic training is that the weights are not jumping as much. So you rather go in the right direction. However, the advantage of stochastic training is that you make updates much more often. You want to get the best compromise, so you make mini-batch training.

Online Learning

"Online" has quite a lot of different meanings:

"online" as in "in the internet".
"online" as in "online algorithms" which get input piece-by-piece and have to make a decision before knowing all of the input (e.g. when you have severe time constraints or when you can't store all the data like in CERN where you have to throw most of the data away right after you get it)
"online" in handwriting recognition in contrast to the offline OCR methods (see On-line Recognition of Handwritten Mathematical Symbols if you're interested). Online methods use the way how symbols were written, offline methods only use the image.

According to Wikipedia:

  Online machine learning is used when data becomes available in a sequential order to determine a mapping from data set corresponding labels. The difference between online learning and batch learning (or "offline" learning) techniques, is that in online learning the mapping is updated after the arrival of every new data point in a scale fashion, whereas batch techniques are used when one has access to the entire training data set at once.

Sources

[1] Andries P. Engelbrecht: Computational Intelligence: An Introduction. Page 62.


Discovering important topics in corpus of text using metadata and text content I am working on a system to classify documents into important/non-important. I have a large (200,000) sample set of documents which have been pre-labelled and using Naive-Bayes I have achieved 95% accuracy in classifying documents using an 80:20 split of training:test data. This result is amazing and I am confident I can tune the system to achieve better results.
However, the system must be able to be used for an indefinite amount of time. This raises the issue that the content of the documents being classified, or what constitutes important/non-important will change over time, so the model must evolve. Thus I need to find a way to train the model without using pre-labeled data. I have only been able to come up with one potential method for this and it didn't work.
The first method I have tried is using LDA topic modelling to create a set of topics from the corpus. I then attempted to find a subset of topics which fit the important documents and a subset which fit the non-important topics. Unfortunately this method did not work as the content of the important/non-important documents is too similar to create obviously distinct topics.
Does anyone have a suggestion for how I could achieve this or any resources that could point me in the right direction?

There is no way to keep a supervised model update as the data distribution changes over time without labeling new instances.


How to run a saved tensorflow model in the browser? After doing my hello world models, I would like to let them available at Github pages, which means that I need to serve the model only with static files. Is it possible?
All the tutorials I found requires nodejs or some backend

You can find exactly that at this site, which offers a handwritten digit recognizer as a static site served from github pages.
Here you can find the article describing how the author did it. You can also have a look at the github repo or directly the html file to understand the how it works. The author also released the colab notebook used to train the model.


SGD, calculating it by hand While I find a lot of material of SGD (Stochastic gradient descent), I am struggling to find one concrete example with numbers e.g. calculating it by hand for let's say, one iteration would help me a lot.

You can see in the following link examples in Numpy of the following optimisers

Stochastic Gradient Descent
Stochastic Gradient Descent + momentum
Stochastic Gradient Descent + Nesterov momentum
Adagrad
Adadelta
RMSProp
Adam
AdaMax
Nadam

Just clone the repository and open the Report.ipyb with Jupyter Notebooks.

https://github.com/ivallesp/awesome-optimizers


What is purpose of the [CLS] token and why is its encoding output important? I am reading this article on how to use BERT by Jay Alammar and I understand things up until:

  For sentence classification, we’re only only interested in BERT’s output for the [CLS] token, so we select that slice of the cube and discard everything else.

I have read this topic, but still have some questions:

Isn't the [CLS] token at the very beginning of each sentence? Why is that "we are only interested in BERT's output for the [CLS] token"? Can anyone help me get my head around this? Thanks!

CLS stands for classification and its there to represent sentence-level classification.
In short in order to make pooling scheme of BERT work this tag was introduced. I suggest reading up on this blog where this is also covered in detail.


Feature selection for tracking user activity within an application I am developing a system that is intended to capture the "context" of user activity within an application; it is a framework that web applications can use to tag user activity based on requests made to the system.  It is hoped that this data can then power ML features such as context aware information retrieval.

I'm having trouble deciding on what features to select in addition to these user tags - the URL being requested, approximate time spent with any given resource, estimating the current "activity" within the system.

I am interested to know if there are good examples of this kind of technology or any prior research on the subject - a cursory search of the ACM DL revealed some related papers but nothing really spot-on.

Well, this may not answer the question thoroughly, but since you're dealing with information retrieval, it may be of some use. This page mantains a set of features and associated correlations with page-ranking methods of search engines. As a disclaimer from the webpage itself:

  Note that these factors are not "proof" of what search engines use to rank websites, but simply show the characteristics of web pages that tend to rank higher.

The list pointed may give you some insights on which features would be nice to select. For example, considering the second most correlated feature, # of google +1's, it may be possible to add some probability of a user making use of such service if he/she accesses many pages with high # of google +1 (infer "user context"). Thus, you could try to "guess" some other relations that may shed light on interesting features for your tracking app.


Error encoding categorical features using sklearn pipelines I am new to sklearn pipelines and am using this post as a guide for my code:

https://www.codementor.io/bruce3557/beautiful-machine-learning-pipeline-with-scikit-learn-uiqapbxuj

I am trying to encode a categorical features using a transformation pipeline, but no matter what encoder I use, I get the same error. As far as I can tell from reading other posts, scikit-learn should be able to handle categorical variables as strings from version 0.20 or greater (namely with the OneHotEncoder.)

ValueError: could not convert string to float: 'Male'

Where I have entered xxxxxxxxxxx below replace with one of the following

ce.OneHotEncoder
ce.TargetEncoder
OneHotEncoder
OrdinalEncoder

from sklearn.pipeline import FeatureUnion
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import Imputer
from sklearn.preprocessing import OneHotEncoder
from sklearn.decomposition import NMF
from sklearn.feature_extraction.text import TfidfVectorizer
import category_encoders as ce
from sklearn.preprocessing import OrdinalEncoder
from sklearn.pipeline import Pipeline
import pandas as pd
import numpy as np

# create example data
example_df = pd.DataFrame({'Sex':['Male','Female','Female'],'Survived':[1,1,0]})
X_train = example_df.drop('Survived', axis=1)
y_train = pd.DataFrame(example_df['Survived'])

# build example pipeline
cat_pipe = ("categorical_features", ColumnTransformer([
            ("categorical", Pipeline(steps=[
                ("impute_stage", Imputer(missing_values=np.nan, strategy="median")),
                ("label_encoder", xxxxxxxxxxx())]), ["Sex"]
            )])
          )

example_pipeline = Pipeline(steps=[cat_pipe])

# fit pipeline
example_pipeline.fit(X_train, y_train)

# Name                    Version
scikit-learn              0.20.3
category_encoders         1.3.0
numpy                     1.16.2
pandas                    0.24.2

In this line:

("impute_stage", Imputer(missing_values=np.nan, strategy="median"))

Because your input type is string, you shouldn't fill the null value to median (we cannot average string value).

From the document, you can fill null value with a string constant like:

Imputer(missing_values=None, strategy="constant", fill_value="NULL")

to represent null value in your string field.


How to get mean of column using groupby() and another condition For the following

df = pd.DataFrame()
df['1'] = 1,2,1,2,1,2
df['2'] = 3,6,5,4,7,8
df['3'] = 1,1,1,2,2,2

I want to do

mean(df.groupby().loc[df['1']==df['3'],'2'].mean()

which doesn't work. And simply doing this :

a=df.groupby(['1','3'])['2'].mean()

gives

OUTPUT:
1  3
1  1    4
   2    7
2  1    6
   2    6

But I only want cases where column 1 and 3 have the same elements:

1  3
1  1    4
2  2    6


How to run PCA and KNN on big-data I work with python and images of watches (examples: watch_1, watch_2, watch_3). My aim is to take a photo of a random watch and then find the most similar watches to it in my database. Obviously, one main feature which distinguishes the watches are their shape (square, rectangular, round, oval) but there are also other ones.

For now, I am just running a PCA and a KNN on rgb images of watches to find the most similar ones among them. My source code is the following:

import cv2
import numpy as np
import os
from glob import glob
from sklearn.decomposition import PCA
from sklearn import neighbors
from sklearn import preprocessing

data = []

# Read images from file
for filename in glob('Watches/*.jpg'):

    img = cv2.imread(filename)
    height, width = img.shape[:2]
    img = np.array(img)

    # Check that all my images are of the same resolution
    if height == 529 and width == 940:

        # Reshape each image so that it is stored in one line
        img = np.concatenate(img, axis=0)
        img = np.concatenate(img, axis=0)
        data.append(img)

# Normalise data
data = np.array(data)
Norm = preprocessing.Normalizer()
Norm.fit(data)
data = Norm.transform(data)

# PCA model
pca = PCA(0.95)
pca.fit(data)
data = pca.transform(data)

# K-Nearest neighbours
knn = neighbors.NearestNeighbors(n_neighbors=4, algorithm='ball_tree', metric='minkowski').fit(data)
distances, indices = knn.kneighbors(data)

print(indices)

However when I try to run this script for more than 1500 rgb images then I get a MemoryError at the point where the data are processed by the PCA method.

Is this normal for a pc with 24GB RAM and 3.6GHz Intel Core CPU without any discrete GPU?

How can I overcome this?

Shall I use another method like Incremental PCA (or a deep learning algorithm) or simply shall I buy a discrete GPU?

KNN is instance based so it will store all training instances in memory. Since you are using images this will add up quickly. KNN on untransformed images might not perform that well anyway, you could look into filter banks to transform your images to a bag-of-word-representation (which is smaller and more invariant).

However if it is accuracy you are aiming for I would recommend skipping all that (it is very 2012 anyway) in favor of using deep learning, fi: construct an auto-encoder and determine similarity on the encoded representation of an image (which could in turn be done using knn btw).


Defining Input Shape for Time Series using LSTM in Keras I have been trying to model Time Series forecast using Keras LSTM algorithm. My dataset consists of weekly sales data from Jan-2016 and I also have external features such as Festivals/Events each modeled as a flag (0/1) for each week. So for e.g. let's say 2016W01 has sales of 100 units and has flag value 1 for the NewYearFlag. Likewise, I have created flags for many such events such as Independence Day, Diwali (this is in a different week every year) and so on.

Based on the Time Series decomposition analysis,I also know that 5 week lags are signficant. So it is Week-1, Week-2, Week-3,Week-4 and Week-5 variables.

Essentially,my dataframe row looks like
Week Number (Row Index), SalesQty, SalesQtyW-1,SalesQtyW-2, SalesQtyW-3,SalesQtyW-4,SalesQtyW-5,NewYearFlag,DiwaliFlag, IndepdenceDayFlag, ChristmasFlag.

So I have X = {SalesQtyW-1,SalesQtyW-2, SalesQtyW-3,SalesQtyW-4,SalesQtyW-5,NewYearFlag,DiwaliFlag, IndepdenceDayFlag, ChristmasFlag}
Y = {SalesQty}

So when I pass my dataframe to Keras LSTM layer, I pass (144,9) as there 144 rows (52+52+40) in my training data.

I always get the error - Expected 3 dimensions but got only two (144,9)

I am not sure where I am going wrong. Please help. I checked most of the questions of similar nature but not able to find the right answer. Dont know where am I going wrong!!!

An LSTM will expect the data to be of the format (samples, time steps, features).  In a univariate case where we will focus on just the time series with the sales data, where you say five lags are required as input:

sales data = [100, 150, 345, 5, 800, 655, 100, 200, 300, 400]

You could format your data like this using a sliding window of step size one:

[100, 150, 345, 5, 800]
[150, 345, 5, 800, 655]
[345, 5, 800, 655, 100]
[5, 800, 655, 100, 200]
[800, 655, 100, 200, 300]
[655, 100, 200, 300, 400]

this would give you six samples each of five time steps and one feature (the feature being the sales data).

So the shape you would take the 2d array above and reshape it to a 3D array of (samples, time steps, features) which you could also think about like (rows, columns, features), giving (6, 5, 1).

Let's say now you have another feature holidays like in your example, in this case you have two features, the sales data and whether or not it's a holiday or not.  So you would need dimensions of (6, 5, 2), six samples each containing five time steps of data each with two series each (or features).


Reinforcement learning: understanding this derivation of n-step Tree Backup algorithm I think I get the main idea, and I almost understand the derivation except for this one line, see picture below:

I understand what we're doing by using the policy probability to weight the rewards from time t + 2 (because getting here depends on the prob of taking an action that gets here). But I don't understand why we similarly subtract the value function from the return...

It also doesn't seem to match the example target return (G) implied for 2 step backup on slide 15 of this lecture's slides:

https://www.dropbox.com/sh/3xowt7qvyadvejn/AABpWQMKWX3KVbeqVlBcxNYra/slides%20(pdf%20and%20keynote)?dl=0&preview=13-multistep.pdf

Thanks for any insight. I could be missing something simple/obvious as I dive into these details.

EDIT - for more context, see pg. 160 of this pdf which is where the picture comes from: http://incompleteideas.net/sutton/book/bookdraft2016sep.pdf

The slides and the book are consistent. Notice how in the slides there is a restriction in the summatory: i.e.  $a \neq A_{t+1}$. For $G^{(2)}$, you need to "remove" from $V_{t+1}$ the term that should not be there, i.e. $A_{t+1}$.

Now, why this term is removed?

If you keep this term you will be adding $A_{t+1}$ twice. In 1-step backup, it is part of the expectation of step $S_{t+1}$.

When you calculate 2-step backup you want to replace $(S_{t+1}, A_{t+1})$ in the 1-step expectation with the discounted expected value of $S_{t+2}$.
So you substract the term and add the discounted expectation for $S_{t+2}$