Problem when exporting scored labels to excel I have done a regression using Azure ML Studio and it ok. The r squared is 95% on the test sample.
When predicting, the results look ok when visualizing the data.
The results appear as expected, with the predicted label having statistics very similar to the trained ones.
But when I try to export it to Excel either:
 1. consuming it from the web or
 2. reading directly the saved dataset
I obtain "weird" number like  instead of 41,36.
How can I obtain right values?

As indicated by TitoOrt's comment the solution is simply changing the comma to the dot character as decimal point in Excel.

The reason was:

I am using the Spanish version of Excel. where is used the character dot (.) for decimals and not comma (,). AZURE uses dot for decimals and Office products in Spanish use comma by default.


Can we make an AI to fine tune other AI hyper parameters? Every time AI gurus talk about fine tuning hyper parameters, they more or less say it's trial and error. But can't we make an AI to tell AI what its hyper parameters should be?

Yes, you can make an AI for that. By AI, you mean algorithm which finds hyperparameters efficiently. There are many ways to find hyperparameters which would then come under the AI like using bandits to find hyperparameters, Bayesian methods to find hyperparameters and many other methods exist. Search for hyperparameter optimization and you will find a ton of methods which improve upon grid search and random search.
AI in the purest sense of it, if I take, then, I will assume that you are talking about reinforcement learning since that deals with taking actions and changing its behaviour (in this case, hyperparameters). You can create an AI for that.


How to go about creating embeddings (especially, token to Id mapping) for categorical columns in tensorflow 2.0+? I have a csv with both categorical and float dtypes. I want to do the following:

For each categorical column i will use pandas to compute the unique values (pd.unique()) that are present in the column. say u_l for a column
I will use the len(u_l) to decide upon the dimension of embeddings that i use for a particular categorical column that i want i embed (this step is the reason i cannot use tensorflow_transform)
I want to create some stateful node that can map category (token) value to embeddings index thus subsequently i can lookup the embedding from embeddings matrix that i created in step 2

I dont know how to go about doing it currently. A very inelegant solution i can see is using tensorflow_datasets:

encoder = tfds.features.text.TokenTextEncoder(u_l,decode_token_separator=' ')
concatenate the entire column using space delimiter (c_l) (c_l is one string now) and then using encoder.encode(c_l)

This is a very basic thing that i think tensorflow would be able to do relatively easily. Please guide me to the right solution

Embedding() layers take sequences as inputs. If there is no time dependency in your data (such as for time series datasets or language corpora) then those layers cannot be used. What kind of data do you have?

Alternatively, you can use dimensionality reduction, one-hot encoding or, in some cases, label encoding. I personally suggest dimensionality reduction whenever possible, even if it's the most computationally intensive choice.


How to implement clipping the reward in DQN in keras How to implement clipping the reward in DQN in keras? especially how to implement clipping the reward?

Is this pseudo code correct:

if reward<-threshold reward=-1
elseif reward>threshold reward=1
elseif -threshold<reward<threshold reward=reward/threshold

And if reward is always positive how we can change clipping the reward?

Since you're using keras-rl, you could use its class Processor. And simply write a new processor and assign to your agent. The new processor will be something like:

class MyProcessor(Processor):
    def process_reward(self, reward):
        """Processes the reward as obtained from the environment for use in an agent and
        returns it.

        # Arguments
            reward (float): A reward as obtained by the environment

        # Returns
            Reward obtained by the environment processed
        """
        # Change min and max according to your needs. I supposed that your threshold was 1.
        min = -1
        max = 1
        return float(np.clip(reward, min, max))

Note that there isn't a real need to cast the reward as a float and in some agent implementation it could also fail (but it's wrong the agent implementation if it does). I've done it because in Gym the reward is defined to be a float. As previously said, usually nothing goes wrong if it is an int, a numpy.float64, or something else but easily castable as a float.


Is epsilon error a standard known error or custom created by this paper? I'm reading this computer vision paper, research paper link, about creating a model to estimate the real age and perceived age of the person in the image (or at least that's what I think it's about). The perceived age is decided by this method: each image is looked at by 10 independent individuals and they estimate the age of the person. The mean and standard deviation are taken from the age guesses of the 10 individuals.
The paper then goes on to use an epsilon error by this equation as part of the model evaluation for perceived age and state the following the evaluation employs fitting a normal distribution with the mean µ and standard deviation σ of the votes for each image. The paper also says that this epsilon error covers the aspect of the uncertainty of the ground truth age.

The results are then posted in a table showing different epsilon error values based on different models.

My questions are the following:

What is that equation and is it standardly used? It says

What is x in this equation? I thought it was some type of feature scaling of the image but that makes no sense because the results are all different in the table and completely out of context anyway. Is x supposed to be the predicted age?

What is that equation and is it standardly used?

I havn't seen exactly that before, so I guess it's not "standard", but it's not so unusual.  The equation is "just"  1 - (normal distribution).  If you think of the normal distribution as a measure of "how similar a sample is to the mean", then that equation turns the similarity (big is good) into a distance (small is good).
It looks related to "robust loss functions";  See below.

What is x in this equation? I thought it was some type of feature
scaling of the image but that makes no sense because the results are
all different in the table and completely out of context anyway. Is x
supposed to be the predicted age?

Your second thought is right.  $x$ is predicted age.  They use that set up since they don't have a single correct answer for their data, but rather, many guesses/votes as what the correct age is.
From the paper:

The LAP challenge evaluation employs fitting a normal distribution with the mean  μ and standard deviation σ of the votes for each image.

Say you have two images. For one, every guess you have for the age is the same.  For another, you get a wide range of guesses for the age.  The second image must be "harder" somehow.  This equation gives you an error measure that tells you how the algorithm performed relative to the many people that guessed the true age.
Robust losses
Check out this work
The "epsilon-error" equation from the paper you linked to will look sort-of like the bottom (orange) trace in Figure 1:


Problems with class embedding in keras I am doing a toy example with mushroom dataset to learn class embedding with keras:

I am trying to embed a single feature:

inputs = Input(shape=[1])
emb = Embedding(input_dim=cap_shape_dummy.shape[1], output_dim=3)(inputs)
output = Dense(units=1,activation='sigmoid')(emb)
model = Model(inputs=inputs,outputs=output)

However I always receive the same error when I try to fit the model:

Error when checking target: expected dense_2 to have 3 dimensions, but got array with shape (8124, 1)

Am I doing anything wrong? Did I miss anything with category embedding?
Thanks.

inputs = Input(shape=[1])
emb = Embedding(input_dim=cap_shape_dummy.shape[1], output_dim=3)(inputs)
x = Flatten()(emb)
output = Dense(units=1,activation='sigmoid')(x)
model = Model(inputs=inputs,outputs=output)


How do you plot overlapping durations? My data contains a set of start times and duration for an action. I would like to plot this so that for a given time slice I can see how many actions are active. I'm currently thinking of this as a histogram with time on the x axis and number of active actions on the y axis.

My question is, how should I adjust the data so that this is able to be plotted?

The times for an action can be between 2 seconds and a minute. And, at any given time I would estimate there could be about 100 actions taking place. Ideally a single plot would be able to show hours of data. The accuracy of the data is in milliseconds.

In the past the way that I have done this is to count for each second how many actions started , ended, or were active. This gave me a count of active actions for each second. The issue I found with this technique was that it made it difficult to adjust the time slice that I was looking at. Looking at a time slice of a minute was difficult to compute and looking at time slices of less than a second was impossible.

I'm open to any advice on how to think about this issue.

Thanks in advance!

Since you want to show so much data, I think that your best choice is going interactive. Check out this demo, it is close to what you want but not quite.

It is very difficult to show a lot of data in a single diagram, together with the finest details and the bird-eyes view. But you can let the user interact and look for the details. To show counts, one option is to use color-coding. Take a look at this image (code here): .

Here rgb channels have been used to encode (the logarithm of) the number of active events (red), events starting (green) and events ending (blue) for windows of different size. The X axis is time, and the Y axis represents window size, that is, duration. Thus, a point with coordinates (10, 4) represents the interval of time that goes from 10 to 14.

To make a lot of data more detailed, it could be a good idea to make the diagram zoomable (like in the demo before), and to give the user the possibility of visualizing just one channel/magnitude.


Overfitting due to features correlating with training set generation rules As background, I am using a Deep Neural Network built using Keras to classify inputs into 5 categories.

The current structure of the network is:

Input layer (~450 nodes)
Dense layer (750 nodes)
Dropout layer (750 nodes, dropout rate = 0.5)
Dense layer (5 nodes)

The issue I'm having if one of overfitting. My model performs well on the held-out test set (a proportion of my training set), with accuracy sitting right around 99%. However, when I look to apply the model onto unlabelled data, it is only able to classify ~67% of observations into any category, before even considering the correctness of those classifications!

I think the issue may be around my feature and training set generation process. I generated the training set using a rules-based string matching method. This generated a training set of around 3.6 million observations (10% of population).

However, one of the largest features for my input layer is an embedding of the same text used to generate the training set. Therefore, the words matched to generate the training set are also embedded and used as features. Worth noting the text is around 140 characters per observation, and I matched bigrams from that text (so there is other information in that text that would be useful as a feature).

I would remove this feature altogether, however this is the richest information associated with each observation.

Is there a way to solve this without removing that feature altogether?

Hope this makes sense and happy to provide more clarification.

Simplified explanation:

My model performs well on my training and test sets.
Performs badly on unlabelled data.
Each observation is associated with a block of text.
To label my training set I used string matching on that text.
The text is also a feature (embedding).
Is this causing my poor performance on unlabelled data (model
learning those string matches?).
If so what can I do?

EDIT: Also happy to hear if you think the issue is something else.

I would say that there are no "problems" in the sense what is happening is to be expected.

First of all, here is a key ML reminder which somehow often gets lost:

Performing well on the test set is pointless if that test set is not representative (i.e. is not similar to unlabeled instances)
Adding more training data does not help if that training data isn't covering new situations

You say you created your training set using string matching rules, I assume that these rules are similar to " "mortgage repayment" was a "housing" observation." as you pointed out in the comments.

Since your model considers bigrams as input, it is not surprising that it found a way to reverse-engineer which string matching rules you used.

To improve your model, I would look at what mistakes your model currently makes on unlabeled data, this should provide you with training instances that do not fit your string matching rules.


I have hourly data of a metric for 15 days, Can i predict the outcome values for same metric for the next 15 days? I have tried a linear regression model for the same data, Since the regression line is continuous i'm not sure if it works to predict the outcome values for next 15 days, or for a given period of time!

Are there any models to be considered?

If its a timer series data, then according to me you should go for RNN based model, typically LSTM.

Also you are looking an output for next "n" days, in this case you have to develop "Multi-Step LSTM Models".

  Please refer a very good link on TimeSeries problems

  Also you can refer TensoFlow code of very basic timeseries problem on
  github.

Note: Please modify code for Multi Step problem.


Create similarity matrix I have a training set and a testing set of vectors. All the vectors are labeled.

For each labeled vector in the testing set, there are 3 vectors in the training set with the same label.

I'm using the cosine distance in order to calculate the similarity between the elements in the vectors. In the picture, we can see the results of applying the cosine distance similarity in a subset of 6 vectors from the testing set and 18 from the training set.

Now, I would like to create a similarity matrix of the labels. So, in this case, I'd need a matrix of 6x6 dimension, but I am not sure how to transform this matrix of scores to a similarity matrix.

You have two scenarios here.

The vectors with the same labels are close to each other.

Then, what you can do is perhaps use the distance between the two closest/furthest elements in the groups with label x and the group with label y.

You could also create an average vector for each label, and then simply get the distance between these.

Your vectors are not close to each other

This is not an issue. Here, what you'll want to see, is if for instance, for each vector with label x, there is a vector with label y close to it.

To do that, you could use the Earth Mover's Distance which gives you a single score when you are trying to see how far two "groups of things" are.


How to use a deep learning algorithm to cluster image *styles* in an unlabeled data set? I have a hard problem, and I'd be interested in hearing people's thoughts. I have a set of images depicting a large variety of phenomena, but only a few styles. The images are already labeled by phenomenon, but not by style. The distinction between image styles is largely based on what technology was used to produce the images. We want to use a clustering algorithm to separate the different styles.

For an analogous problem, imagine that you had a very large dataset of paintings, but with many kinds of different subjects and only a few styles. Further imagine that you don't care what the subject of the painting is, only about its stylistic features (e.g. watercolor vs oil vs acrylic, realistic vs photorealistic vs impressionist vs expressionist). How could you try to go about "telling" a deep learning algorithm to emphasize these stylistic features over what the subject of the painting is, without using any labels?

You can use a pretrained model, extract features, and try to cluster images based on these features. The success will be contingent on the "style" you want to distinguish (and if the derived features represent the different styles). I had a similar problem and used Kerans and VGG16 features to cluster images using KNN. Find my Python code on Github.


Time complexity of function minimizers for neural networks I am trying to train a neural network for recognizing handwritten letters from A to J . I have a training set of size 200000 . Each training set is a list of  784 pixel values. My neural net has input layer of size 784 , hidden layer of size 50 and output layer of size 10.

I am using fmin_cg minimization function of scipy library of python. The problem I am facing is that each iteration is taking a lot of time.

The first iteration took almost 7-10 minutes.
The second iteration took 20 minutes.
Third is still running.

This might be due to my outdated computer with only 2 gb of memory and a slow processor but I have previously trained a neural net with the training set of size 5000 , input layer size if 400 , hidden layer size 25 and the output layer of size 10 . This neural net recognized handwritten digits and it was an exercise problem of coursera course on machine learning by Andrew Ng .

So  yes I know that the current neural network should take more time to train than the previous one as the training set , input layer, and hidden layer are all much larger than previous neural net but still, I think it's taking a lot of time . Why is it so slow ?

Is it normal for the neural network of this size ? Or should I use other faster optimization algorithm ? Is there a way to measure time complexity of neural networks ?

The first paragraph in this tutorial, explains it pretty well.

Also the time complexity will not grow linearly if you increase the size of your network. It will grow squared and so will the number of parameters.


NLP Emotion Detection - Model fails to learn to recognize negations I am working on a nlp emotion detection project. The emotions that I try to predict are 'joy', 'fear', 'anger', 'sadness'. I use some publicly available labeled datasets to train my model e.g. ISEAR, WASSA etc. I have tried the following approaches:

Traditional ML approached using bigrams and trigrams.
CNN with the following architecture: (X) Text -> Embedding (W2V pretrained on wikipedia articles) -> Deep Network (CNN 1D) -> Fully connected (Dense) -> Output Layer (Softmax) -> Emotion class (Y)
LSTM with the following architecture: (X) Text -> Embedding (W2V pretrained on wikipedia articles) -> Deep Network (LSTM/GRU) -> Fully connected (Dense) -> Output Layer (Softmax) -> Emotion class (Y)

The NN models achieve more than 80% accuracy but still when I use the trained model to predict the emotion on text that includes some negation I get the wrong results. For example:
Text : "I am happy with easy jet, it is a great company!"
Predicts Happy
Text: I am not happy with easyjet #unhappy_customer
Predicts Happy
Any suggestions on how to overcome this problem?

In general this is a difficult problem, it's about the problem of Natural Language Understanding which is far from being solved.
The advanced option requires a full syntactic parsing of the sentence, ideally followed by some kind of semantic representation of the sentence, for example by extracting relations. As far as I know this is rarely used because these steps will often cause as many errors as they solve.
Some more reasonable heuristics can be considered, for instance detecting specific negation words and either including this information as feature or modifying the original features accordingly (e.g. when a negation is detected replace the token "happy" with "not(happy)" in the features).
Note that it's unlikely to be perfect anyway, due to the usual obstacles: hedging ("I would assume that it was quite good"), sarcasm ("Sure, I was very happy with the terrible service"), metaphors ("I was feeling like a fish in water"), etc.


Understanding outputs from ANN and how to improve validation loss I apologise if this is a bit long winded, but it was suggested by another user that I post.
I will start by saying that I am very new to the world of machine learning and deep learning. As such, the most important thing I am after is the understanding of what I am doing.
I am trying to build an ANN for binary classification.
I have a binary feature matrix in the form of N x D, where N is number of samples and D is number of features. In my dataset N has a max of ~ 2 million - but I run my testing on ~ 500k due to the time it takes to run (even when utilising my GPU). If I get a promising validation curve after testing, I will run on the full dataset to verify. D in my dataset is 5. Thus, I have a feature matrix of the form 500000 x 5. A snippet below:
[[0 1 0 1 1]
 [0 1 1 0 1]
 [0 0 1 0 1]
 [1 1 0 1 1]
 [1 1 0 0 0]
 [0 1 1 0 1]
 [0 1 0 0 0]
 [1 1 0 1 1]
 [0 0 0 1 1]
 [1 0 0 0 1]]

I have a target matrix in binary form, snippet below:
[1 0 1 1 0 0 1 0 1 1]

Based on my understanding, for binary classification, the Input layer should be the same as D, and your output layer should have 1 node and it should have a sigmoid activation function.
Thus, I have taken this approach. Now, I also understand that machine/ deep learning is a lot of experimentation and so I have gone through many different iterations of changing the number of hidden layers, as well as changing the number of nodes per hidden layer - all to no seemingly obvious benefit. I have also played around with the following: learning rate (for Adam optimiser), train to test ratio (currently at 0.33), random state variable (when splitting the dataset - currently at 42), batch size (currently at 128), epochs (currently at 50). All of this experimentation with these variables still lead to a high validation loss, as I will show further below.
Now, for my code. Below is my code to split the data in to train and test.
train_to_test_ratio = 0.33
random_state_ = 42

# split the data into train and test sets
# this lets us simulate how our model will perform in the future
x_train, x_test, y_train, y_test = train_test_split(X, Y, test_size=train_to_test_ratio, random_state=random_state_)
N, D = x_train.shape

Below is my code to to build the model. You can see I currently have 3 hidden layers. I have played around this a lot - from 1 hidden layer to 10 with varying nodes per layer. The best I have found is the below 3 hidden layers. The learning rate is currently 0.001 which creates the best loss curve - anything bigger is too high.
learning_rate = 0.001

# build the model
i = Input(shape=(look_back_period,))
x = Dense(8, activation="relu")(i)
x = Dense(16, activation="relu")(x)
x = Dense(32, activation="relu")(x)
#x = Dense(32, activation="relu")(x)
#x = Dense(64, activation="relu")(x)
#x = Dense(128, activation="relu")(x)
#x = Dense(64, activation="relu")(x)
#x = Dense(32, activation="relu")(x)
#x = Dense(16, activation="relu")(x)
#x = Dense(4, activation="relu")(x)
x = Dense(1, activation="sigmoid")(x)
model = Model(i, x)
model.compile(optimizer=Adam(lr=learning_rate), loss="binary_crossentropy", metrics=["accuracy"])

I then train the model:
# train the model
# trains on first half of dataset, and tests on second half
b_size = 128
iterations = 50
r = model.fit(x_train, y_train, validation_data=(x_test, y_test), epochs=iterations, batch_size=b_size)

Here is the validation curves:

Here is the accuracy curves:

As you can see the validation loss is very high, and the accuracy is not too high, which to me the model is failing.
What do these outputs mean? In the sense of with all the experimentation I still cannot get the validation loss low with high accuracy. This to me is that the model is incorrect. But what would be the correct model? Is there any advice that can be given to better understand how to move forward and build a better model?

500K is decent data size. So, we can have a larger test set and also the network should be bigger.
Try the below listed suggestions -

Have a bigger model e.g. 200 Neurons, 5-6 layers
The test set should be 25%-30% since you have a decent dataset size
Keep LR to default
Increase batch size a bit as it's still fluctuating
Data should be standardized
Try adding Regularization when it starts overfitting


About the maximum likelihood, when we convert the maximization problem into minimization, why we take the negative? On page 12, we take $log$ on both side.

$\max_{\boldsymbol{w}}L\boldsymbol({w})=\max_{w}\displaystyle\prod_{n=1}^Np(t^{(i)}|x^{(i)};\boldsymbol{w})$

$\ell(\boldsymbol{w})=-logL(\boldsymbol{w})$

$\ \ \ \ \ \ \ =-\displaystyle\sum_{i=1}^Nlog\ p(t^{(i)}|x^{(i)};\boldsymbol{w})$

The $log$ function is increasing as $\boldsymbol{w}$ increase. Why we have to take the negative?

It is common to define optimization problems as minimization problems instead of maximization. And by multiplying your target functions with $-1$ you can transform one into the other:

$$\max_{w} \log{L(w)} \Leftrightarrow \min_{w} -log{L(w)}$$

So to maximize the log-likelihood you minimize the negative log-likelihood. Basically it just comes down to conventions in optimization theory.

Moreover, since $L(w) \in [0,1]$ its logarithm $\log{L(w)}$ will be less than or equal to $0$ (note that $log{0}$ is not defined). Accordingly $\max_{w} \log{L(w)}$ means to maximize a negative number which is, at least to me, less intuitive than minimizing a positive number.

The more interesting part is actually the log-transformation which increases numerical stability of your calculations (since it "transforms" the multiplication to a sum and thereby reduces the risk of underflowing).


YOLOv1 algorithm - how to determine predictor responsibility I am researching Yolo detector, and have read the original paper, but still have some confusion and a few questions regarding the assignment of bounding box predictors to ground truth at training time (in particular calculating the confidence).
Some resources like Andrew Ngs deep.ai here set the confidence of the corresponding label vector bounding box confidence to "1" if there is an object present at that grid cell. In contrast, here is an article from Oracles engineers that set the confidence to maximum Intersection-over-union value between some object and all bounding box predictors. I think the latter is also in line with what the paper is trying to tell us:

"YOLO predicts multiple bounding boxes per grid cell.
At training time we only want one bounding box predictor
to be responsible for each object. We assign one predictor
to be “responsible” for predicting an object based on which
prediction has the highest current IOU with the ground
truth" (You Only Look Once:
Unified, Real-Time Object Detection; Redmon, Divvala, Girshick, Farhadi)

Which of these methods should be used ?
I made a sketch to try to understand the mapping between ground truth and predictor bounding boxes. Please correct me if I'm wrong. The paper states we should assing a b.b. predictor to a ground truth object, but once we have done so, that bounding box predictor is "taken" and married to that object, so it can't be assigned to any other object in that grid cell?
In my example with 3 ground truth bounding boxes and 2 predictors (B=2), we choose a predictor for object b1, which is b2_hat with highest IoU of 0.7, but b2 also has the highest Iou with b2_hat of 0.6, but b2_hat is already taken, so we take the next best thing, which is b1_hat. Now, all the bounding box predictors are assigned, and b3 gets no predictors, even if it has the highest IoU with all the boxes.

Confidence score

Some resources like Andrew Ngs deep.ai here set the confidence of the corresponding label vector bounding box confidence to "1" if there is an object present at that grid cell.

This is because prof. Ng is refering to true probability of an object being in a specific cell ($P(\text{object})$ value), and not the confidence score ($P(\text{Object}) \times\text{IOU}$ value). Thereby, we have two possible situations:
$$
P(\text{object}) = \begin{cases}
1 \text{ if an object is present in the cell}\\
0 \text{ otherwise}
\end{cases}
$$
However, this is not what YOLO is proposing to have as ground-truth value of confidence. Citing the paper:

[...] Formally, we define confidence as $P(\text{Object}) \times\text{IOU}^{\text{truth}}_{\text{pred}}$. If no object exists
in that cell, the confidence scores should be zero [...]

Thereby, the value of confidence that the network is trying to predict (the confidence ground-truth label) is given by:
$$ P(\text{Object}) \times\text{IOU} = \begin{cases}
\text{IOU if an object is present in the cell}\\
0 \text{ otherwise}
\end{cases}
$$
Because of this, the other source mentioned in the question (Oracle's article) is the correct one.

Bounding boxes question

The paper states we should assing a b.b. predictor to a ground truth object, but once we have done so, that bounding box predictor is "taken" and married to that object, so it can't be assigned to any other object in that grid cell?

During training I don't see why this can be a problem. If the predicted bounding box of a certain grid cell, has the highest IOU for two or more ground-truth bounding boxes (whose center fall on this cell), we can interpret that the predicted bounding box is trying to predict both ground-truth boxes (it may happen for example if both ground truth boxes have similar sizes/ aspect ratios).
Concretely, what's said by the authors is the following:

YOLO predicts multiple bounding boxes per grid cell.
At training time we only want one bounding box predictor
to be responsible for each object. We assign one predictor
to be “responsible” for predicting an object based on which
prediction has the highest current IOU with the ground
truth. This leads to specialization between the bounding box
predictors. Each predictor gets better at predicting certain
sizes, aspect ratios, or classes of object, improving overall
recall.

The part in bold only implies that a ground-truth object is related only with one predicted bounding box $\Rightarrow$ a predicted bounding box can be related to more than one ground-truth object.

Edit:
Given what's said in the article, during training if a predicted bounding box has the highest intersection over union with two different objects, then the loss function will be related to both ground-truth bounding boxes.
For example, if we have a look at the part of the loss function related to the prediction of the center of the bounding box $(\hat{x}_i, \hat{y}_i)$:
$$ \lambda_{coord}\sum_{i=0}^{S^2}\sum_{j=0}^B1_{ij}^{obj}\left[ (x_i-\hat{x}_i)^2 + (y_i-\hat{y}_i)^2
  \right]$$
We have that $(x_i,y_i)$ are the coordinates of the center of the ground-truth bounding box, and $(\hat{x}_i,\hat{y}_i)$ are the predicted ones.
Given this, particularizing to a certain grid cell, $i$, and to our predicted bounding box of interest, $j$, we have that its contribution to this part of the loss function is:
$$ \lambda_{coord}\left[ (x_i^{\{0\}}-\hat{x}_i^{\{0\}})^2 + (y_i^{\{0\}}-\hat{y}_i^{\{0\}})^2
  + (x_i^{\{1\}}-\hat{x}_i^{\{1\}})^2 + (y_i^{\{1\}}-\hat{y}_i^{\{1\}})^2\right]$$
Where the superscripts $\{0\}$ and $\{1\}$ represent each ob the ground-truth objects being detected.
As we see, we will be computing this part of the loss using both ground-truth bounding boxes. This makes sense if the objects that are being detected have similar sizes/ aspect ratios and are close to each other (they fall on the same grid cell).
If we focus on the confidence score part of the loss function, where we evaluate the predicted $\text{IOU}$ ($\hat{C}_i$), with the ground truth $\text{IOU}$ ($C_i$):
$$\sum_{i=0}^{S^2}\sum_{j=0}^B 1_{ij}^{obj}(C_i -\hat{C}_i)^2$$
Then, particularizing to a certain grid cell, $i$, and to a certain bounding box, its double contribution will be given by:
$$(C_i^{\{0\}} -\hat{C}_i^{\{0\}})^2 + (C_i^{\{1\}} -\hat{C}_i^{\{1\}})^2$$
Where the superscripts ${0}$ and ${1}$ have been used again to address this specific situation.

Note: Delving into this question I realized that the previous arguments are just a interpretation of what the algorithm would do in this specific situation. I think that another plausible methodology to follow on this situation would be to just use the ground-truth object with the highest $\text{IOU}$.


Need to find out algorithim for entity recognition I am new to named entity recognition. I have a requirement to extract some specific data from corpus.
Two corpus example:

Hi, Rajesh Das want to buy this car. Rakesh Das will use this car.
Hi, Rakesh Das is brother of Rajesh Das. Rajesh Das will buy this car.

From these above example I need to extract the buyer name. So obviously the name which is near to buy word will be buyer name. Here position of Rajesh Das is different for two corpus. I can not figure out which algorithm need to implement for extracting buyer name.
I followed the https://towardsdatascience.com/named-entity-recognition-with-nltk-and-spacy-8c4a7d88e7da post which implement SpaCy and NLTK. It only provides all name list. Can someone provides me the idea how to get buyer name.

For this kind of task if try to build your custom ner model it would be much better. For references you can check my kaggle notebook for custom model of ner which will give you head start in designing your own dataset & preparing dataset for modeling.

Kaggle Notebook Link


Error in using fit() on RandomForest Classifier where X was a pandas.DataFRame object On using fit() method on sklearn.ensemble.RandomForestClassifier I am getting a value error that says.

ValueError: could not convert string to float: 'male'

The data-set used is the one in Titanic:Machine Learning from Disaster competition on Kaggle.
Here is the link- https://www.kaggle.com/c/titanic
Can someone please help me how to deal with this, why is it occurring and how to prevent it in future.

Note-There are no NaN in my DataFrame for train_X, i.e I have replaced all NaN with df.fillna(df.mean()), also I cross-checked that no NaN values exist by using

train_X.isnull().sum()

where, train_X is the training data for features.
Please Help!!

You can't pass a categorical variable as it is to one of sklearn's classifiers. One approach for dealing with this is to dummy-encode the column in question.

I realise that it's not a strict requirement to post minimal, complete, verifiable code on this site, but it will help if you provide code that can be run by others so that they can go straight to the solution without having to guess what's happening from your error message.

Here's some code which reproduces your error:

import pandas as pd
from sklearn.ensemble import RandomForestClassifier

df = pd.DataFrame({'sex': ['male', 'female', 'female', 'male', 'female'], 'survived': [0, 1, 1, 0, 1]})
rf = RandomForestClassifier()
rf.fit(df.drop('survived', axis=1), df['survived'])

We can fix the error by using the get_dummies function from pandas. The following code gives no errors:

df_dummies = pd.get_dummies(df)
rf.fit(df_dummies.drop('survived', axis=1), df_dummies['survived'])


What are the simplest methods for the label noise problem? If I have enough low quality data from unsupervised methods or rule-based methods.

I read from https://github.com/subeeshvasu/Awesome-Learning-with-Label-Noise ,but these methods are a little complex for me.

In detail, I deal with a multi-label classification task. First I crawl web page such as wiki and use regex-based rule to mark the label. The model input is the wiki title and the model output is the rule-matched labels from wiki content. My task is to predict the labels for the wiki title.

Do you think removing the wrong data predicted by trained model is a simple but effective method?

If the labeling people is sufficient,
the best solution is to re-label the noisy data.


sklearn.feature_selection vs xgboost feature_importances? sklearn.feature_selection vs xgboost feature_importances

Can somebody explain in-detailed differences between sklearn.feature_selection and xgboost feature_importances? And how the algorithms work under the hood?

Which module gives the best results?

They are very different things. sklearn.feature_selection is a general library to perform feature selection. xgboost feature_importances is simply description of how each feature is important (for details should refer in the xgboost documentations) in regards with model-fitting procedure and it is simply attribute and it is up to you how you can use this importance.

You can however, apply feature selection on top of xgboost model or models that follow some level of sklearn models api structure, for example using sklearn.feature_selection.RFE .


how to print kmeans cluster python Data-set has 3 features. The number of clusters are two.

I am figuring out how to print clusters using scatter plot for the data having 3 feature column and clustered into 2 clusters using kmeans.

The train data is in dataframe format and data is of activity data set:

X =  pd.concat([train_data['start'], train_data['end'], train_data  ['duration']], axis=1)
kmeans.fit(X)
Y_pred = kmeans.predict(X)
plt.scatter(X.iloc[:,0], X.iloc[:,1], c=Y_pred, cmap=plt.cm.Paired)
plt.legend()
plt.title('train data')
plt.show()
getting following output:

Code Is Self Explanatory...

    from sklearn.cluster import KMeans
    from sklearn.datasets.samples_generator import make_blobs
    np.random.seed(0)
    centers = [[1, 1], [-1, -1]]
    n_clusters = len(centers)
    X, labels_true = make_blobs(n_samples=3000,
                                centers=centers,
                                cluster_std=0.5)

def plot_cluster_data(X, c=[1]*X.shape[0], mu=None):
    fig = plt.figure(figsize=(8, 8))
    ax = fig.add_subplot(1, 1, 1)
    if len(np.unique(c)) == 1:
        ax.plot(X[:,0], X[:,1], 'o')
    else:
        ix = np.where(c==1)
        ax.plot(X[ix,0], X[ix,1], 'o',
                markerfacecolor='red')
        ax.plot(mu[0,0], mu[0,1], 'o',
                markerfacecolor='red',
                markersize=12)
        ix = np.where(c==0)
        ax.plot(X[ix,0], X[ix,1], 'o',
                markerfacecolor='green')
        ax.plot(mu[1,0], mu[1,1], 'o',
                markerfacecolor='green',
                markersize=12)
    if not mu is None:
        ax.plot(mu[0,0], mu[0,1], 'o',
                markerfacecolor='red',
                markersize=12)
        ax.plot(mu[1,0], mu[1,1], 'o',
                markerfacecolor='green',
                markersize=12)
    plt.show()

plot_cluster_data(X)

clst = KMeans(n_clusters=2, random_state=2342)
clst.fit(X)
mu = clst.cluster_centers_
plot_cluster_data(X, mu = mu)


text classification : comparing classification reports I have a 4-leabelled text classification problem.

Could someone help me choose among the below text classifiers ?

I was advised to select the second one ( the one which uses both unigrams and bigrams ) but I cannot really see why.

Okay so keeping it very short and precisely in context of your question-

Accuracy tells us, out of all the documents how many are classified correctly.

Precision tells us out of all documents which are predicted in a category, how often its correct.

Uni -gram- "nasa", "is" "space" , "agency"
bi-gram- "nasa is", "space agency"

Now lets go over the numbers, in both the cases accuracy and precision doesn't have significant difference.

But as  we can see bi-grams can give me much more information and hence can have better performance on unseen data. Try to test the model on unseen data/validation set and compare the difference.May be Try tri-grams etc also.


Parameter Tuning by Cross Validation for Random Forest I train a binary random forest classifier on scikit-learn's 20 newsgroups dataset. I want to tune the parameters and try so by gridsearch and 3-fold cross validation on the training data.

Is there any problem with this methodology?

For the max_depth parameter I get a really high value of 500 and that seems too high. Any advice?

The code is:

from __future__ import print_function
import sklearn
import sklearn.ensemble
import sklearn.metrics
from sklearn.datasets import fetch_20newsgroups
from sklearn.grid_search import GridSearchCV

categories = ['sci.med', 'soc.religion.christian']
newsgroups_train = fetch_20newsgroups(subset='train', categories=categories,
remove=('headers', 'footers', 'quotes'))
newsgroups_test = fetch_20newsgroups(subset='test', categories=categories,
remove=('headers', 'footers', 'quotes'))
class_names = ['medicine', 'christian']

vectorizer =
sklearn.feature_extraction.text.TfidfVectorizer(lowercase=False)
train_vectors = vectorizer.fit_transform(newsgroups_train.data)
test_vectors = vectorizer.transform(newsgroups_test.data)

rf = sklearn.ensemble.RandomForestClassifier(max_features='sqrt')

param_grid = {
           "n_estimators" : [10, 100, 1000],
           "max_depth" : [5, 100, 500],
           "min_samples_leaf" : [1, 20, 40]}

CV_rf = GridSearchCV(estimator=rf, param_grid=param_grid)
CV_rf.fit(train_vectors, newsgroups_train.target)
print(CV_rf.best_params_)

Are you looking at the accuracy on your validation set, rather than your training set? (you should be). Are you making sure your gap between training and validation accuracy is low? (you should be!) Is there enough data to warrant 3-fold cross-validation, or should you do 10-fold (and use more data for training)?

In general, random search (where you sample randomly from the parameter space) will get you a good result faster than grid search.  It has the added benefit that you can specify how many models you want to build, as each model's parameters are sampled independently and not constrained to cover an entire space like grid search.


How to Use Shap Kernal Explainer with Pipeline models? I have a pandas DataFrame X. I would like to find the prediction explanation of a a particular model.

My model is given below:

pipeline = Pipeline(steps= [
        ('imputer', imputer_function()),
        ('classifier', RandomForestClassifier()
    ])
x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=0)
y_pred = pipeline.fit(x_train, y_train).predict(x_test)

Now for prediction explainer, I use Kernal Explainer from Shap.

This is the following:

# use Kernel SHAP to explain test set predictions
shap.initjs()

explainer = shap.KernelExplainer(pipeline.predict_proba, x_train, link="logit")

shap_values = explainer.shap_values(x_test, nsamples=10)

# # plot the SHAP values for the Setosa output of the first instance
shap.force_plot(explainer.expected_value[0], shap_values[0][0,:], x_test.iloc[0,:], link="logit")

When I run the code, I get the error:

ValueError: Specifying the columns using strings is only supported for pandas DataFrames.

Provided model function fails when applied to the provided data set.

ValueError: Specifying the columns using strings is only supported for pandas DataFrames

Can anyone please help me? I'm really stuck with this. Both x_train and x_test are pandas data frames.

The reason is kernel shap sends data as numpy array which has no column names.
so we need to fix it as follows:

def model_predict(data_asarray):
    data_asframe =  pd.DataFrame(data_asarray, columns=feature_names)
    return estimator.predict(data_asframe)

Then,

shap_kernel_explainer = shap.KernelExplainer(model_predict, x_train, link='logit')
shap_values_single = shap_kernel_explainer.shap_values(x_test.iloc[0,:])
shap.force_plot(shap_kernel_explainer.expected_value[0],np.array(shap_values_single[0]), x_test.iloc[0,:],link='logit')


Can we use HDFS and big data Analytics for processing huge log files being processed through some application on some central server? Detailed Question Explanation:

Suppose say our application X is processing huge logs (size varying from MBs to GBs) and giving insight results in these logs(NOT A Social Data logs or Security Logs)

now this logs are in format say log.y with different variety, using C++ as Engine to process these huge logs.(It generates imp. insights about data but need to be processed using our application X only and we don't want to change core way processing of application X)

If this processing happens on some server it under or over utilizes resources (That I already know).

If we use cloud computing for this processing we get that processing power with optimum usage.

How do we see help of BIG data analytics in this particular sort of usage?
Any help or suggestion is very deeply appreciated

If the size of your logs are still growing then a distributed data system is definitely the right way to go. I have been using Mesos in production for almost a year now and it solved the problem of if this processing happens on some server it under or over utilizes resources.

I probably would look into some stacks like this:

Mesos as your fault-tolerant and elastic distributed systems
Spark or some Hadoop-based solutions for log processing and store the output in a DFS like HDFS
Have your applications consume data stored in HDFS as final steps


Is this approach for Active Learning correct? Let's say that we have 10,000 unlabeled documents, and we want to use pool-based sampling with batch size of 5. And we will use least confidence (LC) for querying strategy.

Is this outline of active learning reasonable:

1) Randomly select 30 documents, and get them labelled:=> labelled: 30, unlabelled: 9970.

2) Train the model on 30 labelled data

3) Use the model to make predictions for 9970 unlabelled documents

4) Use LC to select 5 out of all predicted documents that the model is least certain about

5) Get labels for the 5 documents:=> labelled: 35, unlabelled: 9965

6) Retrain the model on labelled data

7) Keep doing steps 2 - 6 until, the labeling resource expires or other stopping criteria is met.

Is this correct? I am mostly concerned about step 3) Should we retrain on all the data on every iteration? It seems like it could get computationally slow, but on the other hand if we don't evaluate every document how would we know which one is the most helpful to get labelled?

Your flow is correct. Model is retrained on new labeled data. Otherwise, the next candidates for labeling will be selected from the same region as previous candidates on which the model is least certain. By training on new labeled data, model will move on to new regions about which is least certain. Here is the diagram for pool-based method from page 5 of this survey on active learning:


How to make this plot with python I want to draw plot by python based on this data

Id     Duration (s)       BinaryLabel
1          5                 correct
2          4                incorrect
3          2                incorrect
4          3                incorrect
5          9                 correct
6          6                 correct
7          12               incorrect

My data is with 800 rows. and the duration is going from 0 to 55 seconds

My X axis contain duration column and I want it to be divided like: 0-----5-----10------15------20-----...-----55

and the Y should contain the percent of 'correct' values.

I.e. :

from 0 to 5, I will have 25% correct,

from 5 to 10 I will have 100% correct .etc.

Maybe you can modify my code:

   import numpy as np

   Duration_ = np.random.choice(range(0,56), 10000)

   Binary_Level = []

   b_ =["correct", "incorrect"]

   Binary_Level = np.random.choice(b_, 10000)

   import pandas as pd

   dd = pd.DataFrame({'Duration_': Duration_, 'Binary_Level':
      Binary_Level})

   xr_ = list(range(0,56,5))

   y_ = []

   for i in range(0,(len(xr_)-1)):

       a_ = np.logical_and(dd['Duration_'].values>=xr_[i],
       dd['Duration_'].values<xr_[i+1])
       b_ = np.logical_and(np.logical_and(dd['Duration_'].values>=xr_[i],
       dd['Duration_'].values<xr_[i+1]),
                               dd['Binary_Level'].values=='correct')
       y_.append(sum(b_)/sum(a_))

    import matplotlib
    matplotlib.pyplot.plot(xr_[1:12], y_)


Predict a sequence given many sequences I'm trying to find an algorithm that would fit this use case:

My data: a bunch of fixed-size integer arrays, e.g.

[0,2,3,4,5]
[1,2,3,1,5]
[4,1,2,4,5]
...

Input: an array of integers, and output: prediction for the rest of the array, e.g.

[0,1] -> [2,4,2]
[3] -> [1,3,2,5,5]
[3,2,4,1] -> [4]

So, the question is how to model the following problem: input a sub section of a sequence, then output the rest of the sequence. This is a sequence-to-sequence problem.
For this, as Derek O has eluded to, I would suggest a encoder-decoder architecture. In the encoder, you would encode your input sequence (using RNN/LSTM), which will give you a "hidden representation", then pass your hidden representation through a decoder (RNN/LSTM) to then decode the hidden representation into the resining sequence of integers.
Here is an article that provides more detail around encoder-decoder models: https://towardsdatascience.com/understanding-encoder-decoder-sequence-to-sequence-model-679e04af4346


Prediction after one hot encoding I have a regression model that I want to make prediction based on values that I will get from an end user.
In my dataset, I have one categorical variable region which I one-hot encoded, which generated 53 new columns (54 regions).
Now my data has the shape 1000x72. I then split into training and testing sets and my model is working fine.
But I'm confused about how my model would predict new values. Since I will only be getting one value for region from the end user, my model will one-hot encode a single value, and it will no longer fit the shape it has been trained on, as it will have the shape 1x18. I'm really confused as in how would I fit it into the model this way... Do I just make 53 other columns and put a dummy 0 in each one??
Sorry if this is a trivial question, I'm very beginner to this and any help would be greatly appreciated!!

region_ohe = OneHotEncoder(categories = "auto", handle_unknown = "ignore")
X_encoded = region_ohe.fit_transform(df['region'].values.reshape(-1,1)).toarray()

With sklearn's OneHotEncoder, the categories are baked in after fitting. You can apply the encoding to new data with region_ohe.transform(x_new).  (And, as you might guess,  fit_transform just calls fit then transform.)


Fitting ANN model to my dataset import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

#Importing Dataset

dataset = pd.read_csv('C:/Users/Rupali Singh/Desktop/ML A-Z/Machine Learning A-Z Template Folder/Part 8 - Deep Learning/Section 39 - Artificial Neural Networks (ANN)/Churn_Modelling.csv')
print(dataset)
X = dataset.iloc[:, [3, 13]].values
Y = dataset.iloc[:, 13].values
print(X)

#Categorical Data

from sklearn.preprocessing import LabelEncoder, OneHotEncoder
labelencoder1 = LabelEncoder()
X[:, 1] = labelencoder1.fit_transform(X[:, 1])
try:
    labelencoder2 = LabelEncoder()
    X[:, 2] = labelencoder2.fit_transform(X[:, 2])
except IndexError: pass

#Dummy Variable
onehotencoder = OneHotEncoder(categorical_features=[1])
X = onehotencoder.fit_transform(X).toarray()

#Splitting the dataset into training set and test set

from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=0)
print(Y_test)

#Feature Scaling

from sklearn.preprocessing import StandardScaler
feature_scaling = StandardScaler()
X_train = feature_scaling.fit_transform(X_train)
X_test = feature_scaling.transform(X_test)
print(X_train)

#Importing Keras libraries and packages

import keras
from keras.models import Sequential
from keras.layers import Dense

#Initialising the ANN

classifier = Sequential()

#Adding the input layer and hidden layer
classifier.add(Dense(input_dim=11, units=6, kernel_initializer='uniform', activation='relu'))

#Adding the second hidden layer
classifier.add(Dense(units=6, kernel_initializer='uniform', activation='relu'))

#Adding the Output Layer
classifier.add(Dense(units=1, kernel_initializer='uniform', activation='sigmoid'))

#Compiling the ANN(Applying Stochastic Gradient)
classifier.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

#Fitting ANN to training set
classifier.fit(X_train, Y_train, batch_size=10, nb_epoch=100)
# predicting the test result
Y_pred = classifier.predict(X_test)
Y_pred = (Y_pred > 0.5)

# Making the confusion Matrix
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(Y_test, Y_pred)
print(cm)

I am trying to fit the ANN model to my dataset, after preprocessing of data and importing the keras library, the following error comes during model fitting.

Traceback (most recent call last):
  File "<input>", line 2, in <module>
  File "C:\Users\Rupali Singh\PycharmProjects\Machine_Learning\venv\lib\site-packages\keras\engine\training.py", line 952, in fit
    batch_size=batch_size)
  File "C:\Users\Rupali Singh\PycharmProjects\Machine_Learning\venv\lib\site-packages\keras\engine\training.py", line 751, in _standardize_user_data
    exception_prefix='input')
  File "C:\Users\Rupali Singh\PycharmProjects\Machine_Learning\venv\lib\site-packages\keras\engine\training_utils.py", line 138, in standardize_input_data
    str(data_shape))
ValueError: Error when checking input: expected dense_1_input to have shape (11,) but got array with shape (3,)

I cannot understand what is wrong, please help.

You ANN is expecting an input of size 11 as you specified with the parmaters input_dim

classifier.add(Dense(input_dim=11, units=6, kernel_initializer='uniform', activation='relu'))

But your training set is sending array of size 3.
It's probably due to tour dummy variables :

#Dummy Variable
onehotencoder = OneHotEncoder(categorical_features=[1])
X = onehotencoder.fit_transform(X).toarray()

you can set your ANN to get correct input:

classifier.add(Dense(input_dim=3, units=6, kernel_initializer='uniform', activation='relu'))


Creating obligatory combinations of variables for drawing by random forest Problem

For my machine learning task, I create a set of predictors.
Predictors come in "bundles" - multi-dimensional measurements (3 or 4 - dimensional in my case).

The hole "bundle" makes sense only if it has been measured, and taken all together.

The problem is, different 'bundles' of predictors can be measured only for small part of the sample, and those parts don't necessary intersect for different 'bundles'.

As parts are small, imputing leads to considerable decrease in accuracy(catastrophical to be more accurate)

Possible solutions

I could create dummy variables that would mark whether the measurement has taken place for each variable. The problem is, when random forests draws random variables, it does so individually.

So there are two basic ways to solve this problem:
1) Combine each "bundle" into one predictor. That is possible, but it seems information will be lost.
2) Make random forest draw variables not individually, but by obligatory "bundles".

Problem for random forest

As random forest draws variables randomly, it takes features that are useless (or much less useful) without other from their "bundle". I have a feeling that leads to a loss of accuracy.

Example

For example I have variables a,a_measure, b,b_measure.
The problem is, variables a_measure make sense only if variable a is present, same for b. So I either have to combine aand a_measure into one variable, or make random forest draw both, in case at least one of them is drawn.

Question

What are the best practice solutions for problems when different sets of predictors are measured for small parts of overall population, and these sets of predictors come in obligatory "bundles"?

Thank you!

You may want to consider gradient boosted trees rather than random forests. They're also an ensemble tree-based method, but since this method doesn't sample dimensions, it won't run in to the problem of not having a useful predictor available to split on at any particular time.

Different implementations of GBDT have different ways of handling missing values, which will make a big difference in your case; I believe R does ternary splits which is likely to work fine.


Excel Data to Venn Diagram I don't really know if I'm posting in the right place but I hope so.

I'm writing my bachelor degree, and I collected some data about fonts and language support.

At the moment, all my data is stored in an excel spreadsheet, in the form of:

Column: language support (e.g. Cyrillic, Cyrillic extended, Arabic, Vietnamese, Devanagari, etc)
Rows: names of the fonts that supports that language, up to 890

Note that some of there fonts, even though they occur only one per columns, they may appear several times in different columns (e.g. Noto Sans will appear in Latin, Latin Extended, Cyrillic, Cyrillic Extended, Simplified Chinese and many others; while others e.g. Padauk will only appear under Myanmar since that font may not support all Latin characters).

What I would like to do is to put these data in visual form, ideally via a Venn Diagram that displays how much each language is supported (area) and how many language are supported at the same time by a given font (opacity overlapping). I don't need to process this plotting step inside excel, I just need to get the data of how many fonts per intersection are there (e.g. latin and cyrillic gives 100, latin and arabic gives 80, latin and arabic and cyrillic gives 10 ... up to latin and cyrillic and arabic and devanagari and [n] gives 0).

I thought I could have parsed these data with javascript and json but I can't get to properly convert my excel file to json in a fashion that I can really use.

Any suggestion in warmly welcomed and I really thank you in advance

If you are famialiar with R, you can use this library: VennDiagram
Without glimpse of your data its hard to help more, but with this library you can build simple Diagrams like this :

Or something insane like this:

You can find boht examples in the library code examples.
Hope this helps!


Plot Two Categorical Variables How to plot two categorical variables in Python or using any library? I want to plot the Playing Role of a Cricketer (Batsman, Bowler, etc.) VS Bought_By (Franchise Names, e.g., CSK, DC, etc.). The logic here is to plot the cricket role vs franchise.
The Columns:
df.Playing_Role
df.Bought_By
One of these columns can be converted to continuous numerical, but is there any direct way without converting them?

Well, there are a few ways to do the job. Here are some I thought of:

Scatterplots with noise:
Normally, if you try to use a scatter plot to plot two categorical features, you would just  get a few points, each one containing a lot of instances from the data. So, to get a sense of how many there really are in each point, we can add some random noise to each instance:

import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

# This is to encode the data into numbers that can be used in our scatterplot
from sklearn.preprocessing import OrdinalEncoder
ord_enc = OrdinalEncoder()
enc_df = pd.DataFrame(ord_enc.fit_transform(df), columns=list(df.columns))
categories = pd.DataFrame(np.array(ord_enc.categories_).transpose(), columns=list(df.columns))

# Generate the random noise
xnoise, ynoise = np.random.random(len(df))/2, np.random.random(len(df))/2 # The noise is in the range 0 to 0.5

# Plot the scatterplot
plt.scatter(enc_df["Playing_Role"]+xnoise, enc_df["Bought_By"]+ynoise, alpha=0.5)
# You can also set xticks and yticks to be your category names:
plt.xticks([0.25, 1.25, 2.25], categories["Playing_Role"]) # The reason the xticks start at 0.25
# and go up in increments of 1 is because the center of the noise will be around 0.25 and ordinal
# encoded labels go up in increments of 1.
plt.yticks([0.25, 1.25, 2.25], categories["Bought_By"]) # This has the same reason explained for xticks

# Extra unnecessary styling...
plt.grid()
sns.despine(left=True, bottom=True)

2. Scatterplots with noise and hues:
Instead of having both axis being feature we can have the $x$ axis be one feature and
the $y$ axis be random noise. Then, to incorporate the other feature, we can "colour
in" instances based on the other feature:
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

# Explained in approach 1
from sklearn.preprocessing import OrdinalEncoder
ord_enc = OrdinalEncoder()
enc_df = pd.DataFrame(ord_enc.fit_transform(df), columns=list(df.columns))
categories = pd.DataFrame(np.array(ord_enc.categories_).transpose(), columns=list(df.columns))

xnoise, ynoise = np.random.random(len(df))/2, np.random.random(len(df))/2

sns.relplot(x=enc_df["Playing_Role"]+xnoise, y=ynoise, hue=df["Bought_By"]) # Notice how for hue
# we use the original dataframe with labels instead of numbers.
# We can also set the x axis to be our categories
plt.xticks([0.25, 1.25, 2.25], categories["Playing_Role"]) # Explained in approach 1

# Extra unnecessary styling...
plt.yticks([])
sns.despine(left=True)

Catplots with hues:
Finally, we can use catplots, and colour in fractions of it based on the other feature:

import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

sns.histplot(binwidth=0.5, x="Playing_Role", hue="Bought_By", data=df, stat="count", multiple="stack")


ML algorithm to predict the state of state of a device (ON/ OFF) based on day, time and location I require to develop an machine learning algorithm to predict that if a secondary car battery should be connected to load at any given time of day based on the automobile usage profile of the user. (the output is either 1 or 0 , ON or OFF)

Some Background : My project car is equipped with a secondary battery which has to run a car Computer, LTE modem etc. So it is important that the battery is not connected to these loads all the time so as to extend the run time of the battery when the car is not in use( primary car battery charges the secondary battery when the car is running.) . These electronic parts are used for IOT operation, running alexa voice interaction etc.

So it is necessary to predict at what times user is using his car and switch the circuit ON before the driver actually gets in the car, So that it can start alexa voice interaction  with user as soon as he enters the car without any delay( if it starts after user gets in , it will take some time before all the internet connections are up and running.).

Any suggestions on which existing machine learning algorithm I can use here? For the start I’m planning to predict the state based on the day, time and location only.  For example if the time is 7.am, weekday, and the location is home,  the car is usually ON to travel to office., if the time is 10.00am and the location is Office , the car is usually OFF.

Please help me with any  ideas/ suggestions that can also  improve the concept.

You need to predict 2 classes (on/off). The first question is: Based on what data (exactly) this happens?

Problem statement

You do have some training data, so this is a supervised learning task. The algorithm choice really depends on the question what data exactly to need and what kind of patterns you expect to detect:

is the prediction based on a single user (data will be scarce then) or on multiple users (enough to determine similar users for predictions)
does it really suffice to consider Daytime + Week/Weekend? What about public holidays or seasons, where the usage behavior likely changes
Will vacation times impact the usage behaviour beyond date/time?

Picking an algorithm

Once you now based on which data you will recognise what kind of pattern you think about algorithms that fit well. For example: If you really only need time + weekday/weekend then very simple algorithms might do. Decisions trees for example (with simple rules as if weekday + 7.a.m. then turn on).

If rules need to be a lot more complex (and I think this is the case here) you might need a more complex model. I have very little experience with time based predictions but things like recurring neural networks come to mind which are heavily used for time based predictions (e.g. web analytics in order to find anomalies in usage patterns for any given point in time).

Concerns

Your problem sounds like you need a very accurate prediction (minute-wise accuracy) while the actual user behaviour will be a bit more random (some times earlier, some times later). Many algorithms try to reduce an error which often leads to in-between values if the actual time of an event fluctuates around a mean value.

When using neural networks you can influence how the error is calculated and you can train it in order to favour one kind of error (being early) over another. This makes it more complex, and requires that being "too early" is not an issue (or that the user is always very reliable).


Can we take of benefit of using transfer learning while training a word2vec models? I am looking to find a pre-trained weights of an already trained models like Google News data etc. I found it hard to train a new model with enough amount (10 GB etc) of data for myself. So, I want to take benefit from transfer learning in which I would able to get pre-trained layer weights and retrain those weights on my domain specific words. So, definitely it will take relatively less time in training. Any sort of help will be highly appreciated.
Thanks in advance :)

Yes, you can take benefit of pre-trained models. Most famous one being the GoogleNewsData trained model which you can find here.

Pre-trained word and phrase vectors https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing

You can then load the vectors in binary format in your model using gensim as shown below.

>>> model = Word2Vec.load_word2vec_format('/tmp/vectors.txt', binary=False)  # C text format
>>> model = Word2Vec.load_word2vec_format('/tmp/vectors.bin', binary=True)  # C binary format

Here is a different pre-built model for English Wikipedia:

https://github.com/idio/wiki2vec/raw/master/torrents/enwiki-gensim-word2vec-1000-nostem-10cbow.torrent

Source: https://github.com/idio/wiki2vec/

Using a prebuilt model

Get python 2.7
Install gensim: pip install gensim
uncompress downloaded model: tar -xvf model.tar.gz
Load model in gensim:
from gensim.models import Word2Vec
model = Word2Vec.load("path/to/word2vec/en.model")
model.similarity('woman', 'man')

You can also use Stanford NLP Glove

http://nlp.stanford.edu/projects/glove/
https://github.com/stanfordnlp/GloVe

Here is a great compilation of pre-trained word2vec models.

https://github.com/3Top/word2vec-api#where-to-get-a-pretrained-models

Some additional pre-trained models:

http://www.zuccon.net/ntlm.html
https://code.google.com/archive/p/word2vec/

More on gensim and code here:  https://radimrehurek.com/gensim/models/word2vec.html

Quora forum with a similar questions

https://www.quora.com/Where-can-I-find-pre-trained-models-of-word2vec-and-sentence2vec-trained-on-Wikipedia-or-other-large-text-corpus
https://www.quora.com/Where-can-I-find-some-pre-trained-word-vectors-for-natural-language-processing-understanding


MLP sequential fitting I am fitting a Keras model, using SGD
Input dataset X_train has 55000 entries.

Can anyone explain the yellow highlighted values?
For me, when each epoch is done, this should correspond to 55000/55000.
model = keras.models.Sequential()
model.add(keras.layers.Flatten(input_shape=[28,28]))
model.add(keras.layers.Dense(300, activation="relu"))
model.add(keras.layers.Dense(100, activation="relu"))
model.add(keras.layers.Dense(10, activation="softmax"))

model.compile(loss="sparse_categorical_crossentropy", optimizer="sgd",metrics=["accuracy"])

history = model.fit(X_train , y_train, epochs=30 , validation_data=(X_valid, y_valid))

These numbers refer to minibatches, not individual samples. The data is not fed to the model one by one, but in small groups called "minibatches" or simply "batches". The size of the minibatch (the number of elements to be included in each minibatch) can be specified as a parameter to the fit method. As you did not provide any value, it took the default value of 32. This is specified in the documentation:

batch_size: Integer or None. Number of samples per gradient update. If unspecified, batch_size will default to 32. Do not specify the batch_size if your data is in the form of datasets, generators, or keras.utils.Sequence instances (since they generate batches).

Your data size divided by the batch size gives us the number of batches, which is the number you see: ceil(55000 / 32) = 1719. As they are not divisible, the last batch would have a few elements less; specifically, the last batch has 24 elements (= 55000 - 32 * 1718) instead of 32.


Time series forecast for small data set I am new in data science so please accept my apology in advance if my question sounds stupid. I want to do a time series forecast of outage mins in the current regulatory year. The regulatory year starts from 1 April and ends on 30 March of next year. I have data of around six months i.e. from April to September. Outage does not occur every day. So I have only 144 data points (or days out of 171 days) where the outage occurred. I have plotted the data in the following graph. The graph shows the cumulative sum of outage mins.

Now I am trying to predict the value from October to March. I wanted to forecast the value that what would be the cumulative outage mins by the end of March next year. I tried to use Exponential smoothing but it did not work, it may be because I don't have a lot of observation. Then I was reading about ARIMA but not sure whether its the right algorithm to use or not as I don't think that there would be any seasonality in this scenario and also I don't have long data points. Could anyone help with which algorithm should I use to forecast the value? I am using Python as a programming language. Any help would be really appreciated.

ARIMA could work, I think it's the right approach. It's simple enough to be used on a small dataset, but sufficiently flexible at the same time. If you are using Python, library statsmodels allows you to implement ARIMA regressions. You have to grid search and find the right parameters to find the best fit, and run the prediction.
If you want to know how to do it, take a look here and here.
Alternatively, even simpler models could work correctly, such as a simple moving average (MA), or auto-regression (AR). But that's something you can find from the ARIMA grid search above.


Organized processes to clean data From my limited dabbling with data science using R, I realized that cleaning bad data is a very important part of preparing data for analysis.

Are there any best practices or processes for cleaning data before processing it? If so, are there any automated or semi-automated tools which implement some of these best practices?

R contains some standard functions for data manipulation, which can be used for data cleaning, in its base package (gsub, transform, etc.), as well as in various third-party packages, such as stringr, reshape/reshape2, and plyr/dplyr. Examples and best practices of usage for these packages and their functions are described in the following paper: http://vita.had.co.nz/papers/tidy-data.pdf.

Additionally, R offers some packages specifically focused on data cleaning and transformation:

editrules (http://cran.r-project.org/web/packages/editrules/index.html)
deducorrect (http://cran.r-project.org/web/packages/deducorrect/index.html)
StatMatch (http://cran.r-project.org/web/packages/StatMatch/index.html)
MatchIt (http://cran.r-project.org/web/packages/MatchIt/index.html)
DataCombine (http://cran.r-project.org/web/packages/DataCombine)
data.table (http://cran.r-project.org/web/packages/data.table)

A comprehensive and coherent approach to data cleaning in R, including examples and use of editrules and deducorrect packages, as well as a description of workflow (framework) of data cleaning in R, is presented in the following paper, which I highly recommend: http://cran.r-project.org/doc/contrib/de_Jonge+van_der_Loo-Introduction_to_data_cleaning_with_R.pdf.


k-mean without label i m try to apply k-means with Python 3 to my dataset (Amazon review)  for classify similar user (from review).

I just have a TF and TF-IDF matrix (and i have a row(user) and columns(words) value in distinct csv file)

I m starting with sklearn from this sketch

https://scikit-learn.org/stable/auto_examples/text/plot_document_clustering.html#sphx-glr-auto-examples-text-plot-document-clustering-py

but in my case i don't have label or category, and i don't understand how k-means can cluster different items from this matrix TD-IDF.

How can I group similar words from my dataset, from the matrix
without having any information?
How do I show these n-clusters in the chart?
And how do I show the similar words of this cluster, if I do not
have any information (like label or category)?

For now this is my code :

k = 5
km = KMeans(n_clusters=k, init='k-means++', max_iter=100, n_init=5)
km.fit(Y) ##Y is my TD-IDF matrix

original_centroids = svd.inverse_transform(km.cluster_centers_)
print(original_centroids.shape)
for i in range(original_centroids.shape[0]):
original_centroids[i] = np.array([x for x in original_centroids[i]])
svd_centroids = original_centroids.argsort()[:, ::-1]

i would like have a set like this(with similar word near)

I m very lost, because i m new in this word.

K-means does not use labels.

The example that you looked at uses labels to compare the clusters to the labels. That part obviously requires labels; but that is not part of k-means.


How to create clusters based on sentence similarity? I have data which looks like following.
Data is a group of sentences which are similar, but have few unique words in between like TABLEA, TABLEB etc.

java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 3523] [SQLState 42000] The user does not have SELECT access to TABLEA
java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 3523] [SQLState 42000] The user does not have SELECT access to TABLEC
java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 3523] [SQLState 42000] The user does not have SELECT access to TABLEB
Dataframe read is null
Dataframe read is null
java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 3807] [SQLState 42S02] Object Y  does not exist.
java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 3807] [SQLState 42S02] Object Z  does not exist.
java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 2652] [SQLState HY000] Operation not allowed: TABLEK is being Loaded.
java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 9804] [SQLState HY000] Response Row size or Constant Row size overflow.
java.sql.SQLException: [Teradata JDBC Driver] [TeraJDBC 15.10.00.22] [Error 1000] [SQLState 08S01] Login failure for Connection to xxx.xx.xx.xx Tue Dec 04 02:49:47 MST 2018

Problem Statement:
I want to group/cluster the data and provide a unique number to each group/cluster.

Assumptions:

The groups/cluster should be formed based on the similarity. Similar sentences should be grouped in one cluster
This should be unsupervised learning. If in future, some new sentence comes which is very less similar to existing cluster, it should create a new group/cluster.
The sentences can be of any length
The common words between sentences can appear
anywhere - starting of string, in between, in the end, or so
The sequence of the words matter

Output:

The outcome should be a dimension table for category like below

Although I do get the problem statement in its abstract form, I do not know a concrete way to do this.

So far I have read about text clustering using various algorithms like cosine similarity and etc, but I am not sure if that will suffice this problem statement.
One of the major problem here is, it is unsupervised. If there are any new sentences whose similarity is very less then it should create a new group.

The bigger picture goes like this

Get list of all unclassified/uncategorised(I am using both interchangeably here since I am not sure which one it falls into) statements
Check in the dimension table, by matching using some similarity threshold(not clear on this).
If the similarity is matching above a threshold, then do nothing
If similarity is less, then create a new group in the dimension table with the Description column which has common words.

I have yet to identify what is the best approach to solve this problem.Please recommend some algorithm or approach to solve this problem.

For this problem, I would start with a simple bag of words model and use that as a baseline.

This is an example : https://medium.com/@MSalnikov/text-clustering-with-k-means-and-tf-idf-f099bcf95183 , http://brandonrose.org/clustering

For logs, this might be sufficient since it will easily filter out common words like date etc.

If this does not work well for a given usecase, next step is to try Doc2Vec and similar methods that project sentences to a vector space.


How relevant is Self Organizing Maps in today's science? Self-Organizing Maps is a pretty smart yet fast & simple method to cluster data. But Self-Organizing maps were developed in 1990 and a lot of robust and powerful clustering method using dimensionality reduction methods have been developed since then. What are some of its applications in today's world of science and engineering?

One important and well known (in their respective research domains) usage of SOMs today is the usage of SOMs for data visualization and clustering.

The U-Matrix uses a SOM with a lot of neurons in order to achieve emergent behavior. Based upon the SOM, a so called U-height is calculated for each neuron.

For a neuron $n$, a weight $w_n$ of the neuron $n$, as well as a set $NN(n)$ of immediate neighbours of $n$ the U-height is calculated as:

$$\sum_{m \in NN(n)} d(w_n - w_m)$$

$d$ is the same distance as used in the SOM training. This U-height is used to visualize the SOM and can be capable of catching quite interesting
inherent structure. It is visualized by using a color scheme similar to topographical maps:

The U-Matrix uses distances for visualization. It can be combined with the so called P-Matrix, which uses densities, to use the density of data points as well, and allows the combination of densities and distances. Based on the U-Matrix one can do classification or clustering as well. An R package is distributed by the research group of Prof. Ultsch.

I wouldn't say that it's important in Machine Learning per se, but in the fields of Data Mining and KDD it is, at least academically. You can clearly see this if you look on the reception of this or similar work on semanticscholar or researchgate.

Disclaimer: I studied with Prof. Ultsch, the head behind U-Matrix and U*-Matrix.

NB: Sorry for the missing links to the papers, I haven't got enough reputation yet to post more than two links.

Papers:

U*-matrix: a Tool to Visualize Clusters in High Dimensional Data, Ultsch 2004
Self-Organizing Neural Networks for Visualisation and Classification, Ultsch 1993

I even spotted the U-Matrix in the wild here. Also see the sompy package.


Can there be in-active neuron in output layer I am new to deep learning, and was studying about it. I know that input from input layer is multiplied with weights and then added with bias. And output of this is passed to a activation function of a neuron in hidden layer. Suppose if we use 'Relu' as activation function and our weighted sum is less than 0, then that particular neuron will not be activated right!
So similar to that what if some neuron in output layer have some value and that value is not accepted by a particular activation function means the value is below the threshold for that activation function, so will that particular neuron remain in-active.

Yes, it is pretty much possible that you can have an in-active neuron in an output layer.
FYI:
Nodes at the input layer called passive nodes
Nodes at hidden layer called active nodes or unobservable outputs
Nodes at output layer called observable outputs


Predicting of the function values There is a function: $f(t,x,y,z)$, where $t$- time, $x,y,z$- some arguments.

The values of $f$ for $t\in [a,b]$ are known (100 samples). What is the most accurate way of predicting the value of $f$ for $t=b+1$, if the values of $x,y,z$ for $t\in [a,b+1]$ are known?

May I implement deep-learning, and if yes, which method would be more reliable?

ADDITIONAL INFO

I'm propagating the satellite position using an analytical method. During the propagation, the errors accumulate. I endeavor to predict the future position errors. $f$ represents the error value and arguments $x,y,z$ are the affecting parameters, such as distance to the Moon, Sun, solar flux, atmospheric density, etc.

Before trying out deep learning I suggest you to try out Random Forests as the nature of your data seems non-linear to me and RFs are good at fitting non-linear data. If it doesn't work out well try doing a simple feed forward neural network with non-linear activation function as it also fits non linear functions really well. Also, machine learning is more about trying and testing as every dataset is different and you can never know in advance which model will work out well.


Please explain input shape in Keras My data consists of 2-dimensional arrays with shape (2,3).
The whole dataset (emp) consists of 12 items (I know that it is too small of a number for an NN training, but it is just a test), so it has shape (12,2,3)

truevals has shape (12,)

When I try to add Keras layer:

model = Sequential()
model.add(Dense(4, input_shape=(2,3), activation='tanh'))
model.add(Dense(1, activation='sigmoid'))
model.compile(SGD(lr=0.5), 'binary_crossentropy', metrics=['accuracy'])
model.fit(emp, truevals, epochs=200)

and then when I fit the model I get the exception:

  Error when checking target: expected dense_30 to have 3 dimensions,
  but got array with shape (12, 1).

What am I doing wrong?

Dense layers doesn't reduce dimensions of inputs, so if you provide a (12,2,3) input, it expects a (12,2,1) output (for your case).

If you want to make it work, you should flatten your input (or output of first layer) and provide a (12,1) output.


How to apply Machine learning model on time series to predict next time step I have done feature engineering on a single variable time series data (spare parts usage), then I turn the time series data into supervised machine learning problem. I have trained and test on the transformed new dataset. My question is how to apply the model to predict one time step, multi time steps into the future? All features are calculated over prior data points, such as lag features, moving window statistics, such as max, min, mean, median over for example, last 5, 10, 15 days.

If you have working examples in Python or any white papers with example, that will be great. Look forward to your help
Shaun

As your question is quite vague I just throw in some tools: MCMC, Kalman, LSTM, ARIMA, one-step-ahead and many more.
Have a look at this to get started.

Can you please provide a part of your data? There are a lot of tools but it depends a lot on your situation and data.


Orange reading in new features from Tab-Delimited File In my tab delimited file I have 65 features and 1 target variable that are numerical and continuous. My data is supposed to be used for regression models. There should be 65 features and 1 target variable in the columns when I upload the file to orange.

However,there are actually 68 features, 2 new features called "Feature 1" and "Feature 2" show up that shouldn't be there. Both of them are categorical. Why are these two new features showing up? I counted the amount of variables I put and the amount of domain and attribute types corresponding to each variable. They both add up to 66 so I don't know where the extra features are coming from.

This should usually not happen. It can only happen if there are some stray entries in columns, columns following the penultimate(last) column of your data. Please check. This usually happens while preparing data. Please let me know if I have solved your problem.


Training a model where each response in the observation data has a different known varience I have a dataset where each response variable is the number of successes of N Bernoulli trials with N and p (the probability of success) being different for each observation. The goal is to train a model to predict p given the predictors. However observations with a small N will have a higher variance and higher N.

Consider the following scenario to illustrate better: Assume coins with different pictures on them have a different bias and that the bias is dependent on the picture on the coin. I have a large number of coins each with a different picture on them and each with a different bias p. I want to create a model that can predict the bias of a coin given only the picture on the coin. I flip each coin a different number of times and record the number of successes and total number of flips. So my data set consists of each picture and its estimate p=successes/flips.

So my question is when training my model how should I handle this. It seems more weight should be given to observations with a higher sample size(number of flips). I don't think it makes sense to include number flips as a predictor variable because the point is to build a model which predicts p using only the picture on the coin so this difference in variance for the response for each observation should be taken into account when training the model.

I am using several types of model but mainly working with keras and xgboost

I may be understanding the question now.  Still using the coin example, as I said above, the number of trials for a given coin only affects the confidence for the estimated probability/bias for that one coin.  So it seems like you are asking how to incorporate that "confidence" into the response variable, if at all.  In other words, you are asking if your model should reflect the uncertainty concerning the true value of $p$ for each coin, given the number of coin flips you performed.

I don't think assigning different weights to observations is appropriate in this situation because, again, the number of flips for one coin does not have anything to do with other coins.

I am not sure if this will satisfy your needs, but there is something called interval regression that is used to model a response/dependent variable that is defined as an interval between lower and upper bounds.  It is a type of regression for censored data (a problem where the true value of response is not known) and is typically used for modeling such variables as income ranges or survival times.  In your case, you could compute a confidence interval for the true value of $p$ for each coin, using the $p$ calculated from your trials and the number of trials specific to each coin.  Then you would use this regression with two response variables: lower limit and upper limit of the confidence interval.

Based on my quick search, I am not finding a lot of Python support for this type of model, except:

Maximum Margin Interval Trees - Decision trees for interval regression

Drouin, A., Hocking, T.D. & Laviolette, F. (2017). Maximum Margin Interval Trees. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.

https://arxiv.org/abs/1710.04234

https://aldro61.github.io/mmit/


How to create a new dataframe using the another dataframe I have created and worked on a DataFrame for a project. It looks like the following:

Critics   Items    Ratings

a...........1..........5

b...........2..........3

b...........3..........2

c...........8..........1

a...........1..........5

b...........4..........4

My DataFrame has 1M+ rows and 8 columns.

I want to create a new DataFrame where the rows are the unique critics, the columns are the unique items, and the individual cells are the rating a critic has given for the particular item. If the critic has not reviewed the item then I want to add an NA over there.

I tried doing the following for the rows:

ratings = pd.DataFrame(f.review_profilename.unique())

For the columns, I saw a lot of answers involving people using

ratings.rename(<individual column names>, axis='columns')

But this doesn't help me since I can't list down all the unique item names.

Edit:
I fixed the issues by using pivot tables. I am new to pandas and was not aware of something like this existing. The exact syntax I used was

ratings = f.pivot_table(index = 'critic',columns = 'item', values = 'ratings')

Not sure what you mean by "unique", I guess if there're n critics, m items, what you need is a dataframe with shape n*m rows and 3 columns, right? If so, try the demo:

#1. the original dataframe
df = pd.DataFrame([['a',1,5],['b',2,3],['b',3,2],['c',8,1],['a',1,5]],columns=['critic','item','rating'])

#2. create the first two columns(critic, item) by their permutation
from itertools import product
first_two_column = pd.DataFrame(list(product(set(df.critic),set(df.item))),columns=['critic','item'])

#3. merge the first two column with ratings, using left join to add nan, drop duplicate first
first_two_column.merge(right=df.drop_duplicates(),on=['critic','item'],how='left')


data processing, correlation calculation I have product purchase count data which looks likes this:

user item1 item2
   a     2     4
   b     1     3
   c     5     6
   ...   ...   ...

These data are imported into python using numpy.genfromtxt. Now I want to process it to get the correlation between item1 purchase amount and item2 purchase amount -- basically for each value x of item1 I want to find all the users who bought item1 in x quantity then average the item2 over the same users. What is the best way to do this? I can do this by using for loops but I thought there might be something more efficient than that. Thanks!

Pandas is the best thing since sliced bread (for data science, at least).

an example:

import pd
In [22]: df = pd.read_csv('yourexample.csv')

In [23]: df
Out[23]:
   user   item1   item2
0     a        2      4
1     b        1      3
2     c        5      6

In [24]: df.columns
Out[24]: Index([u'user ', u'item1 ', u'item2'], dtype='object')

In [25]: df.corr()
Out[25]:
          item1      item2
item1   1.000000  0.995871
item2   0.995871  1.000000

In [26]: df.cov()
Out[26]:
          item1      item2
item1   4.333333  3.166667
item2   3.166667  2.333333

Bingo!


With Stochastic Gradient Descent why we dont compute exact derivative of loss function? In a blog I read this:
With Stochastic Gradient Descent we don’t compute the exact derivate of our loss function. Instead, we’re estimating it on a small batch.
blog.
Now I am confused with the whole concept.
Why we take estimate of the derivative? Please explain.

That's because the whole loss is $\frac{1}{N} \sum\limits_{i=1}^N L(x_i, y_i)$ and that number $N$ is the dataset size, it can be very large. It's just too slow to compute the true gradient, thus we compute its unbiased estimate via Monte Carlo. There are some theorems that say that stochastic gradient descent converges under certain conditions, so it's a reasonable method. You just don't to wait long (computing the true gradient), you can converge faster. The speed isn't the only reason. Also, researchers found out that using small batch size can improve the performance of neural networks and it's reasonable as well because the lower the batch size the higher is the variance of the estimate, and the higher variance (i.e. noise) and the higher variance prevents the net from overfitting.


"help" decision tree by tying 2 features together Assuming I have in my dataset 2 (or more) features that are for sure linked (for example: feature B indicates the amount of relevance of feature A), is there a way I could design a decision tree that will tie both features together (the obvious split would be if feature B is big enough split by A and if not ignore A)?
In other words: I'm looking for a way to "help" my decision tree to not test feature A before testing feature B. Plus, not to look on the improvement of each feature individually, but on the improvement of "B and afterwords A".
Any help will be appreciated!

One way is to create new feature by combining A and B by a function and eliminating A and B from dataset. The less features are correlated the better. Then you don't have redundant features.

But actually, it may be enough to use Random Forest. Then the model will have enough trees to figure out that the best split is first B and then A.


calculating the precision and recall for a specific threshhold I have a logistic regression model in which I calculate the tpr, fpr and thresholds using the roc_curve. After looking at the accuracy rates for different thresholds, I found the most optimal threshold to be 0.63. I have been told that I need to calculate the new precision and recall based on the most optimal threshold which in this case is 0.63. Is there any way to this calculation is python?

y_pred = log_reg.predict_proba(X)[:, 1] > 0.63
print(precision_score(y_true, y_pred), recall_score(y_true, y_pred))


How to build a data analysis pipeline procedure I have a series of scripts. Some are in R, some in Python, and others in SAS. I have built them in such a way that one code outputs a .csv file that the next code obtains and then that code outputs a .csv file, and so on...

I want to create a script that will automatically run each script in order so that the final output can be generated automatically.

What method would be best for this and can anyone point me to any examples of the procedure?

Nomially, I would simply just try to write a bash script (or powershell in windows) and just string the commands together.  However, this approach is rather fragile as in things get overwritten, and if it's an end to end process that has long batches.

I tend to use a workflow packages like luigi or airflow when stringing dependent task together.  The idea for Luigi is that you can break each action into a task.  Each task has three needed functions.

Requirement - what needs to exist before this task runs?
Output - where is the output going?
Run - what is the task?

So essentially you just chain a bunch of task and define your run function to call your previously built scripts using something like subprocess.  In requirement, you would reference the last step, and for output you would point towards where the file is being written.

The pro for doing this is that if your process breaks in task 50 out of 100, you don't have to rerun all 50 task, luigi will go down the dependency tree until it finds a requirement not fulfilled.

Calling R from Pytho
Luigi


What does this linear regression summary tells us? I just started learning predictive modelling in R , however i do understand some terms below but i lack in making more interpretations ,  just want to know what a pro statistician or Data Scientist interpret from it . How does one look at it ?

> summary(model)

Call:
lm(formula = comp$Minutes ~ comp$Units)

Residuals:
    Min      1Q  Median      3Q     Max
-9.2318 -3.3415 -0.7143  4.7769  7.8033

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)    4.162      3.355    1.24    0.239
comp$Units    15.509      0.505   30.71 8.92e-13 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 5.392 on 12 degrees of freedom
Multiple R-squared:  0.9874,    Adjusted R-squared:  0.9864
F-statistic: 943.2 on 1 and 12 DF,  p-value: 8.916e-13

The global test shows that the model is significant as the p value is very low which is p-value: 8.916e-13 in alpha level of 0.05. 98.64% of the variability of Minutes  is described by Units. The effect of Units is significant in predicting Minutes. For each one unit value increase in Units, Minutes increases by 15.509.


Online learning w/ feature weighting/adjusting Let's say I have a supervised learning problem with a sequence of features and labels. First, I learn on the training data and then I decide to stream in data, point by point and do online learning. Is it possible to update the weights or figure out the feature importances as each data point comes in? Also, what online learning algorithms would allow me to do this and can this be done in Python?

Yes, this can be done in Python. Scikit-Learn has a few online learning algorithms available, of which you can derive the feature importances. Look at the following webpage under 6.1.3. Incremental learning:

http://scikit-learn.org/stable/modules/scaling_strategies.html