How can I check if a bigger training data set would improve my accuracy of my scikit classifier? How can I check if a bigger training data set would improve my accuracy of my scikit classifier, is there a method or something?

One idea:

Split your data into train / hold out datasets.
Train the model on a fraction of the training data (say 50%) and test on the holdout dataset.
Train the model on a larger fraction of the training data (say 75%) and test on the holdout dataset.

It's important that you use the same holdout data for testing so you can perform a true test of accuracy.

Since you're doing classification, you should check that your data is balanced, and adjust if not (this may also improve your accuracy without needing larger training data).


How the vector of weights is assigned to a Neural Network Say for example, you have an artificial neural network like the one given below-

You have set of input nodes, hidden nodes and output nodes although I have one output node in my case.

We assign weights to the connections between those nodes. So is there any pattern of assigning weights to those connections i.e. See the first connection between first input(top) node and first hidden(top) node. So is it that the connection is assigned first value from the vector of weights,

the second the connection between first(top) input and second(second from top) hidden node is assigned second value from the vector of weights .

In short, what am I asking is that - Is there any correspondence between the position of the weight in the vector of weights and the connections in the structure. Is there any matrix that is assigned?

Yes, I am asking about Backpropagation algorithm. I want to understand how the weights get assigned to the neural network. If we want to assign a weight matrix to the neural network, how will we assign it?


How do the policy gradient's cost function and gradients work? I am not a math expert but have a basic understanding of linear algebra, calculus and probability and I understand the math behind back propagation. Currently I am trying to learn about policy gradient algorithm, but I am having difficulty understanding the underlying math. The most commonly used cost functions in the neural network training includes a target and output. For example:
MSE:
$$E_{total} = \sum{\frac{1}{2}(target - output)^2}$$
Log Loss:
$$Error = Output(i) * (1 - Output(i)) * (Target(i) - Output(i))$$
The idea is to find parameters $\theta$ which reduces the distance between the target and the output.
But in policy gradient method the cost function is like this:
$$g = \mathbb E\Big[\sum R_t*\frac{(\partial)} {(\partial\theta)}ln\pi_\theta(a_t|s_t)\Big]$$
What is the target and output in policy gradient's cost function?
How does this cost function get minimized and how does it works?

In policy gradients we are interested in maximizing the expected reward. For this we assume that the expected reward is parametrized by parameters $\theta$ (e.g. a Neural Network. This means that in order to maximize the expected reward we need to find these parameters. In math notation:
$$
\theta^{\star}=\arg \max _{\theta} J(\theta) =\arg \max _{\theta} E_{\tau \sim p_{\theta}(\tau)}\left[\sum_{t} r\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)\right]
$$

where $\tau$ is a trajectory sampled from policy $p_\theta$. To solve this we need gradient ascent so our parameters are updated: $\theta = \theta + \alpha\nabla J(\theta)$. Thus, if we compute the gradient of the expected reward we will get the proper update of our parameters towards greater expected rewards. You can take a look at Likelihood Ration and REINFORCE which explains analytically the whole optimization process.

To solve this you need the Policy Gradient theorem which will lead you to the equation you have. In other words, by using the gradient form you mentioned we are updating our parameters towards higher expected rewards. Also, PGs are more close to classification (cost-sensitive) rather than regression.

In terms of Neural Networks (and assuming stochastic policies), typically, your inputs are going to be states and your outputs action distribution and/or expected reward, thus mapping states to action probabilities (and/or reward predictions). The additional reward prediction it has been shown that leads to better results than using only the policy loss, as it drives the NN's representations to also predict expected rewards (instead of only action distributions).


What is the relationship between "square loss" and "Mean squared error"? Loss functions are important part of machine learning.

Square loss is one of the most popular loss functions.

Mean squared error (MSE) measures  the average squared difference between the estimated values and what
 is estimated

The estimated value is denoted as $\hat y$, what is estimated is denoted as $y$.

$${\displaystyle \operatorname {MSE} ={\frac {1}{m}}\sum _{i=1}^{m}(y_{i}-{\hat y_{i}})^{2}.}$$

in the context of distance/difference, it seems that MSE and square loss do the same job.

what is the relationship between "square loss" and "Mean squared error"?

$$V(f(x), y) = (1-yf(x))^2$$

In the context of classification where $y_i$ is $1$ or $-1$:

\begin{align}
MSE &= \frac1m \sum_{i=1}^m (y_i-f(x_i))^2 \\
&= \frac1m \sum_{i=1}^m y_i^2 (1-y_if(x_i))^2\\
&= \frac1m \sum_{i=1}^m (1-y_if(x_i))^2\\
&= \frac1m \sum_{i=1}^m V(f(x_i), y_i)
\end{align}


Dealing with input images of different shapes in PyTorch I've started to work with a leaf classification dataset on Kaggle. All input images have different rectangular shapes. I want to transform the input into squares of a fixed size (say, 224x224) with a symmetric zero-padding either on top and bottom or on the left and right sides of the rectangle. Prior to that, I think that I need to rescale the image (some images in the dataset have shapes >1000). I've encountered the next two problems:

torchvision.transforms.Resize(size) rescales the image so that its smaller side is matched to size (size is a scalar). I want the opposite, the bigger side to be matched to size.
torchvision.transforms.Pad(padding) seemingly works only with some fixed padding, but this transform will not always output a square.

How would you resolve this problem? I'm aware of RandomSizedCrop, but I feel like some datasets (like this one) aren't good for this method (some of the input images are too oblong). Also, I heard that RandomSizedCrop shouldn't be
used for the test data loaders.

More generally, how do you deal with input images of different rectangular shapes in PyTorch? Any help is appreciated.

Try this function that uses cv2 resize function:

def leaf_image(image_id,target_length=160):
    """
    `image_id` should be the index of the image in the images/ folder

    Reture the image of a given id(1~1584) with the target size (target_length x target_length)

    """

    image_name = str(image_id) + '.jpg'
    leaf_img = plt.imread('images/'+image_name)  # Reading in the image
    leaf_img_width = leaf_img.shape[1]
    leaf_img_height = leaf_img.shape[0]
    #target_length = 160
    img_target = np.zeros((target_length, target_length), np.uint8)
    if leaf_img_width >= leaf_img_height:
        scale_img_width = target_length
        scale_img_height = int( (float(scale_img_width)/leaf_img_width)*leaf_img_height )
        img_scaled = cv2.resize(leaf_img, (scale_img_width, scale_img_height), interpolation = cv2.INTER_AREA)
        copy_location = (target_length-scale_img_height)/2
        img_target[copy_location:copy_location+scale_img_height,:] = img_scaled
    else:
        # leaf_img_width < leaf_img_height:
        scale_img_height = target_length
        scale_img_width = int( (float(scale_img_height)/leaf_img_height)*leaf_img_width )
        img_scaled = cv2.resize(leaf_img, (scale_img_width, scale_img_height), interpolation = cv2.INTER_AREA)
        copy_location = (target_length-scale_img_width)/2
        img_target[:, copy_location:copy_location+scale_img_width] = img_scaled

    return img_target

# Test the leaf_image function
leaf_id = 343
leaf_img = leaf_image(leaf_id, target_length=160);

plt.imshow(leaf_img, cmap='gray'); plt.title('Leaf # '+str(leaf_id)); plt.axis('off'); plt.show()

Source: https://github.com/WenjinTao/Leaf-Classification--Kaggle/blob/master/Leaf_Classification_using_Machine_Learning.ipynb


ROC curve shows strange results for imbalanced dataset I have a classifier with a heavily imbalanced dataset (1000 of each negative label for each positive.)

I'm running a GradientBoostingClassifier with moderate success (AUC .75) but the curve has this strange look:

Any good ideas on what would cause the curve to have this behaviour?

Davis and Goadrich have explained the relationship between ROC and PR Curves in their paper. It is always recommended to use PR curve over the ROC curve in the presence of highly imbalanced data.

Back to the behavior of your ROC curve, It seems that you don't have more threshold points! I would also agree with Dan and do K-fold CV.

  Davis, J. and Goadrich, M., 2006, June. The relationship between
  Precision-Recall and ROC curves. In Proceedings of the 23rd
  international conference on Machine learning (pp. 233-240). ACM.


Fitting Lasso model repeatedly on same train dataset I am using lasso for feature selection. I have selected lasso parameters from grid search. Now, I used the the best model which I got to fit on training data set. Thus, out of 70 variables I got 66. I observed if I fit lasso again on the same training data set, the variables get dropped again (by drop I mean their coefficients are zero). My doubt is should I go ahead with first iteration where I got 66 variables or should I fit again and again when there is no further variable drop?

In principle, that shouldn't happen.  If you rerun Lasso on the same data set, with the same hyperparameters, after dropping attributes whose coefficient was set to zero in your first Lasso run, you should get the same result.  If you see something different, something has gone wrong.

You'll need to investigate deeper to see if you can figure out why that is happening.  Perhaps there is a bug in your code, or perhaps you are not providing the right settings to your implementation of Lasso, or perhaps the optimization algorithm is not finding the exact optimum solution, or perhaps something else has gone wrong.  It's hard to say with the level of detail you have provided.

If you're using different hyperparameters the second time you run Lasso, anything could happen.  Don't do that; that doesn't make sense.


Preserve colour in convolutional autoencoder at the moment i work with convolutional autoencoder and now I'am looking for paper or methods that adresses a colour preversation. Most of the AE paper use grayscale images and loss functions such as SSIM that preserve the structure very well are also focused on grayscale images. My networks are good in preserving structure (with SSIM as loss) but have a hard time representing the right colour.

I use a all convolution architecture without any pooling. my downsampling is derived by strides > 1 and the upsampling is obtained by transpose convolution.

Try training with more epochs. I've seen this same thing where the output after M epochs is a grayscale image but continued training results in more and more color.


Ratio of positive to negative sample in data set for best classification Suppose I have 100 positive samples. How many negative samples do I need to have in order to make the classifier work the best. In many papers, I have noticed that they take 4 times or 5 times the number of positive data sample to get the negative data sample. Will such a data set be useful?

It is recommended to have around equal number of instances in each class. If not, then you should normalize the data. 4 or 5 times more positive class size could produce highly biased classifier. However, if your negative class is not well defined then there are ways to train your classes with only one class as mentioned in the paper, Learning from positive examples when the negative class is undetermined- microRNA gene identification.


Is it effective to use one-hot encoding when its dimension is as large as thousands Here I try to construct a classifier using DNN(deep neural network) with its inputs being many portfolios. In essence, each portfolio contains several stocks which are labeled by there inner-code, for example "1430" or "5560", etc. Since the inner-code is discrete digital number, I prefer to use on-hot encoding to represent each portfolio. However, there are as many as a few thousand different inner-code which means that the dimension of on-hot code may also be that large. I wonder whether the dimension is too large for real training and is there any way to solve or ease this problem, such as using PCA afterward.

In neural networks applied to natural language processing normally each possible word (or sub-words) is handled as an individual token, and normally the vocabulary (the set of supported tokens) is around 32K. These tokens are typically one-hot encoded. Therefore, there is not inherent problem in having one-hot encoded vectors with thousands of components. However, the amount and variety of training data should support the dimensionality of the model for it not to overfit. Note that in NLP, these one hot encodings are used as expected outputs of the model, over which to compute maximum likelihood (categorical cross-entropy) together with the output of the model (a softmax).

Depending on how you are using this portfolio information, another option would be to have an embedding layer, which embeds discrete values into a continuous representation space (whose dimensionality is a hyperparameter of the model). This would be possible if this information is used as input to the model. In those cases, you input an integer number, which is used to index the embedded vector table.


Recommender/Clustering data to support a hypothesis. Is this a valid use-case for unsupervised ML? I have a dataset where some items have been labelled (categorized into 4 classes [A,B,C,D]). However, there is a vast majority of the dataset which has not been labelled. My hypothesis is that there are some characteristics which influence which category is applied to each item. Would clustering or maybe even a recommender system be able to suggest where each item should be placed? On a practical level, would I provide the "labels" within the model? Or would I keep it apart until the end and then overlay those labels on whatever the model managed to group together?
The above example seems like a clustering use case. However, can I spin the problem into a recommender system? As in, you labelled item X as A, and it has characteristics 1,2,3... item Y has similar characteristics, maybe you should label it A as well?

Clustering and recommendation are similar tasks, however in recommendation you usually want to recommend several items while clustering usually assigns each sample to only one cluster.
Anyway for your problem a clustering or even a classifier might help.
If labels are assigned on the basis of a similarity metric (and you have a good guess of what this metric might be) you could use a clustering algorithm to impute the missing labels (clustering is based on similarity between samples). However if it's reasonable that samples can be similar according to different metrics (and you are uncertain about it) then a classifier might help - provided you have enough labeled data to train the classifier. Which approach works best will also depend on the number of labeled samples and number of features.
Whichever strategy you choose to use the imputed labels will be noisy (some samples will be mislabeled). This is important to take into account especially if you are going to further process those labels.


How to group chat messages by topic? I am a newbie in this field. Developer since 20 years and more but never done anything (except tutorials) with ML, DL, and NLP. Though I've already read a bunch of articles and tutorials about this technology and I am starting to figure out the steps and the conditions required to make it work.

What I would like to achieve (the reason of my question) is this:

I have a file containing about 2 years of conversations between me and another person. I would like to extract sequences of messages that are related by the same topic. I mean messages that are sequential in time and that belong to the same conversation about a topic.

My goal is to extract the time we spent on every single topic.

There is any model already trained for this task? (This is of course the obvious question :-P )

Or, there is any model that I can use as a start training base?

Or, if not, what should be a good approach (steps, techniques, software) to train one by my self? (If possible).

Thanks

Update

Thanks to Erwan response I made some more research. I am not sure if Erwan's answer can be considered a resolutive answer but no doubts that it pointed me to a possible research direction "removing part of the fog in front of my eyes". Since, in my case I don't have a labeld dataset to train a supervised model I started to search for LDA solutions (as you implicitly suggested and also based on several tutorial I found) and found some: gensim's LDA model and parallelized LDA model; lda-project; I also found a Java implementation that is MALLET LDA class ecc...

I also started to browse arguments like Word2Vec, FastText (and family - I guess), even if I am not sure what are the purpose of these software compared to LDA, possibly because I have not a clear and complete view of the informations that LDA models can "capture" compared to the others.

Though the first step was to find a software that supports Italian language "natively" and I found spaCy a tool that provides lemmatization, tokenization, ecc... and seems to have the purpose to prepare text for other topic detection softwares (even if it seems it has its own class for this, guessing by name, TextCategorizer even if I am not able to figure out how to use it because of the lack of a complete documentation and examples).

So my guess is I could use spaCy to prepare the text to be feed to gensim's LDA (or any other implementation).

For data visualization I found pyLDAvis.

This problem is related to the following standard problems:

topic classification/modeling, which ranges from simple supervised document classification to unsupervised assignment of topics distributions to every document (the advanced option, with Latent Dirichlet Analysis and variants)
sequence labeling, a supervised task which predicts classes for every instance in a sequence, taking into account the order of the instances (e.g. it can leverage the fact that the class of instance $n$ is influenced by the class of instance $n-1$).
text segmentation, more precisely topic segmentation in this case.

I think that one can find a lot of good implementations for the first two problems, which are very common. However adapting these to your case and/or combining them will probably be more complex.

If you have a small set of topics and you have (or can have) a reasonable sample of  messages annotated with these topics, I would suggest starting with sequence labeling with Conditional Random Fields. I think this could give good results given the sequential nature of chat conversations.


Using a feature in prediction even if it gets zero as p-value? I created two binary classification based logistic regression models and I got these results:
Model 1:

Accuracy:  67.51%

AUC:       65.21%

Model 2:

Accuracy:  67,99%

AUC:       65,70%

The second model is created based on an additional column (Col_x).
But, when I try to use the following code to show the correlation, I got the result: (0.16402709962654458, 0.0)
from scipy import stats
stats.pearsonr(data['y'] , data['Col_x'] )

I would like to know if I can considerate Model2 an improvement of Model1 even if I have a p-value = 0.0

For the correlation problem, this basically sums up why I dislike univariate feature selection. Just because a feature has low correlation with your response in isolation, does not mean that the variable itself is not informative. The variable could relate in a non linear way (so transforming it could be useful), it could work well with other variables in a multivariate setting, and also could be used to generate other variables through feature engineering.

With respect to a "low p value", note that a lower p value suggests that the variable is significant, that is, the effect of the variable on the response is statistically different from 0. So if you were to use p values as a means of feature selection, you would select variables with smaller values. However, this is not at all a good way to select variables if your goal is to improve predictive performance. P values are meant to test very specific statistical hypotheses related to statistical inference, not prediction. Removing variables with high p values will likely lead to a loss in predictive performance because p values don't even answer the right question in this context. Also, realize that by conducting multiple hypothesis (looking at individual p values for coefficients) you further increase your chance of a false positive beyond what you set your significance level at (i.e. the cutoff that you use to deem a variable to be significant).

If you want to do feature selection within the context of a linear model, look into the LASSO/Elastic Net. These models will shrink the coefficients of variables to zero if they do not improve the fit of the model. Otherwise, if you only have a handful of variables then go with model 2 since it is the highest scoring model (on this specific test set anyway). Possibly repeat model evaluation on different partitions of the dataset to see if model 2 is consistently better than model 1. This would allow you form confidence intervals which could be useful in deternining if the extra column gives a significant performance increase over a simpler model.

Finally, I recommend not using AUC or accuracy for comparing the performance of models. These metrics are not proper scoring rules and may lead you to selecting models that don't discriminate between your classes as well as others.


Gradient decent in Python I just finished working on my first machine learning algorithm i.e Linear regression. I want to reduce the rmse by optimising the model. I found out that gradient decent does the same job. But i dont know how to do it in python. I refered to some videos on youtube but every video explains y=MX+c . But i have about 50 variables in my model.

Is there any library for it. Please help me out.

You want to implement gradient descent on a multivariate linear regression.

You can find an implementation in python here :
https://towardsdatascience.com/implementation-of-multi-variate-linear-regression-in-python-using-gradient-descent-optimization-b02f386425b9


Tool for designing CNN architectures I am currently building a (simple sequential) fully convolutional network for an object recognition task.

Designing the architecture essentially amounts to choosing kernel size, stride and unit number for each layer.
While doing this, I have to keep track of several properties of my layers:

parameter count
field of view of each kernel, wrt to the input/data
a margin in the last layer that I need to account for when creating the label images

I've written a small script that computes these quantities for different CNN configurations. It's not very complicated, but…

I'm under the impression that everyone using CNNs has to do this, so there should be a (graphical) tool for this. It could also be used to draw the fancy architecture diagrams. Because it works on the same input data.

Is there such a tool? How do you compute these quantities for your models?

Given the number of parameters and the number of options for each parameter, a graphical tool (aka, graphical user interface or GUI) is not very practical for CNNs. Most practitioners store and manipulate parameters in text form.

There are two graphical projects for TensorFlow. Playground which is an in-browser network that can be modified and run on sample datasets. TensorBoard which visualizes the computational graph and helps with debugging models.


What does it mean when we say most of the points in a hypercube are at the boundary? If I have a 50 dimensional hypercube. And I define it's boundary by $0<x_j<0.05$ or $0.95<x_j<1$ where $x_j$ is dimension of the hypercube. Then calculating the proportion of points on the boundary of the hypercube will be $0.995$. What does it mean? Does it mean that rest of the space is empty? If $99\%$ of the points are  at the boundary then the points inside the cube must not be uniformly distributed?

Speaking of '$99\%$ of the points in a hypercube' is a bit misleading since a hypercube contains infinitely many points. Let's talk about volume instead.

The volume of a hypercube is the product of its side lengths.
For the 50-dimensional unit hypercube we get $$\text{Total volume} = \underbrace{1 \times 1 \times \dots \times 1}_{50 \text{ times}} = 1^{50} = 1.$$

Now let us exclude the boundaries of the hypercube and look at the 'interior' (I put this in quotation marks because the mathematical term interior has a very different meaning). We only keep the points $x = (x_1, x_2, \dots, x_{50})$ that satisfy
$$
0.05 < x_1 < 0.95 \,\text{ and }\, 0.05 < x_2 < 0.95 \,\text{ and }\, \dots
\,\text{ and }\, 0.05 < x_{50} < 0.95.
$$
What is the volume of this 'interior'? Well, the 'interior' is again a hypercube, and the length of each side is $0.9$ ($=0.95 - 0.05$ ... it helps to imagine this in two and three dimensions).
So the volume is $$\text{Interior volume} = \underbrace{0.9 \times 0.9 \times \dots \times 0.9}_{50 \text{ times}} = 0.9^{50} \approx 0.005.$$
Conclude that the volume of the 'boundary' (defined as the unit hypercube without the 'interior') is $1 - 0.9^{50} \approx 0.995.$

This shows that $99.5\%$ of the volume of a 50-dimensional hypercube is concentrated on its 'boundary'.

Follow-up: ignatius raised an interesting question on how this is connected to probability. Here is an example.

Say you came up with a (machine learning) model that predicts housing prices based on 50 input parameters. All 50 input parameters are independent and uniformly distributed between $0$ and $1$.

Let us say that your model works very well if none of the input parameters is extreme: As long as every input parameter stays between $0.05$ and $0.95$, your model predicts the housing price almost perfectly.
But if one or more input parameters are extreme (smaller than $0.05$ or larger than $0.95$), the predictions of your model are absolutely terrible.

Any given input parameter is extreme with a probability of only $10\%$. So clearly this is a good model, right?
No! The probability that at least one of the $50$ parameters is extreme is $1 - 0.9^{50} \approx 0.995.$
So in $99.5\%$ of the cases, your model's prediction is terrible.

Rule of thumb: In high dimensions, extreme observations are the rule and not the exception.


What regression to use to calculate the result of election in a multiparty system? I want to make a prediction for the result of the parliamentary elections. My output will be the % each party receives. There is more than 2 parties so logistic regression is not a viable option. I could make a separate regression for each party but in that case the results would be in some manner independent from each other. It would not ensure that the sum of the results would be 100%.

What regression (or other method) should I use? Is it possible to use this method in R or Python via a specific library?

Robert is right, multinomial logistic regression is the best tool to use. Although you would need to have a integer value representing the party as the dependent variable, for example:

1= Conservative majority, 2= Labour majority, 3= Liberal majority....(and so on)

You can perform this in R using the nnet package. Here is a good place to quickly run through how to use it.


DBMS or Software for privacy sensitive data We have a dataset of very privacy sensitive people data and want to build a database with it. The data protection department in our company doesn't like the idea that the data scientists are able to see any data specific to a person (even if anonymized). We can't preaggregate the data in the database because there are hundreds of different possible aggregations that could be interesting.

Is there a software or DBMS that could ensure that users can only query aggregated results that contain at least groups of N people?

How else would you solve this problem technically?

Two possible options:

Have the database administer set limits on SQL queries that only allow result tables with aggregated results with minimum of N people to be returned.

Apply differential privacy which a query result cannot be used to infer much about any single individual, and therefore provides privacy.


How to deal with new features values in my classification model? Lets say i have a categorical feature having a set of values equal to ['Single','Married','Divorced','Unknown']. Okay, so with the help of the other features, i create my model, i test it, all is fine and i deploy it.

Now , some user using my model as a webservice, types in 'Widow' for that feature value.

What's the proper way of dealing with it? Is there a way to ignore it? and predict based on the rest of the features only? or do i have to handle unknown feature values by assigning Unknown to them too? or worst case scenario, i handle the exception, return a message saying : hey man (or woman) use one of the available values on that feature! ??

Note : I'm trying to avoid assigning Unknown to it, because 'Unknown' has its weight. ( My real question is : if there is a way to set a feature's weight to 0 when doing inference )

Depending on the implementation, that variable could have been one-hot-encoded and in that way, the "widow" value could be assigned to "zero" in all the encoded variables (maybe equal to unknown) and the path designed to that 0 will be followed.


Making fake result in data mining using weka j48 algorithm I have 130 records in one CSV dataset and I'm using j48 decision tree. I used the whole training set for testing, and the result was 79 correctly classified and 51 incorrectly classified records.
Now! I want a result between 95% - 100% and don't have ability to add or remove any record to those 130. But I'm allowed to play a little with my dataset, for example use 10 out of 130 in such a way that all of those 10 must be classified correctly.

The Quality of tree is not Important
The Number of test samples is not so important (10-15 is good)
The Only Important thing is Accuracy between 95-100 %

FIRST, I tried to move 10 correct samples to bottom of dataset and use "split by percentage" (92.30%) but it wasn't useful.

At SECOND, I tried to choose them by "try and error" and chose randomly, and I was just testing the last sample (130th), the I found some correct samples and keep it there, then tried another below the last successful sample and tested the last two lines, the result should have been 50% or 100%, but surprisingly was 0% (Total instants:2 correct:0 Incorrect:2).

Can anyone help me please?
I already thank a lot dear readers...

I don't understand the purpose when the quality of the tree is not important but you should be able to make a tree to have 100% accuracy on training data set easily. Just avoid any pruning and let the tree grow as maximum as possible. As far as I remember Weka set a pruning policy for J48 by default, you should disable that. Also check to see that the split nodes with minimum number of instances possible.

Now that I an considering better, there are cases when full accuracy is impossible. Think for example a simple toy data set with 10 instances. Suppose that all instances are the same for input variables, but for target variable 5 are positive and 5 are negative. Either way you take a decision for prediction, the accuracy on training data would be 0.5. In this case you simply do not have enough data to discriminate on.


Does torch.cat work with backpropagation? I was wondering if it was okay to use torch.cat within my forward function. I am doing so because I want the first two columns of my input to skip the middle hidden layers and go directly to the final layer.

Here is my code: you can see that I use torch.cat at the last moment to make xcat.

Does the gradient propagate back? or does the torch.cat cover up what happened to my hidden variables?

class LinearRegressionForce(nn.Module):

    def __init__(self, focus_input_size, rest_input_size, hidden_size_1, hidden_size_2, output_size):
        super(LinearRegressionForce, self).__init__()

        self.in1 = nn.Linear(rest_input_size, hidden_size_1)
        self.middle1 = nn.Linear(hidden_size_1,hidden_size_2)
        self.out4 = nn.Linear(focus_input_size + hidden_size_2,output_size)

    def forward(self, inputs):
        focus_inputs = inputs[:,0:focus_input_size]
        rest_inputs = inputs[:,focus_input_size:(rest_input_size+focus_input_size)]
        x = self.in1(rest_inputs).clamp(min=0)
        x = self.middle1(x).clamp(min=0)
        xcat = torch.cat((focus_inputs,x),1)
        out = self.out4(xcat).clamp(min=0)
        return out

I call it like so:

rest_inputs = Variable(torch.from_numpy(rest_x_train))
focus_x_train_ones = np.concatenate((focus_x_train, np.ones((n,1))), axis=1)
focus_inputs =  Variable(torch.from_numpy(focus_x_train_ones)).float()
inputs = torch.cat((focus_inputs,rest_inputs),1)

predicted = model(inputs).data.numpy()

Yes, torch.cat works with backward operation.


Sklearn SVM - how to get a list of the wrong predictions? I am not an expert user. I know that I can obtain the confusion matrix, but I would like to obtain a list of the rows that have been classified in a wrong way in order to study them after classification.
On stackoverflow I found this Can I get a list of wrong predictions in SVM score function in scikit-learn but I am not sure to have understood everything.
This is an example code.
# importing necessary libraries
from sklearn import datasets
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split

# loading the iris dataset
iris = datasets.load_iris()

# X -> features, y -> label
X = iris.data
y = iris.target

# dividing X, y into train and test data
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 0)

# training a linear SVM classifier
from sklearn.svm import SVC
svm_model_linear = SVC(kernel = 'linear', C = 1).fit(X_train, y_train)
svm_predictions = svm_model_linear.predict(X_test)

# model accuracy for X_test
accuracy = svm_model_linear.score(X_test, y_test)

# creating a confusion matrix
cm = confusion_matrix(y_test, svm_predictions)

To iterate through the rows and to find the wrong ones, the proposed solution is:
predictions = clf.predict(inputs)
for input, prediction, label in zip(inputs, predictions, labels):
  if prediction != label:
    print(input, 'has been classified as ', prediction, 'and should be ', label)

I didn't understand what is "input"/"inputs". If I adapt this code to my code, like this:
for input, prediction, label in zip (X_test, svm_predictions, y_test):
  if prediction != label:
    print(input, 'has been classified as ', prediction, 'and should be ', label)

I obtain:
[6.  2.7 5.1 1.6] has been classified as  2 and should be  1

Is the row 6 the wrong row? What are the numbers after the 6.? I am asking this because I am using the same code on a dataset that is bigger than this one, so I would like to be sure that I am doing the right things.
I am not posting the other dataset because unfortunately I can't, but the problem there is that I obtained something like this:
  (0, 253)  0.5339655767137572
  (0, 601)  0.27665553856928027
  (0, 1107) 0.7989633757962163 has been classified as  7 and should be  3
  (0, 885)  0.3034934766501018
  (0, 1295) 0.6432561790864061
  (0, 1871) 0.7029318585026516 has been classified as  7 and should be  6
  (0, 1020) 1.0 has been classified as  3 and should be  8

When I count every line of this last output, I obtain the double of the lines of the test set... So I am not sure that I am analysing exactly the wrong list of predicted results…
I hope to have been enough clear.

The following method works for all kinds of classification problem.

Use list comprehension to find all indices of wrong prediction:

indices = [i for i in range(len(y_test)) if y_test[i] != y_pred[i]]

wrong predictions will then be:

wrong_predictions = test_dataframe.iloc[indices,:]

You can also make indices a new column of wrong_predictions, it would be convenient to compare :)


TFLearn Does Not Load Model Im trying to learn and get the hang of tflearn.  One thing that I have tried to do is save my model, and then reload it.  I get an error that says:

W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for quicktest.model

Below I have my model as well as that checkpoint file that gets created with the saving of a model.  I have googled this error and tried many different file changes but nothing has fixed it.

    import tflearn
    data, labels = load_csv('my_csv', target_column=0, categorical_labels=True, n_classes=2)
    net = tflearn.input_data(shape=[None, 32])
    net = tflearn.fully_connected(net, 32)
    net = tflearn.fully_connected(net, 2, activation='softmax')
    net = tflearn.regression(net)
    model = tflearn.DNN(net)
    model.fit(data, labels, n_epoch=1, batch_size=10, show_metric=True)
    model.save('quicktest.model')
    model.load('quicktest.model')

checkpoint

model_checkpoint_path: "quicktest.model"
all_model_checkpoint_paths: "quicktest.model"

Found out what the issue was:

model.load('filename')

Should actually be:

model.load('./filename')

Hopefully this helps out someone else.


Keras, DNN ending with sigmoid - model.predict produces values < 0.5. This indicates...? I'm trying a simple Keras project with Dense layers for binary classification. About 300000 rows of data, labels are

training_set['TARGET'].value_counts()
0    282686
1     24825

My model looks like this

def build_model():
    model = models.Sequential()
    model.add(layers.Dense(64, activation='relu', kernel_regularizer=regularizers.l2(0.001),
                           input_shape=(train_data.shape[1],)))
    model.add(layers.Dropout(0.5))
    model.add(layers.Dense(32, kernel_regularizer=regularizers.l2(0.001), activation='relu'))
    model.add(layers.Dropout(0.5))
    model.add(layers.Dense(1, activation='sigmoid'))
    model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy'])

    return model

So it's binary classification that ends with a sigmoid. It's my understanding that I should get values close to 0 or close to 1? I've tried different model architectures, hyperparameters, epochs, batch sizes,  etc. but when I run model.predict on my validation set my values never get above 0.5. Here are some samples.

20 epochs, 16384 batch size
max 0.458850622177124,  min 0.1022530049085617
max 0.47131556272506714,  min 0.057787925004959106

20 epochs, 8192 batch size
max 0.42957592010498047,  min 0.060324762016534805
max 0.3811708390712738,  min 0.022215187549591064

20 epochs, 4096 batch size
max 0.3163970410823822,  min 0.0657803937792778

20 epochs, 2048 batch size
max 0.21799422800540924,  min 0.03832605481147766

Is this an indication that I'm doing something wrong?

Training and validation loss

I think the dropout is a bit high and if it's a binary classification, then why in the end a single node?

Make sure your target variable is having proper shape in case of softmax...(one hot/ to_categorical())

def build_model():
    model = models.Sequential()
    model.add(layers.Dense(64, activation='relu', kernel_regularizer=regularizers.l2(0.001),
                           input_shape=(train_data.shape[1],)))
    model.add(layers.Dropout(0.3))
    model.add(layers.Dense(32, kernel_regularizer=regularizers.l2(0.001), activation='relu'))
    model.add(layers.Dropout(0.3))
    model.add(layers.Dense(**num_classes**, activation='softmax'))
    model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy'])

    return model

To improve it further, you need to use some techniques, such as cross-validation, batch normalization, and increase the epochs(maybe).


Question answering (QA) vs Chatbots Are Question answering (QA) the same as Chatbots? I can not understand the difference between them.

For me it's the same thing: interact with a robot that answers questions.

Question-answering (QA) is sometimes used to refer to the task where the input to the system is a question and a list of possible answers (normally only a handful) or a paragraph where the answer is supposed to be found, and the expected answer is the index of the correct answer or the start/end positions where the answer located within the text.

In chatbots, the task is normally defined more generically as "answer the given question". This can be approached in many ways, like with chit chat chatbots, where you simply train your system with pairs of question and answer much like a machine translation system, or with information retrieval chatbots, which make use of documents to identify the relevant pieces of composes the answer based on them.

Therefore, in the usual QA definition, QA is much more constrained (i.e. easier) than chatbots.


Difference of sklearns accuracy_score() to the commonly accepted Accuracy metric I am trying to evaluate the accuracy of a multiclass classification setting and I'm wondering why the sklearn implementation of the accuracy score deviates from the commenly agreed on accuracy score: $\frac{TP+TN}{TP+TN+FP+FN}$

For sklearn the sklearn.metrics.accuracy_score is defined as follows(https://scikit-learn.org/stable/modules/model_evaluation.html#accuracy-score):

$\texttt{accuracy}(y, \hat{y}) = \frac{1}{n_\text{samples}} \sum_{i=0}^{n_\text{samples}-1} 1(\hat{y}_i = y_i)$

This seems like its completly neglecting the true negatives of the classification.

Example:

Predicted        1                     2                 3
Actual
1               5                      2                 0
2               8                      6                 2
3               3                      4                 12

And here the TP,TN,FP and FN:

                TP                     TN                   FP                FN
1               5                      24                   11                2
2               6                      20                   6                 10
3               12                     21                   2                 7
SUM             23                     65                   19                19

In the "standard" average score I would calculate:
$\frac{23+65}{23+65+19+19}=0,698$

In the sklearn implementation however it would be:
$\frac{1}{42}*23= 0,548$

Why is this different? And is the other metric somewhere mentioned in the literature, I couldn't find anything so far.

Your "commonly agreed on" and "standard" accuracy is meant for binary classification, in which case it agrees with the other formula from sklearn.  In that case, "positive/negative" refer to the two classes, so this is also a little different from your version.

Your version of it is a sort of average of (the "mediant" of) the implicit one-vs-rest classifiers.  As such, your score is meaningful, but will generally be larger than the actual common multiclass accuracy metric.  For a balanced problem, a constant classifier will get a mediant-of-OVR-accuracy score of $(n-1)^2/n^2$ but an accuracy score of just $1/n$.  (Back to the binary case, to compare your method, you'd have to interpret e.g. "Sum(TN)" as including both diagonal entries, so the "accuracy" there is actually $1/2n$, which agrees with the mediant-of-OVR score.)

As such, your metric is similar to macro-averaged scores (though I've never heard of that for accuracy, only precision/recall/Fbeta).  Micro Average vs Macro average Performance in a Multiclass classification setting

Finally, as an opinion, since accuracy measures the probability of getting the prediction right, it's easier to interpret; your metric gives credit for not misclassifying a sample into each class it's not put in, hence the inflation.  Of course, this also perhaps makes a multiclass model's accuracy sound terrible (sklearn says "this is the subset accuracy which is a harsh metric since you require for each sample that each label set be correctly predicted").


Definition of a model in machine learning This definition does not quite apply since we are not always assuming an underlying distribution. So what is a model really? Can a Gradient Boosted Model (GBM) with specified hyperparameters be considered a model? Is a model a collection of rules?

I was interested in the same question recently and came to the realization that there is no single definition of a "model" in machine learning. It's highly dependent on the sources you're consulting, which may be the documentation for a particular software program, the slang adopted by its user community, or the definitions used in published academic papers, which can vary widely from journal to journal. Moreover, I had to learn to keep in mind that such papers are written not just by specialists in machine learning, but by experts in other disciplines who have a need for applying machine learning techniques (such as imaging, various medical fields, etc.). Many of them do not explicitly define the term "model," which is often used loosely. Here are just a couple of different definition of "model" I've seen in all this widely scattered literature:

• Statistical models, particularly the stats related to probability distributions.

• Regression data and related statistics.

• Mathematical models as mentioned by Neil Slater above.

• The data models used in machine learning, such as the columns involved, their data types, the data sources and other metadata. This is particularly tricky because there's nothing mathematical about this definition at all, unlike the first three I listed. For an example, see all the documentation for SQL Server "mining models," which serve double-duty for machine learning purposes.

• Sometimes all of the definitions above are expanded to include machine learning structures built on top of the equations and the metadata, such as the specifications of neural nets. In other cases, these are considered separate entities.

All of the above are sometimes mixed and matched together, depending on the source. I'm sure there are other definitions of "model" I've left off this list, which will complicate the matter even further. To deal with this ambiguity, I'm trying to train myself to divine the intentions of the author whenever they use the term "model." Sometimes it's easy to determine based on the context or the field the author works in, but other times I have to read deeply into an article or documentation before figuring it out. I wish I could be more definitive about it, but it's really a naturally fuzzy term; there's never going to be a simple one-size-fits-all answer to this. I hope that helps.


Classification of articles fetched from the web as describing stock (trending) increase or decrease I'm building an AI that works with natural language processing integrated in my school project application (an app dedicated to traders). I'm wondering what set of word is used most to express negativity (or positivity) in stock market? If a certain stock is decreasing in value (or increasing) what words are most likely to be appear on the press?

Ultimate Goal: Classifying an article fetched from the web as describing stock increase or decrease.

Are there any public free data sets tackling this subject?

Else, is there a data set that classifies sentences as trending or not trending?

Any suggestion, help or even collaboration would be very appreciated!

You can use a deep learning network to generate sentiment analysis. Pretrained models can be found which have been trained using vectorized word lists which are commonly associated with positivity and negativity. However, these may not generalize well to your specific use case.

Due to the historic nature of stocks you can easily compile your own database from online sources. For example, you can construct instances in your dataset as follows. Tokenize all the words found in articles pertaining to Google for February 5th 2001, then the target label can be the stock value for February 7th 2001. You can choose the time lag as you please. If you do this for a large amount of instances, your dataset should be able to contain some notion of what words found in articles will cause a stock price to rise or lower.


Can I cluster an aggregated data-set (grouped by) and apply dimensionality reduction? I have data of sales, however it is in the millions, about 500M rows. I aggregate this data by factors such as location, shoptype, country_of_shop, cardtype, and then the aggregated statistic is:

number of transactions
sum of amount in dollars

My questions is can i cluster this say using kprototypes or something similar? And can i reduce the dimensionality using Factor analysis?

There are no clear restrictions on can you cluster the data or not based on the data itself (in most of the cases), but you might face restrictions based on computational speed. The question is what you want to achieve in the end, but you can for sure cluster the data or reduce dimensions. You can also take a subset of your data, cluster it and then train ML model, something like catboost or lightgbm to predict clusters, instead of doing clustering on whole dataset. In my experience it will be faster to use model for cluster predictions.
For dimension reduction, you can go for PCA or TruncatedSVD, here are some useful links:
https://scikit-learn.org/stable/modules/unsupervised_reduction.html
https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html
But remember, that this techniques will reduce the number of features, not the rows, so you can still face computational issues.
If you want a way to map your high-dimensional data (around ~50 dimensions or less usually), you can use t-SNE
https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html


What Kernel is suitable for the following data for SVM classification? I have the following 2 class data, as shown below. .

Its a hand crafted example using two ellipse equations. I want to know what might be a recommended kernel to be used with this problem if I want to use an SVM based binary classifier.

I think the Rbf (Radial basis function) kernel (Wikipedia for rbf) is the most suited for this problem, as its contour plots are circles.

For instance,  this blog solves very similar problems using the same kernel.


What's the point of the test set? I get the point of a validation and training set, but the importance of a test set doesn't click for me.

Let's say you train a model and you try your best to avoid overfitting by testing your model on the validation set.

After you've decided you have a model you're proud of, you do a final sanity check on the test set, and let's say the performance is trash. Are you really going to start all over? What decision making does it inform? In my workplace, the way timelines are structured, there's no time to start over.

The point of a test set is to give you a final, unbiased performance measure of your entire model building process. This includes all modelling decisions in your pipeline, so any preprocessing, algorithm selection, feature engineering, feature selection, hyper parameter tuning and how you trained your model in general (5 fold? Bootstrapping? etc.). All of these decisions can lead to overfitting; for instance, selecting a set of hyperparameters that are coincidentally optimal for a particular validation set but not for the general population. If we have no test set you would not be able to identify this and would potentially be reporting highly optimistic scores.

Also, because the above modelling pipeline can get very complex, the possibility of leaking data and overfitting becomes very high. If you tune to your validation set, how will you know if your entire modelling process is not leaking data (and therefore overfitting?)

You bring up a good point; of course if we see that the test set score is poor then we will probably go back and tweak again. Thus, this just demotes the test set into a validation one if you use it too many times as you now run into the possibility of overfitting the test set (see almost every Kaggle competition). However, through repeated test set evaluation (train the model, then test it, then repeat with a different partioning) you will at least get a gauge on how variable your model is to help mitigate this problem. The amount of times you repeat will depend on how much the test set scores vary and how much uncertainty you are willing to accept (also time constraints).

In my opinion, in the business setting you should always make time to properly test your model. The dangers of overfitting are way too high and even worse; you would not even know it. If the test set scores end up being "trash" then at least you know the model is trash and you don't use it and/or you change your approach. This is way better than thinking the model is fantastic based off non rigorous validation and then having the model fail in production. The scientific method is there for a reason right?


Fuzzy Clustering for Categorical Data I have a dataset in which each feature is either 0 or 1 (like BBOW). I want to cluster the data such that one point can belong to more than one cluster(soft assignment).
I searched about this and I found that fuzzy k-modes can be applied for this problem. Since I am new to ML coding, Is there any implementation available online for fuzzy k-modes or any other similar algorithm?

Fuzzy C-means is implemented in Python and you just need to google it e.g. this one,  however you can implement it yourself as well.

My answer will be more about your task. You have categorical data which means any data point in your problem is on the corner of a high-dimensional simplex. A simple example is 3 points on three vertices of a triangle. How would you cluster them and even more, how would you fuzzy cluster them? (because they are always ON EACH OTHER which makes fuzzy clustering a bit meaningless)

I suggest this:
Find a similarity score between your points. For instance, as a very initial and simple approach, capture the normalized number of attributes in which two points have the same value.

a = [1,0,0]
b = [1,0,1]
similarity = 2/3

You can think of more similarity measures (what about Chi-Squared test?). When you have these scores between sample pairs, you can construct the affinity matrix and apply spectral clustering on it.

But where is the "Fuzzy" term? (or soft-clustering characteristic)

Spectral clustering works as follows:

You get the affinity matrix
Capture the eigen vector corresponding to the second smallest eigen value
That eigen vector is a representation of your data. Cluster that instead of data itself! Here you can use soft-clustering methods for clustering eigenvector elements.
See these steps in Section.2 of this.


Validation Curve Interpretations for Decision Tree I'm working on a machine learning class, and we're using supervised learning right now, starting with decision trees. I'm using the UCI Credit Card dataset (whether or not certain people will default in their payments due to past history).

Using a decision tree classifier for this attempt. Running a validation curve using scikit-learn, I'm getting a plot I'm not quite sure how to interpret. As you can see from the axes, the parameter is min_samples_leaf, and I'm varying it from 1 to 30 (by 2).

Based on this plot and some Googling, I believe the correct way to interpret this is that this dataset has high bias with no variance and nothing is really being learned. Or, in other words, decision trees are not a good algorithm for this dataset, since there doesn't really seem to be a trade-off.

For max depth, I'm getting a validation curve that looks like this:

Based on what I see here, there is quite a bit of bias at the smaller set, and more variance as the depth increases. Given that GridSearchCV returns an ideal max_depth of 5 and min_samples_leaf of 19. (edit: corrected numbers). Those numbers seem to indicate a very high bias, and there really is nothing to be learned here using decision trees.

Overall, based on the min_samples_leaf, I would hesitate to recommend a decision tree for this data set. However, the learning curve and the max_depth validation curve both seem to show there might be some value.

Puzzling to me is that the accuracy score (using metrics.accuracy_score and the ideal parameters from GridSearchCV) is 82%, which doesn't seem that bad. How do I reconcile these crappy validation curves with the accuracy score?

A few comments on the top of my head:

both parameters min_samples_leaf and max_depth are not very important for decision trees, so it's not surprising not to see much variation (or not all) across different values:

the fact that min_sample_leaf doesn't influence the performance simply means that the algorithm finds enough good predictors in the features to create leaves with a high number of instances, apparently always more than 30.
I'm less sure about max_depth, but I assume that it's a parameter which is used to prune the tree in order to avoid overfitting. This is exactly what happens here as soon as it's increased above 5: the algorithm creates a deeper (more complex) tree by using very specific conditions on the features which turn out to be specific to the training set, hence the divergence between training score and CV score (that's a clear sign of overfitting).

Value 5 for max_depth is indeed optimal, as seen on the second graph (due to overfitting with higher values). The value 19 returned by grid search for min_samples_leaf is as optimal as any other value between 1 and 30, as seen on the first graph: grid search just happened to pick this one but it doesn't have any impact anyway.
The 82% accuracy is completely normal: that's the Y axis on both graphs ;) It indeed looks like a decent performance, but we can't say for sure since there's no comparison: maybe the dataset is super easy and 82% is just the majority class, or maybe it's super hard and 82% is a great achievement.

Actually these graphs don't show much about the learning, they just show that the parameters studied are not really relevant. To observe something more interesting try:

Ablation study: pick a random subset of the training set instances and train only with this subset. Do this for say 10%, 20%, ..., 100% of the data, then plot the performance as a function of the size of the training set. This will show how many instances are needed for the model to reach its max performance (educated guess: not that many).
Feature selection: use only the N most informative features (as measured by information gain for instance) and plot the performance for different values of N. If you're lucky you might see an increase until a particular optimal value of N followed by a small decrease when uninformative features are added. If it happens, that last part would be due to overfitting.


How to feed my JSON dataset in Keras for character level text classification I have a JSON dataset, for example:

{"candidate00001": "graceful ones."
 "candidate00002":"One more,Marvelous said, sounding royally bored from his seat."
 "candidate00003":"She tired,Joe said, though not unkindly."
}

in which, candidate00001 is my first class, candidate00002 is the second class and so on.

I am new to python, so i want to implement a character level classification by Keras using this tutorial.

my problem is that I don't now how to convert my JSON file as x_train and y_train.

As mentioned by @Frankstr you want a tutorial on "character-level classification", not handwritten digit recognition as the one you have linked.

Character-level classification is typically done with an RNN or a 1D CNN. See this implementation of Character-level Convolutional Networks for Text Classification for example.

In order to convert your json data to x_train and y_train which can be fed to Keras, you can read it into a dictionary, extract keys and values separately, and binarize your class labels:

import json
from sklearn.preprocessing import LabelBinarizer

with open('data.json', 'r') as f:
    train = json.load(f)

y_train = list(train.values())

lb = LabelBinarizer()
x_train = lb.fit_transform(list(train.keys()))

Make sure your json is properly formed. The example you gave is missing commas after each value. It should be:

{"candidate00001": "graceful ones.",
 "candidate00002":"One more,Marvelous said, sounding royally bored from his seat.",
 "candidate00003":"She tired,Joe said, though not unkindly."
}


"Super" Optimizer concept I was wondering why there isn't a feature built into common-use ML libraries, like Keras, that plugs many different combinations of layers and nodes to multiple models and trains them simultaneously to single out the best NN architecture for your problem?

For example, given training data, validation data, and a loss function, it compares a model consisting of two hidden Dense layers with 256 neurons each, and another model consisting of two hidden Dense layers, the first with 256 and the second with 64. It would then save the model with higher accuracy according to the input loss function.

Does something like this exist already? I know something like this exists with SKLearn's GridSeachCV, but I wasn't sure if that's a common practice outside of SKLearn. I feel like I might be oversimplifying a complex problem.

Thank you!

You could have a look at the AutoML work, which offers a few different ways to optimise a model of parameter space and goes well beyond Sklearn. They even have a tool that wraps around Sklearn!

Here is a summary from their homepage:

Here is a longer description, and here is a full example based on Keras.

These libraries can essentially combine the best known practical methods of optimisation, such as:

standard grid-search over parameters
Bayesian optimisation of parameters
a combination of the two methods: HpBandSter (HyperBand on STERoids)


Is backpropogation used in convolutional neural networks? Do convolutional neural network use the backpropogation algorithm? I am not understanding what exactly happens in fully connected layers?

Both Full Connected Neural Networks and Convolutional Neural Networks use backpropagation for training.

What you said is right, both are feed forward neural networks, which means that the connections in the neural network start from left (input) and move towards right (output). But, it is feed forward only during inference time. During training i.e. when you are trying to find the weights/bias of the neural network for the task at hand, you have to use backpropagation.

Training has both forward pass and backward pass.
Inference has only forward pass.

Also, when we say feed forward it means that there are no loops in the network. L1->L2->L3->L4 and so on. But, if you see recurrent nets, they have loops like L->L->L->L


Why are there currently no content-based evaluation metrics for information retrieval? Consider the problem of learning to rank for Google-like searching - i.e., learning to return a good ordering of URL's when given a query. Most (if not all) current evaluation metrics for this problem are URL-based.  Current evaluation metrics like Mean Reciprocal Rank (MRR) and Discounted Cumulative Gain (DCG) sum up weighted relevance scores for the pages in a ranked list.  If a ranked list contains n pages, for instance, these metrics give a sum of n terms.

From my own literature review, it seems no metrics incorporate words.  Although some theoretically incorporate "information nuggets", none incorporate "information nuggets=words" in practice.  I have not found recent or past literature that concretely utilizes words, phrases, or anything similar in scoring ranked lists. Since I haven't found past literature, it does not seem that word-based metrics were attempted once and then replaced with coarser URL-based metrics.  Documents contain words, as do the snippets returned in a Google-like ranked list, so it seems natural that word-based metrics should have been attempted.  Thoughts?

(Note: One obvious objection is that TREC-based assessment of "word relevance" is a vague and even expensive endeavor (asking assessors to mark for words for relevance).  But there's other ways to get relevance feedback that ignore this problem, such as using the dwell time on pages as a proxy.)

Using a word-based metric would explicitly favor word-level retrieval methods.  The theory is that (just as you suggest with dwell time), the URL-level metric measures more directly the desired result.

More concretely, consider a search of "alcohol from potatoes." Assume we have two pages:

1) A page that is simply a grocery list (containing "alcohol" and "potatoes")
2) A page that describes vodka or moonshine which only uses the words "ethanol"
   and "potatoes" but not "alcohol."

A word-level metric would prefer the first result, since it contains both alcohol and potatoes..  A URL-level metric would likely label the second result as more relevant, which would allow a model to form associations beyond simple word-matching (i.e. it could learn that "alcohol" and "vodka" are associated).


Developing an Empirical Model for Non-Linear Data I have collected Temp vs. Time data for a controlled environment. I performed three tests, with the thermocouple in the same location for all three tests. I now have three Temp. vs. Time curves for this location. The graphs are shown here:

I am looking to generate an empirical model based on the Temp. vs. Time at this location that can predict the temperature at this location over time. How should I approach this? My initial approach was to identify the local minima and maxima, and average the three points together (for each maximum and minimum). And then correlate that average to a sequence of time, for example, 30C between 750 and 1250s, or something like that. I really don't think this is an efficient approach...

What do you all think would be an efficient approach to this?

A couple of thoughts...

First as a (former) physicist:

As a scientist/physicist I would likely never be satisfied to analyze the data that you have presented because there are some obvious issues with it.  I was initially thinking that it was way to schizophrenic, but then I noticed that you referred to the time units as seconds, in which case this is a pretty slowly varying time series.  Are the units really seconds?  If they happen to be milliseconds or microseconds then you might want to think about a probe that is less noisy (i.e. a larger probe with more of a temperature sink) and check weather you are seeing the effects of ground loop feedback in your system or just EMF interference from your electrical system.

Secondly, there are clear patterns in the data and these do not line up very well, so I would rerun the experiment with things timed much more accurately.  For instance, the event that occurs midway through the experiment that drops the temperature (looks like an icicle or dagger) occurs 5-8 minutes apart in comparing the dark blue experiment to the purple experiment.  It would be great if these could be synced better.  The point being that if you can't sync them better, then the noise of the time series renders most phenomenon apart from a flat line, almost useless.

Now as a data scientist:

This data is really noisy or at least fluctuates significantly!  You should stay away from averaging minima and maxima as you suggest in your post.  extrema are very sensitive to noise where as you are trying to reduce the noise in your system.

You should perhaps think about applying some sort of smoothing method to reduce the jitter in your data.  Perhaps a second-order exponentially weighted moving average like Holt-Winters.

Finally, you could probably just average the three signals to produce a mean signal.  Means are more suseptable to outliers than medians,so the median signal is also an option.

Some other options include separating the trend from the general behavior using some type of de-seasoning or double ARIMA procedure.  But, it seems like these are too extreme given that the dagger/icicle event looks significant and you don't want to transform it away.  So I would only use these if there is some type of periodicity to your data.

Hope this helps!


Text Mining of Research Paper Abstracts I'm trying to build a system which tackles the following problem. Given a list of research papers L, I would like to extract the following information from each paper:

The problem addressed by the paper
Solution to the problem proposed by the paper
(Optional) Results, i.e. how good is the solution

I have dealt with the easy part, i.e. setting up the blocks that interact with web search engines (e.g. Google scholar) and gather the data, given a set of search keys.

However, I don't have the knowledge to set up the 'intelligent' part, i.e. the one which extracts the aforementioned information from the text data. I would prefer to extract information based on the Abstract, but analysis the complete text wouldn't be a dealbreaker for me. I'm mostly interested in Computer Networking papers, e.g. published on SIGCOMM, CoNEXT, etc.

My questions are:

Are there tools - preferably open source - which can extract this information from research papers?
If not, what techniques do you recommend me to start looking at (e.g. research papers, tutorials, etc.) to accomplish this?

In order to train some supervised learning algorithm to identify 'Problem' and 'Solution', you need to somehow generate some data that has labels of these things, which may be your best bet.  So you would need pieces of article text that you know are 'Problems' and 'Solutions'.  Then the trained algorithm could identify new articles based on the training set.  But I imagine this is a lot of work to generate if you don't have this training data to begin with.

You can also try some unsupervised method, for example you may split each article into multiple overlapping parts, and compute TFIDF vectors of each part.  Do this for many articles, then cluster the TFIDF vectors.  You can then look at your clusters and hope one of them is generally capturing the 'Problem', and another the 'Solution'.  It will probably be very messy, and you'll have to clean it up with some post-hoc heuristic, depending on what the result looks like.  I think this approach would be hard to actually get to work well, but it would avoid having to obtain (or create) a labeled data set.

Finally, if you look at articles you may be able to devise generally good heuristics (e.g. the solution might be the first paragraph in a section titled 'Discussion' 70% of the time) to bolster both of these approaches, and if the article patterns are repeatable enough you may be able to just use heuristics to do a generally decent job.  Since articles are often very similarly structured, this is where I would start to see how far I can get, and how much I can simplify the problem using heuristics before passing the simplified problem to the other methods.


How to reduce the Root mean square error I have dataset which describe "how many passenger arriving in some airport " and I would like to predict how many passenger arriving in monthly bases for next year. The features that I have is the following :

year,
month,
airport,
number of passenger (monthly)

In the data, three out of of 50 airports are usually have huge passengers arriving.

I have used random forest classifier but the issue that i'm encounter is I have RMSE is high. As a result i see huge different between the actual value and predicting value. How to fix this issue ?

The problem is that you are using a classifier. You should use RandomForestRegressor or other type regressors.

Those three airports seems like leverage points. Check this wiki topic. You can log transform your target values or simply remove those 3 airports from your training set.

But still you have very less observations. You may want to add new data to your set. After train test split 35 observations for train 15 for test seems not so satisfactory.


Encoding features in sklearn Suppose I have a dataset of size(10000, 45). One of the features in the dataset is activity_type in which the values vary from 1 to 15 as shown below:
df = pd.read_csv('actTrain.csv')
df['activity_type'].head()

The output of the above code is as:
0    1
1    1
2    2
3    1
4    3
Name: activity_type, dtype: int64

Will encoding the activity_type in the above code using OneHotEncoder in sklearn improve the model in anyway? Is it necessary to encode that feature? And if yes, which one should I choose : LabelEncoder or OneHotEnocder?

LabelEncoder converts strings to integers, but you have integers already. Thus, LabelEncoder will not help you anyway.

Wenn you are using your column with integers as it is, sklearn treats it as numbers. This means, for example, that distance between 1 and 2 is 1, distance between 1 and 4 is 3. Can you say the same about your activities (if you know the meaning of the integers)? What is the pairwise distances between, for example, "exercise", "work", "rest", "leasure"?

If you think, that the pairwise distance between any pair of activities is 1, because those are just different activities, then OneHotEncoder is your choice.


Multiple testing of unbalanced data by R I have a general question of unbalanced data. I'm performing a T test on Group A and Group B. Group A has 20 data while Group B has 500. I set unequal variance (Welch) for the adjustment and the P-value is 0.01. Can I conclude that there is significance?

I have another idea that I'm not sure about. I randomly select about 20 samples from Group B multiple times and test against the Group A. Should I do some adjustment for the result? I am not sure if this is works or not. If it works, what kind of adjustment I should use for multi-testing?

Thank you!

I appreciate any replies!

It sounds like you are using Welch's t-test to compare groups of different sizes. That should work in your situation. It's more reliable that the standard Student's t-test when you have unequal variances and unequal sample sizes.

Note that sample sizes do not have to be identical for a Student's t-test. The important thing is that the variances are the same. If the variances are not equal, this is where the extra robustness of Welch's becomes helpful.

Whether or not you can conclude significance depends on what P-value you are you are looking for. It sounds like you are targeting p<.01, so if your result is less than .01, you can report that you found a significant difference.


String Values in a data frame in Pandas Suppose I have a data frame like this :
Hospital_name    State    Employees    ......
Fortis           Delhi    5000         ......
AIIMS            Delhi    1000000      ......
SuperSpeciality  Chennai  1000         ......

Now I want to use this data frame to build a machine learning model for predictive analysis. For that, I must convert the strings to float values. Also, some of these columns in Hospital_name and State contains 'NAN' values. In such a case, how should I prepare my data for building a model in Keras?

To convert from string to float in pandas (assuming you want to convert Employees and you loaded the data frame with df), you can use:
df['Employees'].apply(lambda x:float(x))

You have not given enough information about your input and expected output. So let us assume that hospital name or anything for that matter which is the input for your model is nan. You would like to remove it from the dataset because extracting features from 'nan' wouldn't make sense. Apart from that, if they are just other peripheral features, then it might be alright. In that case, if you wish to convert them into blank, then use:
df.replace(np.nan,' ', regex=True)`

Else, if you wish to remove that frame, you can check for nan using this.


In an SVM, does a more negative/positive decision score mean that it is further from the seperating hyperplane? For example, if I have a sample with a decision score of -6 and another  with a score of -3, which sample is closer to the hyperplane?

Also, does the probability of a sample belonging to a class increase with its respective distance from the hyperplane (larger distance from hyperplane = larger probability)?

I am using sklearn's decision_function and predict_proba functions to get these values if it makes a difference.

Thanks.

Larger scores do mean farther from the separating hyperplane.

The SVM model itself isn't modeling probability, so you can't quite say that farther points are considered more probably in one class or the other by the model. There are ways to estimate a probability (see the scikit implementation) for example and yes it will generally be true if so that farther points are more certain, according to that estimate.


Comparing excel data sets in Pandas Pretty new to Python, but as an SEO I'm looking at the benefits of using notebooks in my workflow.

I've got two excel files which I've cleaned and imported into a new notebook using pandas.

I'm trying to compare position changes and create a new dataframe with new columns to show previous, new, and changes in positions.

Have a look at the screengrabs[! of the data below. Thanks in advance.

You can do a pandas.DataFrame.join if you know how this works.

-- Edit: merge is apparently a better choice: See the example at the end.

I think you need an outer join on Keyword.

This should give a new DataFrame, that contains unique rows for the Keyword in both tables. Some entries may be NULL/None. This indicates that in the old or new table, the keyword was not present and you should treat is as a new keyword, or a keyword that has dropped from the list.

Rename the columns in the new table appropriately, and then apply a smart value between columns, taking into account that some values are NULL.

You can do a similar thing in Excel: https://superuser.com/questions/1023123/how-to-simulate-a-full-outer-join-in-excel

Edit:

Minimalistic example:

import pandas as pd

old = pd.DataFrame({'keyword': ['football', 'soccer', 'rugby'], 'position': [2, 1, 3]})
new = pd.DataFrame({'keyword': ['hockey', 'rugby', 'soccer'], 'position': [3, 2, 1]})

old.keyword = old.keyword.astype(str)
new.keyword = new.keyword.astype(str)

old.set_index(['keyword'])
new.set_index(['keyword'])

old = old.rename(columns={"position": "position_old"})
new = new.rename(columns={"position": "position_new"})

print(old)
print(new)

merged = pd.merge(old, new, how='outer', on='keyword')
print(merged)

Output:

    keyword  position_old
0  football             2
1    soccer             1
2     rugby             3
  keyword  position_new
0  hockey             3
1   rugby             2
2  soccer             1
    keyword  position_old  position_new
0  football           2.0           NaN
1    soccer           1.0           1.0
2     rugby           3.0           2.0
3    hockey           NaN           3.0


LSTM not learning with extra nontemporal data added after LSTM layer - Keras I have three different inputs I would like to send into my LSTM - the sequence of "words", extra temporal information, as well as extra nontemporal information.

Following Adam Sypniewski excellent answer to this question (Adding Features To Time Series Model LSTM) I came with this architecture:

word_input = Input(shape=(sequence_length,))
temporal_metadata = Input(shape=(sequence_length, 1))
nontemporal_metadata = Input(shape=(nontemporal_num_features,))

embedding = Embedding(input_dim=vocab_size, output_dim=100)(word_input)
temporal_info = concatenate(inputs=[embedding, temporal_metadata])
rnn = LSTM(50, dropout = .5, recurrent_dropout=.5)(temporal_info)
all_info = concatenate(inputs=[rnn, nontemporal_metadata])
dense_1 = Dense(10, activation='relu')(all_info)
dropout_1 = Dropout(.5)(dense_1)
dense_2 = Dense(1, activation='sigmoid')(dropout_1)

model = Model([word_input, temporal_metadata, nontemporal_metadata], dense_2)
model.compile(loss='binary_crossentropy', optimizer='adam'
model.fit(x=[my_words, my_temporal, my_nontemporal], y = my_y,
     epochs = 30,
     validation =([my_words_test, my_temporal_test, my_nontemporal_test], y_test))

This does not learn, and gets an AUC of 0.5

However if I don't add the nontemporal data and change the above code to

word_input = Input(shape=(sequence_length,))
temporal_metadata = Input(shape=(sequence_length, 1))

embedding = Embedding(input_dim=vocab_size, output_dim=100)(word_input)
temporal_info = concatenate(inputs=[embedding, temporal_metadata])
rnn = LSTM(50, dropout = .5, recurrent_dropout=.5)(temporal_info)
dense_1 = Dense(10, activation='relu')(rnn)
dropout_1 = Dropout(.5)(dense_1)
dense_2 = Dense(1, activation='sigmoid')(dropout_1)

model = Model([word_input, temporal_metadata], dense_2)
model.compile(loss='binary_crossentropy', optimizer='adam'
model.fit(x=[my_words, my_temporal], y = my_y,
     epochs = 30,
     validation =([my_words_test, my_temporal_test], y_test))

It works great. Can anyone please help me find what is going wrong in the first code sample that would make the model unable to learn. Thanks!

if anyone stumbles upon this - it seemed like my dimension of nontemporal data was too high and sparse. After filtering only for the most common/important variables in the nontemporal data, the model learned.


Application of Machine learning in modelling experimental system as a PhD topic I am working as a control Engineer in a high energy physics lab. I am looking for a PhD thesis topic.
Our experimental system is highly non linear and can't be modeled using standard white box approach. I am thinking of developing Neural Net models of the sensors attached to the experimental system. Also in this way I can model the system with primary inputs and outputs.

So the basic idea is to have a data driven system identification and data driven sensor model.

I wanted your feedback if this can be potential thesis problem.
Thanks.

It's very unlikely that anybody here has the expertise needed to assess the validity of a PhD topic, especially about a highly specialized (and unspecified) experimental system. You should discuss this with your potential academic supervisor, and I'd suggest you try to find a co-supervisor or collaborator expert in DL as well.

In general for a decent PhD topic you would need to make sure that it hasn't been done before and that there are reasonable grounds to think it's feasible. Study the state of the art and find out if anything similar has been published:

If something very close already exists, it might be too trivial
If there's nothing even remotely close to it, it's probably not feasible or very hard.

Good luck!


XGBOOST - different result between train_test_split and manually splitting I am trying to train XGBOOST model.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=43, stratify=y)

when I'm using train_test_split and pass the model X_train, Y_train and for eval_set X_test, Y_test, The model seems to be a very good one.
CM example:

But when I manually split the Dataset :
splitValidationIndex = round(dataset.shape[0]*0.6)
splitTestIndex = round(dataset.shape[0]*0.8)
X_train = X[:splitValidationIndex]
y_train = y[:splitValidationIndex]

Pass it to fit
X_val = X[splitValidationIndex:splitTestIndex]
y_val = y[splitValidationIndex:splitTestIndex]

Pass it to eval_set
X_test = X[splitTestIndex:]
y_test = y[splitTestIndex:]

Check the model prediction on that
that produced a much worse model
example:

What am I missing/doing wrong?

All looks correct, but you have to get to know about some details.

train_test_split splits arrays or matrices into a random train and test subsets.
If you not specifying random_state, you will get a different result, it means that
every time when you run your code train and test datasets would have different values each time.

If you fixed specific value like random_state = 42 then yours data in test and train set conserve the same values.

So in the first way splitting,  model is learning on the specific(always the same) chunk of data, which could give a good results, in second way every time model is learning on different chunks of data, so results could be worse(by the way, its looks like overfitting on chunk(in first way of split).

In general no matter what way you choose, but you have to use cross-validation to create a good performing model.


What does the output of model.predict function from Keras mean? I have built a LSTM model to predict duplicate questions on the Quora official dataset. The test labels are 0 or 1. 1 indicates the question pair is duplicate. After building the model using model.fit, I test the model using model.predict on the test data. The output is an array of values something like below:

 [ 0.00514298]
 [ 0.15161049]
 [ 0.27588326]
 [ 0.00236167]
 [ 1.80067325]
 [ 0.01048524]
 [ 1.43425131]
 [ 1.99202418]
 [ 0.54853892]
 [ 0.02514757]

I am only showing the first 10 values in the array. I don't understand what do these values mean and what is the predicted label for each question pair?

The output of a neural network will never, by default, be binary - i.e. zeros or ones. The network works with continuous values (not discrete) in order to optimise the loss more freely in the framework of gradient descent.

Have a look here at a similar question that also shows some code.

Without any kind of tweaking and scaling, the output of your network is likely to fall somewhere in the range of your input, in terms of its nominal value. In your case, that seems to be roughly between 0 and 2.

You could now write a function that turns your values above into 0 or 1, based on some threshold. For example, scale the values to be in the range [0, 1], then if the value is below 0.5, return 0, if above 0.5, return 1.


Elements of Statistical Learning - question on p. 18 In the following expression:

What do $E_{X}$ and $E_{Y|X}$ mean, respectively?

In case helpful, I note that the authors derived that expression from the following:

Based on the above, how would one conclude that that minimizing $EPE(f)$ can be accomplished as follows?

Maybe this helps:

On YouTube, there's a video with explanations for the main topics in Introduction to Statistical Learning (sorry, not for ELS).

https://www.youtube.com/watch?v=3jQs02dbfrI


String handling by OneHotEncoder I am reading everywhere on new questions and blogs that since version 0.20, OneHotEncoder is able to handle string features.
Moreover, the documentation is what looks more ambiguous. Here are the first two lines from the documentation:

Encode categorical integer features as a one-hot numeric array. The input to this transformer should be an array-like of integers or
strings, denoting the values taken on by categorical (discrete)
features.

First line says it

encodes categorical integer features

and the next line says

input should be array like of integers or strings.

When i tried it, i still got the value error.
print(X.columns)
encoder = OneHotEncoder(categorical_features=[1,4,5])
encoder.fit(X)

Index(['age', 'sex', 'bmi', 'children', 'smoker', 'region'], dtype='object')
ValueError: could not convert string to float: 'female'

I am aware of the means to handle encoding of string features with LabelEncoder, ColumnTransfomer and pd.getDummies() but specifically want to understand about this.

This seems to fail only when you're using categorical_features, which was deprecated at the same time the encoder was extended to strings.  Using the now-recommended ColumnTransformer to specify which columns to encode works with strings (as does applying the encoder to the entire frame, though that's not what you want, with features like bmi).

E.g.,

onehot = OneHotEncoder(...)
cat_cols = [1,4,5]
preproc = ColumnTransformer(transformers=[('onehot', onehot, cat_cols)],
                            remainder='passthrough')
preproc.fit_transform(X)


Size of datasets over years I am looking for statistics, to understand the evolution of the size of the (public) dataset over the years.
I just found the following statistics:

The poll of KDnuggets that actually shows that over the years is the following:

Gigabytes still rule: Majority of answers (56% in 2018, 57% in 2016, 56% in 2015, 54% in 2014, 53% in 2013) are in Gigabyte range. The overall median response was again between 11 and 100 GB (which comfortably fits on one laptop) for each year since 2012.

Do you know where to find better statistics?
I guess that the average size of the dataset has increased.
If not, I would like to see if at least the size of company-data available has increased.

After having a look at the article you quote of KDNuggets, in this article they analyze the data from this last year (poll), but they have been doing that for several past years.
Perhaps by looking at past years you can extract some better data.
Also in this other forum https://opendata.stackexchange.com/
they talk about open data, maybe here you can get a more precise answer


Pandas Groupby makes kernel die in Jupyter notebook/Python I have a groupby in jupyter-notebook that takes ages to run and after 10 minutes of running it says 'kernel died...',

The groupby looks like this:

df1.groupby(['date', 'unit', 'company', 'city'])['col1',
'col2',
'col3',
'col4',
  ...
'col20'].mean()

All of the 'col' columns are float values. I am running everything locally. Any ideas?

UPDATE:

The shape of df1 is:

(1360, 24)

Memory and dtypes:

dtypes: category(3), datetime64[ns](2), float64(17), int64(2)
memory usage: 266.9 KB

The unique size of city, date, company, unit:

len(df1.date.unique()) = 789
len(df1.unit.unique()) = 76
len(df1.company.unique()) = 205
len(df1.city.unique()) = 237

I have 16GB of memory on MacBook Pro.

UPDATE 2:

It works only if I have date and unit inside the groupby columns as the only 2 columns. If I add either a company or city, it doesn't work anymore, it keeps running indefinitely.

I thought it might be because of the different types being used in the columns, but I created an example below, which works fine over mixed column types. The only real different is the size - that is why I think you are probably running out of memory.

  Working example

I use int, str and datetime objects:

In [1]: import pandas as pd

In [2]: import datetime

In [3]: df = pd.DataFrame({'Branch': 'A A A A A A A B'.split(),
                           'Buyer': 'Carl Mark Carl Carl Joe Joe Joe Carl'.split(),
                           'Quantity': [1, 3, 5, 1, 8, 1, 9, 3],
                           'Date':[datetime.datetime(2013, 1, 1, 13, 0),
                                   datetime.datetime(2013, 1, 1, 13, 5),
                                   datetime.datetime(2013, 10, 1, 20, 0),
                                   datetime.datetime(2013, 10, 2, 10, 0),
                                   datetime.datetime(2013, 10, 1, 20, 0),
                                   datetime.datetime(2013, 10, 2, 10, 0),
                                   datetime.datetime(2013, 12, 2, 12, 0),
                                   datetime.datetime(2013, 12, 2, 14, 0)]})

In [4]: df
Out[4]:
  Branch Buyer  Quantity                Date
0      A  Carl         1 2013-01-01 13:00:00
1      A  Mark         3 2013-01-01 13:05:00
2      A  Carl         5 2013-10-01 20:00:00
3      A  Carl         1 2013-10-02 10:00:00
4      A   Joe         8 2013-10-01 20:00:00
5      A   Joe         1 2013-10-02 10:00:00
6      A   Joe         9 2013-12-02 12:00:00
7      B  Carl         3 2013-12-02 14:00:00

In [5]: df.shape
Out[5]: (8, 4)

Now I just repeat the dataframe again, but add one hour to each of the datetime values, just to increase the number of groupby combinations to expect:

In [14]: df.iloc[0:8, 3] += datetime.timedelta(hours=1)

Now perform a groupby over all columns, and sum only on Quantity (it is my only numeric column).

The reuslts are as expected:

In [16]: df.groupby(["Branch", "Buyer", "Quantity", "Date"])["Quantity"].sum()
Out[16]:
Branch  Buyer  Quantity  Date
A       Carl   1         2013-01-01 13:00:00    1
                         2013-01-01 14:00:00    1
                         2013-10-02 10:00:00    1
                         2013-10-02 11:00:00    1
               5         2013-10-01 20:00:00    5
                         2013-10-01 21:00:00    5
        Joe    1         2013-10-02 10:00:00    1
                         2013-10-02 11:00:00    1
               8         2013-10-01 20:00:00    8
                         2013-10-01 21:00:00    8
               9         2013-12-02 12:00:00    9
                         2013-12-02 13:00:00    9
        Mark   3         2013-01-01 13:05:00    3
                         2013-01-01 14:05:00    3
B       Carl   3         2013-12-02 14:00:00    3
                         2013-12-02 15:00:00    3
Name: Quantity, dtype: int64

  Break your problem down

It might be difficult to break down your problem, because you need to whole data for the groupby operation. You could however save each of the groups to disk, perform the mean() computation on them separately and merge the results yourself. The name of each group is actually the combination of the groupby columns selected. This can be used to build the index of the reuslting dataframe.

It could look something like this:

for name, group in df1.groupby(['date', 'unit', 'company', 'city']):
    print("Processing groupby combination: ", name)    # This is the current groupby combination
    result = group.mean()
    _df = pd.DataFrame(index=[name], data=[result])
    _df.to_csv("path/somewhere/" + name + ".csv

You will then have a folder full of the results for each group and will have to just read them back in and combine them.

  Other methods

It is known that Pandas does not handle many operations on huge datasets very efficienty (compared to e.g. the data.table package). There is the Dask package, which essentially does Pandas things in a distributed manner, but that might be overkill (and you'll of course need more resources!)


How can you adjust a prediction based on features in the future being different than predicted? I have a model that takes mostly cumulative data, and makes a prediction. It's not baseball, but I'm using this as a pretty accurate analogy. You put in all the totals so far, and it make a prediction of where a target will be in the future.

games_played|runs|runs_against|hits|hits_against|games_won
          20| 100|         120| 200|         240|       10

Expected games won after 5 more games: 12

What I'd like to do is answer questions like "How many runs are we expected to have in N games? And if we had more runs than that, how many more wins would that get us?"

             |games_played|runs|runs_against|hits|hits_against|games_won
          now|          20| 100|         120| 200|         240|       10
    predicted|          25| 120|         150| 240|         300|       12
user_adjusted|          25| 130|         140| 260|         280|       13

The only way I can think of to do this right now is to have many models, one for each feature, and put all the predicted features into one other model. Then users can update individual features. That feels like excess work and prone to errors, but I could be totally wrong. I can't just linearly forecast the features in the future because the relationship is more complex.

What is the algorithm/method/tool that's best suited to this? I'm using scikit-learn right now.

Try these two approaches:

First:

Make a model, using any ML algorithm and divide your data into train and test.
Now using the previous features, check the train and test accuracy.

Now add the new features to the previous ones, again divide data into train and test.  Check the train and test accuracy.

If the new features help improve the test accuracy then only use them otherwise just skip them.

Second:

Add the new features to the previous dataset and set their values as null/0. So while predicting the results before wont be affected by their values. And later when these features do get values, they will help improve the predictions.

I hope this helps!


Commercial Text Summarization Tools I'm looking for commercial text summarization tools (APIs, Libraries,...) which are able to perform any of the following tasks:

Extractive Multi-Document Summarization (Generic or query-based)
Extractive Single-Document Summarization (Generic or query-based)
Generative Single-Document Summarization (Generic or query-based)
Generative Multi-Document Summarization (Generic or query-based)

There are a couple of open source options I know of -

LibOTS - http://libots.sourceforge.net/

DocSum - http://docsum.sourceforge.net/docsum/web/about.php

A couple of commercial solutions -

Intellix Summarizer Pro - http://summarizer.intellexer.com/order_summarizer_pro.php

Copernic Summarizer - http://www.copernic.com/en/products/summarizer/

And this one is a web service -

TextTeaser - http://www.textteaser.com/

I'm sure there are plenty of others out there.  I have used Copernic a good deal and it's pretty good, but I was hoping it could be automated easily, which it can't - at least it couldn't when I used it.