Strategy for dealing with giant sample size There's a not well-known fact in statistics. That as your sample size increases, more p-values become significant.

I'm working with a massive sample size consisting of 3 million samples (~ 1% of the U.S. population). It's a Logistic Regression, and p-values of several relations are significant at 5%, 1% and ~0%.

I fielded my paper's abstract to several journals to see if there was interest. Only a German journal I submitted to (those smart Germans, really) caught on that an extremely large sample size like this might more easily produce significant p-values and that I'd need to somehow adjust for that.

To be clear, an excessively large sample size doesn't produce spurious p-values. It's that more effects with tiny effect sizes start showing significant p-values.

I want your views on strategies for how to deal with such a situation, to produce reliable research. Some strategies I've thought of are:

Conduct the research as is, and only consider as relevant results that have effect sizes that are above a certain threshold (What threshold though?)
Do what they do in Data Mining. Split up my data into train, validate, and test sets. (Is this done with Logistic Regression though? Can anyone point me to past papers where the technique is illustrated?)
Skip the usual Logistic Regression and instead use a Data Mining technique such as CART (which may also use Logistic Regression internally).

I'd like your views on the relative worth of these strategies. Other suggestions, too, would be welcome. Especially welcome will be pointers to prior papers where the authors illustrate how to deal with a similar problem.

My link in the comment has useful advice. I'd like to emphasis:

This is a very well known fact in statistics
Large sample size is good. There's nothing wrong with more and better quality data.
It's silly to split up the data in your second point. You are voluntarily giving away information and statistical power. People do that for training an unbiased model, but not to reduce sample size for the p-values.
There's no reason to switch to another model. The standard error in your logistic model should be small, which makes it ideal for predictive modelling. Your model should be robust and stable.
You should do your analysis on effects size if you have a large sample. Read @Sympa's answer in the link I post. What's your magnitude of the effect size? What's the relative effects?
The threshold you have for your first point is domain specific. If you write a paper, you should know it very well. You could compare your data with the literature. Simple descriptive statistics like percentage and how it compare with the standard deviation (which is independent to sample size) would be useful.

Please also take a look at https://stats.stackexchange.com/questions/125750/sample-size-too-large.

There is a long but excellent article on Bayes factor and p-value:

  https://replicationindex.wordpress.com/2015/04/30/replacing-p-values-with-bayes-factors-a-miracle-cure-for-the-replicability-crisis-in-psychological-science/

I should draw to your attention the following paragraph:

  The more interesting argument against p-value is not that significant results in large studies are type-I errors, but that these results are practically meaningless. To make this point, statistics books often distinguish statistical significance and practical significance and warn that statistically significant results in large samples may have little practical significance. This warning was useful in the past when researchers would only report p-values (e.g., women have higher verbal intelligence than men, p < .05). The p-value says nothing about the size of the effect. When only the p-value is available, it makes sense to assume that significant results in smaller samples are larger because only large effects can be significant in these samples. However, large effects can also be significant in large samples and large effects in small studies can be inflated by sampling error. Thus, the notion of practical significance is outdated and should be replaced by questions about effect sizes. Neither p-values nor Bayes-Factors provide information about the size of the effect or the practical implications of a finding.

The reviewer was correct. Neither p-value nor the bayes-factor reveals anything about the effect sizes.


How to work with predictor variables which are all categorical but the target variable is an integer? I have historical data with time (hour, day, etc.) and region code linked to number of events in the integer format. I was starting to build a model in R to predict number of events given time and region. I explored SVM in R but seems like it is a non-starter with categorical variables. Any other approach which might of use here. Thanks for your response.

I think you need to create a dummy variables from your categorical variables. Almost all machine learning algorithms can't work with categorical variables.


Pandas dataframe with multiple observations per model I currently have a pandas dataframe with the following format

    Model      Metric                     Value
--------------------------------------------------------------
0     Ours     Accuracy  [0.79, 0.79, 0.82]
1   Theirs     Accuracy  [0.68, 0.56, 0.64]
2     Ours  Sensitivity  [0.64, 0.55, 0.55]
3   Theirs  Sensitivity  [0.82, 0.82, 0.78]
4     Ours  Specificity  [0.68, 0.48, 0.6]
5   Theirs  Specificity  [0.68, 0.48, 0.6]

In the evaluation script I am writing I want to be able to take into account situations where the training of a model is repeated multiple times, the results are stored in an numpy array (Value column). For visualization with seaborn, i believe i need a long form where it looks something like this:

    Model      Metric                     Value
--------------------------------------------------------------
0     Ours     Accuracy                   0.79
1     Ours     Accuracy                   0.79
2     Ours     Accuracy                   0.82
3     Theirs   Accuracy                   0.68
4     Theirs   Accuracy                   0.56
5     Theirs   Accuracy                   0.64
6     Ours     Sensitivity                0.64
...   ....     ...                        ...

I cannot figure out how to do this.

# create the df
df = pd.DataFrame({'model':['our','theirs'],'metric':['acc','sen'],'value':[np.array([1,2,3]),np.array([4,5,6])]})
# unzip the array into three columns
df[['ex1','ex2','ex3']]=pd.DataFrame(df['value'].to_list(),index=df.index)
# melt df to long df
df.melt(id_vars=['model','metric'],value_vars=['ex1','ex2','ex3'])

You may need to sort the dataframe as you like.


Is it OK to try to find the best PCA k parameter as we do with other hyperparameters? Principal Component Analysis (PCA) is used to reduce n-dimensional data to k-dimensional data to speed things up in machine learning. After PCA is applied, one can check how much of the variance of the original dataset remains in the resulting dataset. A common goal is keeping variance between 90% and 99%.

My question is: is it considered a good practice to try different values of the k parameter (size of the resulting dataset's dimension) and then check the results of the resulting models against some cross-validation dataset in the same way as we do to pick good values of other hyperparameters like regularization lambdas and thresholds?

Your emphasis on using a validation set rather than the training set for selecting $k$ is a good practice and should be followed. However, we can do even better!

The parameter $k$ in $\text{PCA}$ is more special than a general hyper-parameter. Because, the solution to $\text{PCA}(k)$ already exists in $\text{PCA}(K)$, for $K > k$, which is the first $k$ Eigenvectors (corresponding to $k$ largest Eigenvalues) in $\text{PCA}(K)$. Therefore, instead of running $\text{PCA}(1)$, $\text{PCA}(4)$, ..., $\text{PCA}(K)$ separately on training data, as we do for a hyper-parameter in general, we only need to run $\text{PCA}(K)$ to have the solution for all $k \in \{1,..,K\}$.

As a result, the process would be as follows:

Run $\text{PCA}$ for the largest acceptable $K$ on training set,
Plot, or prepare ($k$, variance) on validation set,
Select the $k$ that gives the minimum acceptable variance, e.g. 90% or 99%.

And, N-fold cross validation would be as follows:

Run $\text{PCA}$ for the largest acceptable $K$ on N training folds,
Plot, or prepare ($k$, average of N variances) on held-out folds,
Select the $k$ that gives the minimum acceptable average variance, e.g. 90% or 99%.

Also, here is a related post that asks "why do we choose principal components based on maximum variance explained?".


Predicting products to be sold in a store - problem formulation I have a data from a store for the products that sold since more than 5 years. Each sell process has a customer id, date, and the quantity of the product.
I want to build a machine learning model to predict the products that will be sold in the next day/s for each of the customers, giving that I have N products (~2k) and M customers (~50).
I am not able to formulate this problem. It's a regression task (probably), but I don't know how can I formulate it to predict the products that a given customer will buy.
Since we have N products, this doesn't mean that a customer will buy all of them; x customer might buy only 5 products in the next day.

To answer your question, in that dataset the first thing to do is to determine the types of products in that product sold column or whatever. If you have like 3 or more different kinds of products to be sold. Then you can rename those types and use a classifier algorithm to make your prediction. So the product sold will serve as your target variable and other closely correlated variables will now serve as your predicting variables.
So to add to the above, you want to predict the number of products to be purchased by a customer, then I suggest you look out for good multi-target regression models.


Running Tensorflow MobileNet from Java I am trying to run Tensorflow for image recognition (classification) in Java (JSE not Android).

I am using the code from here, and here.

It works for Inceptionv3 models, and for models retrained from Inceptionv3.

But for MobileNet models it does not work, (such as following this article).

The code works, but gives the wrong results (wrong classify labels). What code/settings are required to use MobileNet from Java?

The code that works for Inceptionv3 is

try (Tensor image = Tensor.create(imageBytes)) {
            float[] labelProbabilities = executeInceptionGraph(graphDef, image);
            int bestLabelIdx = maxIndex(labelProbabilities);
            result.setText("");
            result.setText(String.format(
                            "BEST MATCH: %s (%.2f%% likely)",
                            labels.get(bestLabelIdx), labelProbabilities[bestLabelIdx] * 100f));
            System.out.println(
                    String.format(
                            "BEST MATCH: %s (%.2f%% likely)",
                            labels.get(bestLabelIdx), labelProbabilities[bestLabelIdx] * 100f));
        }

This works of Inceptionv3 models, but not MobileNet,
Gives the error, "Expects args[0] to be float but string is provided"

For MobileNet we tried the code,

try (Graph g = new Graph()) {
  GraphBuilder b = new GraphBuilder(g);
  // Some constants specific to the pre-trained model at:
  // https://storage.googleapis.com/download.tensorflow.org/models/inception5h.zip
  //
  // - The model was trained with images scaled to 224x224 pixels.
  // - The colors, represented as R, G, B in 1-byte each were converted to
  //   float using (value - Mean)/Scale.
  final int H = 224;
  final int W = 224;
  final float mean = 128f;
  final float scale = 1f;

  // Since the graph is being constructed once per execution here, we can use a constant for the
  // input image. If the graph were to be re-used for multiple input images, a placeholder would
  // have been more appropriate.
  final Output<String> input = b.constant("input", imageBytes);
  final Output<Float> output =
      b.div(
          b.sub(
              b.resizeBilinear(
                  b.expandDims(
                      b.cast(b.decodeJpeg(input, 3), Float.class),
                      b.constant("make_batch", 0)),
                  b.constant("size", new int[] {H, W})),
              b.constant("mean", mean)),
          b.constant("scale", scale));
  try (Session s = new Session(g)) {
    return s.runner().fetch(output.op().name()).run().get(0).expect(Float.class);
  }
}

This works, but gives the wrong labels.

Don't give to use in java :)

I had the same problem, try to change the scale value, I had the same labels from java and python after that.

        // - The model was trained with images scaled to 224x224 pixels.
        // - The colors, represented as R, G, B in 1-byte each were converted to
        // float using (value - Mean)/Scale.
        final int H = 224;
        final int W = 224;
        final float mean = 117f;
        final float scale = 255f;


how to evaluate top n recommendation system with movie lens dataset? Based on my research a recommendation system are a subclass of information filtering system that seek to predict the "rating" or "preference" that a user would give to an item. And I'm currently developing a collaborative filtering recommendation system, and basically is recommends the top 'n' items to a user (I've used the user-item algorithm).

So on top of that, I will try to evaluate my recommendation system using the movie lens dataset, and basically from my research typical evaluation measures for top-N recommendation are normalized discounted cumulative gain (NDCG) and precision/recall. So my question is how can I evaluate using these metrics (or if you have any other suggestion for another metric) using the movie lens dataset (since it's a rating of users to items).

Thanks. Any suggestions are welcome.

For various metrics feel free to look at various benchmarking libraries including MyMediaLite and LibRec. If you are doing a TOP N approach, then the way to evaluate this using a Movielens system is simple convert the ratings into binary likes and dislikes based on some threshold. Essentially you would take the "likes" of a user. Find the user is the testing set. Find the number of items that the user likes and see if the recommendation guesses the missing data correctly.

The key to thinking about these datasets is that the fact that they chose to rate a movie at all means the user actually went out an saw the movie. The fact that they saw the movie indicates an implicit approval of the basic premise. As such, a good recommender should be able to predict which items that user rates in the future.

The easiest way to evaluate the recommendation system though is just to plug into an evaluation framework such as LibRec, MyMediaLite, or various other ones and compare the returned metrics to your algorithm. These libraries typically have code that will generate the testing, training, and evaluation subsets for you and all you have to do is hook up your code as a module.

As a bonus, feel free to add your recommendation algorithm as a pull request to the library so other researchers can try it too and you'll have an excellent open source implementation of your algorithm that future researchers can reference in their own work.


Decision trees for Rstudio v3.3 Anyone know the best packages to build a decision tree in Rstudio v3.3?

I want to look at some data-driven segmentation for my data. I was thinking of doing chaid analysis (mainly because this is why I've done in the past).

I've looked around myself, but can't find the best package for this version!

The "best" package depends on your goals and data really.

A few tree / forest packages that I've come across:

randomForest - an implementation of the original algorithm
ranger - a flexible implementation designed to run on high-dimensional data (my personal favourite forest package)
party's ctree / cforest functions - conditional inference trees / forests
rpart

Some of these packages focus on forests rather than trees; you can coerce the forest algorithm into building a tree by flexing appropriate parameters.


Is there a name for a scale which mixes ordinal and nominal? The textbooks I have differentiate between nominal, ordinal, interval and ratio scales. The ordinal scale is quite popular in the wild, used for basically all subjective data, and also for dividing interval and ratio scales into better digested bins.

But when dealing with the data itself, the need for different "exceptions" arises. Sometimes it is just different kinds of null flavors, which can be written off as a meta data issue. But there are also more interesting cases. I have encountered satisfaction measurements where the overall satisfaction with a feature is considered to be "excellent", "good" or "bad" if the distribution of answers peaks at the expected points in the scale, but "polarising" if the distribution is U-shaped, and "unclear" if there is a flattish or weirdly multimodal distribution. Obviously, the "excellent", "good" and "bad" part is ordinal, but the last two are nominal. Other examples include a "prefer not to say" box in self-reported measures, if the reluctance of giving information is meaningful under the employed model.

I am pretty sure that we don't yet have qualitative methods to analyze these mixed scales. But I hope that the situation is at least recognized in theoretical texts, even if the only recommendation is "treat them like nominal". Also, while a mixed statistical analysis is difficult, maybe there are at least visualization approaches capable of representing the true complexity of the situation. But I don't know how to look for such literature, since I have never encountered a name for this kind of scale.

Does somebody know of a name (even if it is not widely established) and can point to sources which define it and deal with it?

One option is create new features. One feature that is completely nominal, and another feature that is completely ordinal.
Otherwise, the scale types are "rounded down". There is no danger in treating ordinal data as nominal. However, treating truly nominal data as ordinal can cause analysis problems.
There is not much literature on mixing scale types because that could easily lead to misinterpretation of results.


Using Scorer Object for Classifier Score Method for scikit-learn I have written my custom scorer object which is necessary for my problem and which I've called "p_value_scoring_object".

For the function sklearn.cross_validation.cross_val_score one of the parameters is "scoring", which allows to use this scorer object.

However, this option is not available for the score method of a classifier. Is sklearn just lacking that feature, or is there a way around it?

from sklearn.datasets import load_iris
from sklearn.cross_validation import cross_val_score
from sklearn.tree import DecisionTreeClassifier
clf = DecisionTreeClassifier(random_state=0)
iris = load_iris()
cross_val_score(clf, iris.data, iris.target, cv=10,scoring=p_value_scoring_object)

This works. However, this doesn't:

clf.fit(iris.data,iris.target)
clf.score(iris.data,iris.target,scoring=p_value_scoring_object)

Why should it work? scoring is an argument of cross_val_score, not score. The documentation says score "returns the mean accuracy on the given test data and labels." This behavior is not specific to DecisionTreeClassifier, but common to classifiers in sklearn.

Perhaps you want sklearn.metrics?


Difference between multinomia logistic regression and N 1 vs all binary Is there a fundamental difference between building a set on N logistic regressions in 1 vs all fashion, as compared to training a single mutlinomial logistic regression? Put another way, are there any optimization techniques that treat the 1 to N classes logistic regression problem in a way that's markedly different from N independent regressions?

Intuitively is seems like the answer ought to be yes, since there should be a lot of information sharing between various problems if two classes are similar. But since I'm not entirely well versed on how  common 1 to N solvers actually work, I can't tell if I'm right or these problems are treated in ways that fundamentally the same.

I think that we can see that there could be a difference between the two models, but I'm not entirely sure. Googling the matter revealed a few arcane discussing about the subject, but I was unable to find an authoritative discussion of the matter.

Quote from Alan, Agresti. "Categorical data analysis." A John Wiley and Sons, Inc. Publication, Hoboken, New Jersey, USA (2002), page 273:

  The separate-fitting estimates differ from the ML estimates for simultaneous fitting of the J-1 logits. They are less efficient, tending to have larger standard errors. However, Begg and Gray 1984 showed that the efficiency loss is minor when the response category having highest prevalence is the baseline.


Can all statistical algorithms be parallelized using a Map Reduce framework Is it correct to say that any statistical learning algorithm (linear/logistic regression, SVM, neural network, random forest) can be implemented inside a Map Reduce framework?  Or are there restrictions?

I guess there may be some algorithms that is not possible to parallelize?

Indeed there are:

Gradient Boosting is by construction sequential, so parallelization is not really possible

Generalized Linear Models need all data at the same time, although technically you can parallelize some of the inner linear algebra nuts and bolts
Support Vector Machines


Convert filters pre-trained with ImageNet to grayscale? I am wondering how to convert caffe reference model trained with ImageNet (color pics) for grayscale image to save memory and to speed up.

The filters in the caffe convolution layers are different for RGB but this can be, for example, averaged to form one channel filter.

Since my data is grayscale and speed at test (prediction) is important for me, this may help.

I can of course convert my grayscale images in RGB representation to reuse pretrained convolution layers as they are. But I want to avoid it if possible.

My questions are:

Would this be possible?

How exactly one should combine filters? (Average?)

After filters converted, weights between conv layers need to be re-trained?

I doubt either of your proposed approaches will work, these convolutional layers learn all kind of relationships between the colors which will be non-existent in grayscale, upsampling it to RGB makes your input space very different from the input space that ImageNet was trained on and averaging these filters of the trained ImageNet model makes a whole lot of assumptions that are unlikely to hold. The second approach does seem more promising than the first, so if you want to test an approach I would go for the second one. It's possible that it is at least a better initialization than randomly. I think you will end up needing a lot of data and computation anyway, so it might be better to start off with your own network shape and start from scratch.


Confused by kmeans results I am using kmeans to cluster some data with 2 features. Not sure I understand why kmeans is producing the clusters I see:

Why would kmeans not cluster these points in a way that matches what we would expect visually looking at the data? Why are seemingly random points in the middle of visual clusters being put into a second cluster like that?

The code I am running:

cols = ['col1', 'col2']
features = map(lambda x: df[x], cols)
input = np.matrix(list(zip(*features)))

scaler = StandardScaler()
scaler.fit(input)
input_scaled = scaler.transform(input)

algo = KMeans(n_clusters=2)
algo.fit(input_scaled)
df['cluster'] = pd.Series(algo.labels_)

sns.lmplot(x=cols[0],y=cols[1],data=df, fit_reg=False, hue='cluster')

Got it. Thanks to everyone who helped. The issue had nothing to do with kmeans.

I did not realize that when you do this:

dataframe[col] = Series, the series gets merged based on the index, as opposed to a simple appending of the column.

My dataframe had been filtered before I generated the lists of features, so the index was not 0,1,2 but rather 0,2,5, etc. I needed to perform reset_index() on the original dataframe before assigning a new column in the dataframe to the labels from the algorithm.


What does it mean to be “stable to deformation”? In the context of image classification, what does it mean to be stable to deformation? Say I were trying to classify digits, what would the difference be between an operation that is stable vs unstable to deformation?

Images are susceptible to deformations, i.e. afine or arbitrary deformations, such as "melting" effects.

In computer vision algorithms for non-constrained environments it may be desirable for a predictive model to be "stable", i.e. robust, ideally invariant, to arbitrary transformations that are common in that problem. E.g. a digit classifier would benefit from been stable to common paper deformations such as those caused by wrapping and unwrapping a long letter.

It is said that pooling layers insert certain stability to these deformations, this is analysed by Ruderman on his paper Learned Deformation Stability in Convolutional Neural Networks and in Pooling is neither necessary nor sufficient for
appropriate deformation stability in CNNs.

Some features, such as SIFT, BRISK and HoG try to deal with the most common deformations in image (Scale and Rotation). Most of convolution-dependent methods are already invariant to shifting as that is a feature of convolutional filtering itself.


Evaluating model with categorical target variables I converted all the numeric target variables of MNIST dataset into categorical variables. So, 0 became zero, and so on. Next, I tried the following model on this dataset:
from sklearn.linear_model import SGDClassifier
from sklearn.multiclass import OneVsRestClassifier

sgd_clf = OneVsRestClassifier(SGDClassifier(random_state=42))
sgd_clf.fit(X_train, y_train)

When I try sgd_clf.predict(X_test), I get an array of categorical values. I want to understand how to evaluate my model since confusion_matrix is not working on this.

You could convert your categorical outputs back into their numeric equivalents, and then use the confusion matrix and normal


Creating the optimal set of utterances to train a natural language processing engine Context

I am using natural language processing engines such as IBM's Watson Conversation and Microsoft's LUIS, which take a sentence and classify its intent. For example, "i want to buy food" => "Buy_Food".

Problem

I am trying to come up with the best set of utterances to train those engines to understand natural language for a given application. My goal is to improve the accuracy of the NLP engine predicting the correct intent.

Suppose I have a few thousand utterances, each with a known intent. I need to come up with a set of just a few hundred best utterances that will be used to create a model in the NLP engine.

What I tried

At first I tried to take a few hundred sentences randomly, create an NLP engine model, and test its accuracy using a few hundred other sentences from my set. Then I would replace one sentence in the NLP engine model, and re-run the test again. At first, Microsoft's LUIS would guess the intent correctly around 90-95% of the time, and after a few dozen iterations the accuracy goes up to 91-96% - very slightly.

I am also thinking about using part-of-speech tagging to not just replace one sentence randomly, but to strive for syntactical diversity of those sentences.

Reason behind doing this

The reason why I am doing all of this is that it seems that dumping more utterance examples into the NLP engine doesn't seem to make it better; in fact, it seems to make it worse.

This is why I am trying to think of a way to create a limited set of utterances that will be used to train the NLP model and improve the rate of how often the engine guesses the correct intent.

Great question, I've been checking back to see if anyone responds but nothing yet, so I'll do my best.

It seems like me, you have also found out that simply adding more and more training data does not increase the model’s accuracy.

You need to make sure your model contains balanced, representative samples for each intent. Simply picking utterances at random means they may not be represent all of the types of language a user may use.

For example, let’s say you had two intents - “buy_food” for buying some food,  and “list_menu”  for finding out what is available for order. If you were to heavily populate buy_food with lots of utterances that begin with “I want to”, the NLP engine will associate the phrase “I want to” with buying food, even though someone may quite rightly say “I want to see what’s on the menu”. To overcome this, make sure you balance out each intent’s utterances and don’t focus on phrases that aren’t specific to the intent. In this case, you could counterbalance by having some utterances in your “list_menu” intent such as “I want to see what’s on the menu”.

As a rule, for each intent I would try and include up to 5 or 6 ways to say something, and then for each of these, include 4 or 5 variations of this, leaving you with about 25 samples for an intent in total.

There is also the question around how you plan your intents. For example, do you have a “buy_food” intent and a “buy_drinks” intent, or should you simply have a “buy” intent, and then have the thing they want to buy as an entity? (think of your intent as a method, and your entity as a parameter) My suggestion would be to keep your intents broader and have the specifics handled by entities. This will help avoid confusion between intents.

I recommend checking out a tool called QBOX. It connects to LUIS/Watson/DialogFlow and lets you score and visualise intents so you can see any possible confusion between them. This should help you to plan your intent structure and visualise the confusion between your intents, and know if you're causing regressions as you expand your model.
(Disclaimer: this is a product I am working on, and built precisely because of the type of problem you're having, so I've not linked to it, but if you Google the name along with the term 'chatbot', you should find it!)


survey data analysis (discrete data) I did small survey and get such data:

|-------------| Yes | No | Dont_Know |
|-------------|     |    |           |
| Employee    | 60  | 5  | 5         |
| Workers     | 17  | 0  | 1         |
| Businessmen | 71  | 5  | 10        |
| Jobless     | 4   | 30 | 0         |

R code

dt <- data.frame(workers = c("Employee",
                             "Workers",
                             "Businessmen",
                             "Jobless"),
                 yes = c(60,17,71,4),
                 no = c(5,0,5,30),
                 dont_know = c(5,1,10,0)
                )

What kind of test I must do, if I want to show, that the Jobless people are often choosing No answer?
Is the difference between Jobless and Businessmen answers significant?
And what is about other groups?
What another information I can get from such data or what kind questions I can ask from such data?

Here are some things to try.

Plot a bar graph. The bar graph will clearly show jobless people are often choosing NO. Try an 1-way ANOVA test. If the p < delta (i.e. delta=0.05), try a post-hoc test (i.e. Tukey's HSD) to do a pairwise comparison.
Like I said earlier, try a multiple comparison test (1-way ANOVA) first, if there is a statistically significant difference, you can try a pairwise comparison test (post-hoc test).
Maybe try a clustering algorithm? Be careful, because the marginal sums (by rows or columns) are not equal. Maybe create a similarity matrix by profession? To me, it seems that Employees and Businessmen are in one group (very similar), while Workers and Jobless are each in their own group. If you turn those frequencies into proportions, then you might just have 2 groups; one for employees + workers + businessmen, and one for jobless.
Use contingency table analysis to see if the responses (yes/no/don't know) are associated with profession.


Class Imbalance Problem I'm making a multiclassifier model with 5 classes.
(it is not important in my question whether it has 2 classes or 5 classes, though).

class distribution is very imbalanced.

So, I did resampling for oversampling to match each number of classes.

MY question is.
Is there any statistical error or problem
if I resample classes for oversampling(bootstrap resampling or SMOTE)?

can anyone explain for this approach?

Thanks in advance.

This depends on the data in your minority classes.

The data in each class can be considered as a sample of observations from some population. This sample may or may not represent well the whole population of instances from that class.

If the sample represents the population well, then oversampling will introduce only a small error. However, if the sample does not represent the population well, then oversampling will produce data that have statistical properties different from those of the population.

All estimates (like confidence intervals, prediction intervals) are calculated from the statistical properties of the sample (mean, variance, etc), the exact calculations being different for different distributions and learning algorithms. If statistical properties of your oversampled data are different from the statistical properties of their populations, you will get wrong estimates of the confidence and prediction intervals for your model.

I will illustrate this with an example.

Let's assume that you have 2-dimensional data (two features in each observaton) that belong to 2 classes. Data in each class are normally distributed with the standard deviation for each feature = 1. The population mean of the class 1 is (-2, 0). The population mean of the class 2 is (1, 0).

I illustrate a large population, taking 500 points for each class. The classes can be separated by logistic regression line as follows:

The regression line is almost exactly between the population means and is almost vertical because both ordinates of the population means are zero. If I take a couple of thousand points then it will be vertical.

The code for this picture:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LogisticRegression

n_points1 = 500
n_points2 = 500
mu1 = np.array([-2,0])
mu2 = np.array([1,0])

sig = 1

cl1 = sig * np.random.randn(n_points1,2) + mu1
cl2 = sig * np.random.randn(n_points2,2) + mu2

print('Red points:')
print('mu_x = %.3f, mu_y = %.3f, std_x = %.3f, std_y = %.3f'
    % (np.mean(cl2[:,0]), np.mean(cl2[:,1]), np.std(cl2[:,0]), np.std(cl2[:,1])) )

y1 = np.zeros((cl1.shape[0],1))
y2 = np.ones((cl2.shape[0],1))

X = np.vstack((cl1,cl2))
y = np.vstack((y1,y2))

logreg = LogisticRegression()

logreg.fit(X, y.ravel())
w = logreg.coef_[0]
a = -w[0] / w[1]

xx = np.linspace(-4,4)
yy = a * xx - (logreg.intercept_[0]) / w[1]

plt.scatter(cl1[:,0], cl1[:,1], s = 50, marker='.', color='green')
plt.scatter(cl2[:,0], cl2[:,1], s = 50, marker='.', color='red')
plt.plot(xx, yy, 'b-')
plt.xlim((-5, 4))
plt.ylim((-5, 5))
plt.grid()
plt.show()

Now, let's assume that the red class is under-represented (a minority class). I take 500 points for the green class, and 10 points for the red class. Then I oversample the red data by two methods. One is duplicating them 50 times, like bootstrap resampling (I color them red), and another is something like SMOTE (they are magenta). This is not exactly SMOTE. I simply added data that are in the middle between red data points. I was too lazy to calculate nearest neighbors for each observation in this simple example but it illustrates SMOTE nevertheless because in SMOTE, synthetic examples are generated inside the convex hull of existing minority examples which reduces the variance of the data.

This is what I get:

The blue line is for bootstrap, and the black line is for SMOTE. The sample means and standard deviations for the minority class are as follows:

Red points:
mu_x = 1.600, mu_y = -0.182, std_x = 0.719, std_y = 0.753

Magenta points:
mu_x = 1.625, mu_y = -0.174, std_x = 0.513, std_y = 0.550

The x-component of the sample mean is overestimated (it should be = 1) and standard deviations are underestimated (they should be = 1). As a result, the line separating the classes is different from the true line on the previous picture, so a lot of new data will be classified incorrectly.

Let's take a couple of more samples randomly from the same population. Again, the size of the minority class is 10, and I resample them using the same two methods.

Red points:
mu_x = 0.944, mu_y = 0.289, std_x = 0.943, std_y = 0.867

Magenta points:
mu_x = 0.974, mu_y = 0.298, std_x = 0.700, std_y = 0.617

This time x-component of the sample mean is OK, its y-component is a little overestimated, and the standard deviation is underestimated. The separating line is again incorrect.

Red points:
mu_x = 0.922, mu_y = -0.236, std_x = 1.066, std_y = 1.097

Magenta points:
mu_x = 0.924, mu_y = -0.244, std_x = 0.757, std_y = 0.749

In the last picture we are lucky to get a sample that has almost the same mean and standard deviation as the population. Therefore, the separating line is very close to the line in the first picture where the data were balanced. Notice that the standard deviation for SMOTE is always smaller because new data are added between the existing data, not outside of them.

You might consider undersampling instead of oversampling. Check this link.

The code for the last 3 pictures:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LogisticRegression

n_points1 = 500
n_points2 = 10
mu1 = np.array([-2,0])
mu2 = np.array([1,0])

sig = 1

cl1 = sig * np.random.randn(n_points1,2) + mu1
cl2 = sig * np.random.randn(n_points2,2) + mu2

cl2 = np.tile(cl2,(n_points1//n_points2,1)) # oversampling

# A kind of SMOTE but not quite
cl3 = np.zeros_like(cl1)
for k in range(len(cl3)):
    i = np.random.randint(low=0, high=n_points2, size=2)
    cl3[k,:] = (cl2[i[0]]+cl2[i[1]])/2

print('Red points:')
print('mu_x = %.3f, mu_y = %.3f, std_x = %.3f, std_y = %.3f'
    % (np.mean(cl2[:,0]), np.mean(cl2[:,1]), np.std(cl2[:,0]), np.std(cl2[:,1])) )

print('Magenta points:')
print('mu_x = %.3f, mu_y = %.3f, std_x = %.3f, std_y = %.3f'
    % (np.mean(cl3[:,0]), np.mean(cl3[:,1]), np.std(cl3[:,0]), np.std(cl3[:,1])) )

y1 = np.zeros((cl1.shape[0],1))
y2 = np.ones((cl2.shape[0],1))

X = np.vstack((cl1,cl2))
X3 = np.vstack((cl1,cl3))

y = np.vstack((y1,y2))

logreg = LogisticRegression()

logreg.fit(X, y.ravel())
w = logreg.coef_[0]
a = -w[0] / w[1]

xx = np.linspace(-4,4)
yy = a * xx - (logreg.intercept_[0]) / w[1]

plt.scatter(cl1[:,0], cl1[:,1], s = 50, marker='.', color='green')
plt.scatter(cl2[:,0], cl2[:,1], s = 50, marker='.', color='red')
plt.plot(xx, yy, 'b-')

logreg.fit(X3, y.ravel())
w = logreg.coef_[0]
a = -w[0] / w[1]
yy = a * xx - (logreg.intercept_[0]) / w[1]

plt.scatter(cl3[:,0], cl3[:,1], s = 10, marker='.', color='magenta')
plt.plot(xx, yy, 'k-')
plt.xlim((-5, 4))
plt.ylim((-5, 5))
plt.grid()
plt.show()


How to Visualise a CNN model using Python I am new to Deep Learning and have been trying to show a plot of the CNN architecture using Python in Google Colab. Besides importing the necessary libraries, I have noticed from other resource that normally, we would declare a model {model = sequential()}, and then model.add (Conv2d or MaxPool or Activation etc.) and for visualising, print(model.summary()).

However, I have developed the CNN architecture using the below code and struggling with generating a model print. Could anybody tell what I am missing here. Thanks

input = Input(shape=X.shape[1:])                                  # 154x154x3
x = Conv2D(12, (3, 3), padding='same', activation='relu')(input)  # 154x154x12
x = Conv2D(12, (2, 2), strides=(2, 2), activation='relu')(x)      # 77x77x12
x = Conv2D(16, (3, 3), padding='same', activation='relu')(x)      # 77x77x16
x = Conv2D(16, (2, 2), strides=(2, 2), activation='relu')(x)      # 38x38x16
x = Conv2D(24, (3, 3), padding='same', activation='relu')(x)      # 38x38x24
x = Conv2D(24, (2, 2), strides=(2, 2), activation='relu')(x)      # 19x19x24
x = Conv2D(32, (3, 3), padding='same', activation='relu')(x)      # 19x19x32
x = Conv2D(32, (2, 2), strides=(2, 2), activation='relu')(x)      # 9x9x32
x = Conv2D(48, (3, 3), padding='same', activation='relu')(x)      # 9x9x48
x = Conv2D(48, (2, 2), strides=(2, 2), activation='relu')(x)      # 4x4x48
x = Dropout(0.5)(x)                                               # 4x4x48
x = Conv2D(n_classes, (1, 1))(x)                                  # 4x4x62
x = GlobalAvgPool2D()(x)                                          # 62
output = Activation('softmax')(x)                                 # 62

Add this two lines below of your code.

from keras.models import Model
model = Model(inputs=input, outputs=output)
print(model.summery)


What features from sound waves to use for an AI song composer? I am planning on making an AI song composer that would take in a bunch of songs of one instrument, extract musical notes (like ABCDEFG) and certain features from the sound wave, preform machine learning (most likely through recurrent neural networks), and output a sequence of ABCDEFG notes (aka generate its own songs / music).

I think that this would be an unsupervised learning problem, but I am not really sure.

I figured that I would use recurrent neural networks, but I have a few questions on how to approach this:
- What features from the sound wave I should extract so that the output music is melodious?
- Is it possible, with recurrent neural networks, to output a vector of sequenced musical notes (ABCDEF)?
- Any smart way I can feed in the features of the soundwaves as well as sequence of musical notes?

First off, ignore the haters.  I started working on ML in Music a long time ago and got several degrees using that work. When I started I was asking people the same kind of questions you are. It is a fascinating field and there is always room for someone new. We all have to start somewhere.

The areas of study you are inquiring about are Music Information Retrieval (Wiki Link) and Computer Music (Wiki Link) .  You have made a good choice in narrowing your problem to a single instrument (monophonic music) as polyphonic music increases the difficulty greatly.

You're trying to solve two problems really:

1) Automatic Transcription of Monophonic Music (More Readings) which is the problem of extracting the notes from a single instrument musical piece.

2) Algorithmic Composition (More Readings) which is the problem of generating new music using a corpus of transcribed music.

To answer your questions directly:

  I think that this would be an unsupervised learning problem, but I am
  not really sure.

Since there are two learning problems here there are two answers. For the Automatic Transcription you will probably want to follow a supervised learning approach, where your classification are the notes you are trying to extract.  For the Algorithmic Composition problem it can actually go either way. Some reading in both areas will clear this up a lot.

  What features from the sound wave I should extract so that the output music is melodious?

There are a lot of features used commonly in MIR. @abhnj listed MFCC's in his answer but there are a lot more.  Feature analysis in MIR takes place in several domains and there are features for each.  Some Domains are:

The Frequency Domain (these are the values we hear played through a speaker)
The Spectral Domain (This domain is calculated via the Fourier function (Read about the Fast Fourier Transform) and can be transformed using several functions (Magnitude, Power, Log Magnitude, Log Power)
The Peak Domain (A domain of amplitude and spectral peaks over the spectral domain)
The Harmonic Domain

One of the first problems you will face is how to segment or "cut up" your music signal so that you can extract features.  This is the problem of Segmentation (Some Readings) which is complex in itself.  Once you have cut your sound source up you can apply various functions to your segments before extracting features from them. Some of these functions (called window functions) are the: Rectangular, Hamming, Hann, Bartlett, Triangular, Bartlett_hann, Blackman, and Blackman_harris.

Once you have your segments cut from your domain you can then extract features to represent those segments.  Some of these will depend on the domain you selected.  A few example of features are: Your normal statistical features (Mean, Variance, Skewness, etc.), ZCR, RMS, Spectral Centroid, Spectral Irregularity, Spectral Flatness, Spectral Tonality, Spectral Crest, Spectral Slope, Spectral Rolloff, Spectral Loudness, Spectral Pitch, Harmonic Odd Even Ratio, MFCC's and Bark Scale. There are many more but these are some good basics.

  Is it possible, with recurrent neural networks, to output a vector of sequenced musical notes (ABCDEF)?

Yes it is. There have been several works to do this already. (Here are several readings)

  Any smart way I can feed in the features of the soundwaves as well as sequence of musical notes?

The standard method is to use the explanation I made above (Domain, Segment, Feature Extract) etc.  To save yourself some work I highly recommend starting with a MIR framework such as MARSYAS (Marsyas).  They will provide you with all the basics of feature extraction.  There are many frameworks so just find one that uses a language you are comfortable in.


Ridge regression model creation using grid-search and cross validation I created python code for ridge regression.For that I used cross validation and grid-search technique in together. i got output result. I want check whether my regression model building steps correct or not? can some one explain it?

from sklearn.linear_model import Ridge
ridge_reg = Ridge()
from sklearn.model_selection import GridSearchCV
params_Ridge = {'alpha': [1,0.1,0.01,0.001,0.0001,0] , "fit_intercept": [True, False], "solver": ['svd', 'cholesky', 'lsqr', 'sparse_cg', 'sag', 'saga']}
Ridge_GS = GridSearchCV(ridge_reg, param_grid=params_Ridge, n_jobs=-1)
Ridge_GS.fit(x_train,y_train)
Ridge_GS.best_params_

output - {'alpha': 1, 'fit_intercept': True, 'solver': 'cholesky'}

Ridgeregression = Ridge(random_state=3, **Ridge_GS.best_params_)
from sklearn.model_selection import cross_val_score
all_accuracies = cross_val_score(estimator=Ridgeregression, X=x_train, y=y_train, cv=5)
all_accuracies

output - array([0.93335508, 0.8984485 , 0.91529146, 0.89309012, 0.90829416])

print(all_accuracies.mean())

output - 0.909695864130532

Ridgeregression.fit(x_train,y_train)
Ridgeregression.score(x_test,y_test)

output - 0.9113458623386644

Is 0.9113458623386644 my ridge regression accuracy(R squred) ? if it is, then what is meaning of 0.909695864130532 value.

Is 0.9113458623386644 my ridge regression accuracy(R squred) ? if it is, then what is meaning of 0.909695864130532 value.

These are both R^2 values.
The first score is the cross-validation score on the training set, and the second is your test set score.   The first is perhaps a little biased, since those models are built using hyperparameters selected while using that dataset (but a different cv-split I think).  The second score should be unbiased, and should probably be your reported test score.  (The second score is a bit better than what should be a optimistically-biased score; perhaps just randomly, perhaps because of the train/test split, and perhaps because that model has seen more data.)

I want check whether my regression model building steps correct or not? can some one explain it?

Things look OK to me.
You could save yourself some code and training time; by default GridSearchCV refits a model on the entire training set using the identified hyperparameters, so you don't need to fit in the last code block.  It also has the cv_results_ and best_score_ attributes to provide you with cross-validation scores, but since you've used the them in selecting the optimal hyperparameters, the best score is no longer an unbiased estimator for future performance.
Note too that there's a builtin for tuning the regularization parameter, but it uses a different CV approach and has RMSE as default scorer.


Regarding join in pyspark Given a rdd in pyspark, $A=[(0, 1), (1, 2), (1, 3), (3, 0)]$

I am trying to find self join of A such that the second element of previous pair is equal to first element of next pair i.e.

$result=[(0, 2), (0, 3), (1, 0)]$

I tried taking cartesian but it is giving other possibilities.
Also I tried with reverse of the above and then joining but it is also giving many use cases.
Please let me know the correct approach to solve the above.

I got the answer by taking reverse of the above list and then using inner join of rdd to get all corresponding transitive pairs. After removing duplicates i.e. (b,a) and same edges (a,a) or (b,b) got the resulting rdd.

Thanks for the help.


How to store strings in CSV with new line characters? My question is: what are ways I can store strings in a CSV that contain newline characters (i.e. \n), where each data point is in one line?

Sample data

This is a sample of the data I have:

data = [
    ['some text in one line', 1],
    ['text with\nnew line character', 0],
    ['another new\nline character', 1]
]

Target CSV

I want a CSV file, where the first line is "text,category" and every subsequent line is an entry from data.

What I tried

Using the csv package from Python.

import csv
field_names = ['text', 'category']

# Writing
with open('file.csv', 'w+', encoding='utf-8') as file:
    csvwriter = csv.DictWriter(file, field_names)
    csvwriter.writeheader()
    for d in data:
        csvwriter.writerow({'text': d[0], 'category':d[1]})

# Reading
with open('file.csv', 'r', encoding='utf-8') as file:
    csvreader = csv.DictReader(file, field_names)
    data = []
    for line in csvreader:
        data.append([line['text'], line['category']])

I can read and write, but the output file.csv is the following:

  text,category

  some text in one line,1

  "text with

  new line character",0

  "another new

  line character",1

So not one line per data point.

I assume that you want to keep the newlines in the strings for some reason after you have loaded the csv files from disk. Also that this is done again in Python. My solution will require Python 3, although the principle could be applied to Python 2.

The main trick

This is to replace the \n characters before writing with a weird character that otherwise wouldn't be included, then to swap that weird character back for \n after reading the file back from disk.

For my weird character, I will use the Icelandic thorn: Þ, but you can choose anythin that should otherwise not appear in your text variables.
Its name, as defined in the standardised Unicode specification is: LATIN SMALL LETTER THORN. You can use it in Python 3 a couple of ways:

weird_name = '\N{LATIN SMALL LETTER THORN}'
weird_char = '\xfe'             # hex representation
weird_name == weird_char        # True

That \N is pretty cool (and works in python 3.6 inside formatted strings too)... it basically allows you to pass the Name of a character, as per Unicode's specification.

Replacing \n

Now we use this weird character to replace '\n'. Here are the two ways that pop into my mind for achieving this:

using a list comprehension on your list of lists: data:

new_data = [[sample[0].replace('\n', weird_char) + weird_char, sample[1]]
             for sample in data]

putting the data into a dataframe, and using replace on the whole text column in one go

df1 = pd.DataFrame(data, columns=['text', 'category'])
df1.text = df.text.str.replace('\n', weird_char)

The resulting dataframe looks like this, with newlines replaced:

               text              category
0         some text in one line      1
1  text withþnew line character      0
2    another newþline character      1

Writing the results to disk

Now we write either of those identical dataframes to disk. I set index=False as you said you don't want row numbers to be in the CSV:

FILE = '~/path/to/test_file.csv'
df.to_csv(FILE, index=False)

What does it look like on disk?

  text,category

  some text in one line,1

  text withþnew line character,0

  another newþline character,1

Getting the original data back from disk

Read the data back from file:

new_df = pd.read_csv(FILE)

And we can replace the Þ characters back to \n:

new_df.text = new_df.text.str.replace(weird_char, '\n')

And the final DataFrame:

new_df
               text               category
0          some text in one line      1
1  text with\nnew line character      0
2    another new\nline character      1

If you want things back into your list of lists, then you can do this:

original_lists = [[text, category] for index, text, category in old_df_again.itertuples()]

Which looks like this:

[['some text in one line', 1],
 ['text with\nnew line character', 0],
 ['another new\nline character', 1]]


How to alter word2vec wikipedia model for n-grams? I have a very little data, so my word2vec model does not perform well. My intention is to identify words similar to technical terms such as 'support vector machine', 'machine learning', 'artificial intelligence' etc.

I am interested in knowing if I can use the Google's wikipedia model for this. But according to my model most of the words I will be dealing with are n-grams. Hence, how can I utilise this Google's wikipedia model that is based on unigrams to achieve my task?

I am happy to provide more examples if needed :)

To clarify, an n-gram usually refers to a sequence of characters, i.e. the word "clear" is comprised of the trigrams {cle, lea, ear}. I think the term you are looking for is "multi-word phrases".

Embedding collections of words is referred to a couple different ways, including "sent2vec", "doc2vec" or "thought vectors". These terms generally refer to embedding a "complete" set of words,  either a sentence, paragraph, collection of paragraphs, or a document. A common --
if somewhat inelegant -- approach to using a pre-trained word2vec model to embed multiple words is to embed each word separately and then take the average as the embedding for their combination.

I think you'll find this article relevant: Representation learning for very short texts using weighted word embedding aggregation


BERT classifier with Ktrain API is unable to predict new data I have trained a classifier for sentiment analysis using BERT architecture.
I am able to train the classifier and I am getting a validation accuracy of 87%. But whenever I feed in test data, or some simple sentence like "What an amazing movie", "I would love that book", etc, the model predicts the class for one and says list index out of range for other.
I tried to find if there is a bug in my code as well.

(x_train, y_train), (x_test, y_test), preproc = text.texts_from_csv(TWITTER,
                                                                       preprocess_mode='bert',
                                                                       text_column = 'text', label_columns = ['target'])

model = text.text_classifier('bert', (x_train, y_train), preproc=preproc)
learner = ktrain.get_learner(model,train_data=(x_train, y_train), val_data=(x_test, y_test), batch_size=6)

learner.fit_onecycle(2e-5, 1)

predictor = ktrain.get_predictor(learner.model,preproc)

predictor.predict(['I am very happy to meet you!'])

Error that I am getting

IndexError                                Traceback (most recent call last)
<ipython-input-28-82b8da552189> in <module>()
----> 1 predictor.predict(['I am very happy to meet you!'])

1 frames
/usr/local/lib/python3.6/dist-packages/ktrain/text/predictor.py in <listcomp>(.0)
     56                 preds = np.squeeze(preds)
     57                 if len(preds.shape) == 0: preds = np.expand_dims(preds, -1)
---> 58         result =  preds if return_proba or multilabel or not self.c else [self.c[np.argmax(pred)] for pred in preds]
     59         if multilabel and not return_proba:
     60             result =  [list(zip(self.c, r)) for r in result]

IndexError: list index out of range

My problem is that if I have any bugs in my code, then I should get this error every time I execute the predict method. Say I have 100 new test points, I am getting this error only for 30 to 40 test points and the remaining are classified properly. I tested this theory by feeding in tweets one tweet at a time. But I do not understand why is this happening.

try without list operator []:

predictor.predict('I am very happy to meet you!')


Visualising K-Means clusters for 3D data in R I have an excel file that contains 485k rows x 3 columns of integer values. Sample data:
cluster.x    cluster.y    cluster.z
1            3            3
3            1            4
4            2            5
...

I did a kmeans clustering with 6 clusters, but I am figuring out what is the best way to display the result. I plotted it using plot3d() as shown below, but it is hard to differentiate different clusters since the values are all integers.
Does anyone have any suggestions? If possible, I also want it to be able to visualise how many data points there are in each cluster (e.g a bubble plot where the size of the bubble depicts the number of data points).

Is it necessary to plot all points? If not, you can use plot3d to display 6 bubbles centered at each clusters center. You can use mean within distance for each cluster to set the radius of each cluster (you will have to normalize this) and a color range to display the number of points in each cluster.


Data prepration for logistic regression : Value either "not available" or a "year" I have some data of houses that have been renovated.

In my data there is one column (among others) that captures this information.

It is either "-1" if there has not been yet any renovation, or the information is the year of renovation like "1995" or "2008".

I would like to apply logistic regression.

However, I do not know how to treat this value.

IMHO it looks like a missing value although it is not a missing information.

So, does anybody have an idea how to put these (unordered) values into relation to the ordered years?

On alternative I could think of is binning the information. Like 1990-1995, 1996-2000,...2016-2019.

Any suggestions are highly appreciated.

First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.

Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:

bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $\rightarrow$ (1, 0, 0)
[2000, 2010] $\rightarrow$ (0, 1, 0)
[2010, 2020] $\rightarrow$ (0, 0, 1)

This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.

I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:

renovation     (has renovation, renovation age)
-1             (0, -1)
2010 in 2019   (1, 9)

Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.


Incremental learning with new samples I have a very large dataset with timestamp data. Till now, I loaded the whole dataset in order to train some models using python (statsmodels.api,  statsmodels.formula.api, keras.models.Sequential).
Now, I would like to train models with all historical data just one time, and then update the model just with new data (each day I have new samples).
Is it possible and how with these python libraries?
Thanks!

This is what Machine Learning models are used for..(to predict what they think will happen in the near time based on the data they have been fed into....)

A simple answer is this

The first couple days of data are the most important , you need to gauge and have a watch over your model..

As with any incremental learning, you can learn more recent stuff but also underfit the past significantly more than without learning the new stuff.
Therefore, it's why there's a monitoring to perform on models which use incremental learning (especially on production systems).

Provided that other things remains similar with time..(same Preprocessing, same scaling etc..)

But let's say if you have tons of data everyday, then nearly at the month end, the model can be completely trained on the last months data or else checkpoints needs to be kept


Difference between segmentation and effect of explanatory variables In the context of discrete choice models, what difference does it make in segmenting my sample based on a particular "criteria" and study the effects of explanatory variables on each segment VERSUS Just adding the "criteria" as another explanatory variable in the unsegmented population

Say, I want to build a commuter mode choice model using MNL, what difference does it make when I split my population via gender and study the effects of various explanatory variables VERSUS adding gender as another explanatory variable in the entire unsegmented population.

If you choose to build a model where one of the categorical features such as a gender plays a big role in the outcome, then the overall model results will give a large weight to the gender, and other features will have a smaller weight. While your model will be applicable for the unsegmented population, the predictive power might not be very good.

Now, suppose that you wanted to identify if the weights of the other features differed for males vs. females and if the segmented models had better predictive power, then by segmenting the population and creating models for each gender, you could explore changes in the magnitude or direction of the weights of each feature & evaluate the goodness of fit of your models to see if there is an improvement in the segmented population models vs. the overall model.

If performing logistic regression, you could scale the coefficients or Z-values returned in the model parameters for each model and check if the relative magnitude & direction of the overall vs. the segmented models gave you similar or different results. You could also look at the confusion matrix and use various accuracy measures to assess if the segmented models performed better, worse or the same as the overall model. In this way, you could determine whether for the outcome independent gender-based models are more appropriate for your question than a single overall model.

But, this leads to the question of if I have "n" features, should I create multiple models by segmenting the "x" categories in each of the "n" features?  This really depends on the data question being asked. In some cases, it is valuable to run multiple models where each feature is dropped one-by-one to assess feature importance. In some cases, just evaluating the weight & relevance of each feature from the overall model is sufficient. Going beyond, one could look at optimizing models by choosing which of all "n" features should be segmented to yield the model with the best predictive power. This paper from FICO describes one methodology. This article details an approach and comparison of various models.


How can I use Machine learning for inter-relationship between Features? Machine learning is used mostly for prediction and there are numerous algorithms and packages for this.

How can I use machine learning for studying inter-relationships between features? What are major packages and functions for this? Are there any packages for graphics in this area? Can artificial neural networks also be used for this purpose? If so, any particular type is specifically suited for this?

I do not want to limit to any particular language like Python or R.

I use pair plots to study the inter-relationship between features. Pair plot gives the first level information of the features. Seaborn library has pairplot function and even matplotlib has the same.

Another thing you can use heatmap which gives the co-relation between features, by using heat map we can see features that are high co-related and may eliminate one of them. As a word of caution you must good reason or domain knowledge to drop a feature


What types of features are used in a large-scale click-through rate prediction problem? Something that I often see in papers (example) about large-scale learning is that click-through rate (CTR) problems can have up to a billion of features for each example. In this Google paper the authors mention:

  The features used in our system are drawn from a variety of sources,
  including the query, the text of the ad creative, and various
  ad-related metadata.

I can imagine a few thousands of features coming from this type of source, I guess through some form of feature hashing.

My question is: how does one get to a billion features? How do companies translate user behavior into features in order to reach that scale of features?

That really is a nice question, although once you're Facebook or Google etc., you have the opposite problem: how to reduce the number of features from many billions, to let's say, a billion or so.

There really are billions of features out there.

Imagine, that in your feature vector you have billions of possible phrases that the user could type in into search engine. Or, that you have billions of web sites a user could visit. Or millions of locations from which a user could log in to the system. Or billions of mail accounts a user could send mails to or receive mails from.

Or, to swich a bit to social networking site-like problem. Imagine that in your feature vector you have billions of users which a particular user could either know or be in some degree of separation from. You can add billions of links that user could post in his SNS feed, or millions of pages a user could 'like' (or do whatever the SNS allows him to do).

Similar problems may be found in many domains from voice and image recognition, to various branches of biology, chemistry etc. I like your question, because it's a good starting point to dive into the problems of dealing with the abundance of features. Good luck in exploring this area!

UPDATE due to your comment:

Using features other than binary is just one step further in imagining things. You could somehow cluster the searches, and count frequencies of searches for a particular cluster.

In a SNS setting you could build a vector of relations between users defined as degree of separation instead of a mere binary feature of being or not being friends.

Imagine logs that global corporations are holding on millions of their users. There's a whole lot of stuff that can be measured in a more detailed way than binary.

Things become even more complicated once we're considering an online setting. In such a case you do not have time for complicated computations and you're often left with binary features since they are cheaper.

And no, I am not saying, that the problem becomes tractable once it's reduced to a magical number of billion features. I am only saying that a billion of features is something you may end up after a lot of effort in reducing the number of dimensions.


Data Visualisation for Dataframe having 3 columns I have the below data, I am looking for a effective visualization or graphical method (in Python) to understand how the err is distributed with respect to min and max.

Data_X_err.head()
Out[137]:
     min    max   err
0  35435  35933  1.40
1  35155  36382  3.43
2  35305  36042  2.07
3  35216  36225  2.82
4  35196  36259  2.98

Thank you.

use matplotlib:

import matplotlib.pyplot as plt
%matplotlib inline
f, (ax1, ax2) = plt.subplots(1, 2)
ax1.plt(Data_X_err.min, Data_X_err.err)
ax2.plt(Data_X_err.max, Data_X_err.err)

Alternatively you could use seaborn's pairplot

import seaborn as sns
%matplotlib inline
sns.set(style="ticks")

sns.pairplot(df)


how to extract substrings from a dataframe column I have a dataset as below:

 col1

{:fbbbhbhbh{,:50k: ghgjlj45llj,ljhjlhlhj,clause :59:jhjhhjxjhj65@,jjjhjhd :70

I want to add 2 more cols such as col_50 & col 59 as below:

 col2                                        col3
 ghgjlj45llj,ljhjlhlhj,clause                jhjhhjxjhj65@,jjjhjhd

I tried for col 2 like below:

   def find_between( s, first, last ):
      try:
        start = s.index( first ) + len( first )
        end = s.index( last, start )
        return s[start:end]
     except ValueError:
        return ""
  df['col2']=find_between( df1['col1'].str, "50k:", ":" )
  df['col3']=find_between( df1['col1'].str, "59:", ":" )

Such that it extracts all characters between 50k: & untill next ':' appears, however not getting desired result. Can anyone please help?

In your specific case, when you want to find something between two different markers, you can use the .split(marker) method as follows:

In [1]: s = "{:fbbbhbhbh{,:50k: ghgjlj45llj,ljhjlhlhj,clause :59:jhjhhjxjhj65@,j
   ...: jjhjhd :70"

In [2]: s.split("50k:")[1].split("59:")[0]
Out[2]: ' ghgjlj45llj,ljhjlhlhj,clause '

If you want to get rid of the spaces at the end, just throw in another method at the end: strip():

In [3]: s.split("50k:")[1].split("59:")[0].strip()
Out[3]: 'ghgjlj45llj,ljhjlhlhj,clause'

You can add that to a function as you did with your own code, and put the results into a Pandas Dataframe.

def my_parser(s, marker1, marker2):
    """Extract strings between markers"""
    base = s.split(marker1)[1].split(marker2)
    part1 = base[0].strip()
    part2 = base[1].strip()
    return part1, part2

Now use the function to iterate over all thew string you have

results = []    # to collect results of each example
m1 = ...        # put whatever markers you need here
m2 = ...
for line in lines:
    results.append(my_parser(line, m1, m2))

# Create a dataframe from a list of lists (i.e. from_records). Each inner list becomes a row
df = pd.DataFrame.from_records(results)

Edit:

Given you have your strings in a DataFrame already and just want to iterate over them, you could do the following, assuming you have my_col, containing the strings:

for line in df.my_col:
    results.append(my_parser(line, m1, m2))    # results is a list as above

So that is what you said you wanted to extract, but it will maybe not generalise well. For example, if there are multiple of those markers in your sentence, you might get unexpected results, or at least only the first occurrence of what you want to extract (if there can be many in your other text examples).


Performance and architecture of neural network for increased dimensions I posted this question on Cross Validated before I realized that this existed. I think it is better suited here and got no answers over there so I have deleted other post. I have reproduced the question below:

I have been playing around with the neural network toolbox in MATLAB to develop an intuition for how the architectural requirements scale with feature dimension.

I put together a simple example, and the results have surprised me. I am hoping someone can point to either (a) an unrealistic expectation of mine, or (b) a mistake/misuse of the neural network toolbox.

The example is as follows: I have a simple un-normalized one-dimensional Gaussian that I am trying to learn. I do the following:

x = -5:0.2:5;
y = exp(-x.^2/2);

net = feedforwardnet(2);
net = configure(net, x, y);
net = train(net, x, y);
y2 = net(x);

plot(x, y, 'o', x, y2);
legend('Data', 'NN');

This gives me good results. I get the plot below.

Now, I try to extend this to 2 dimensions and this is where I run into trouble. I don't think I'm asking too much. My data is not noisy, or is it sparse. I figure if I double the number of neurons that should be sufficient for an increase in dimensionality. Here's my code:

x1 = -5:0.2:5;
x2 = -5:0.2:5;
[x1g, x2g] = meshgrid(x1, x2);
xv = [x1g(:)'; x2g(:)'];
yv = exp(-dot(xv,xv)/2);

net = feedforwardnet(4);
net = configure(net, xv, yv);
net = train(net, xv, yv);
y2v = net(xv);
plot3(xv(1,:), xv(2,:), yv, xv(1,:), xv(2,:), y2v, 'o');
legend('Data', 'NN');

The plot I get is this:

This is pretty poor. Perhaps I need more neurons? Maybe if I double the number of dimensions, I need to quadruple the number of neurons. I get this for 8 neurons:

Maybe with 8 neurons I have a lot of weights to fit, so let me try training with regularization. I get the plot below with trainbr:

It's only at around 16 neurons that I start getting something I would consider reasonable.

However, there are still oscillations which I don't like. Now I know I'm using it out of the box in a naïve manner, or perhaps I'm expecting too much. But this simple example resembles the real problem I want to tackle. I have the following questions:

Why is it that an increase from 1 to 2 dimensions increases the number of neurons required to get a decent fit considerably?
Even when I go to a larger number of neurons, I get oscillations that are going to be a problem in my real world application. How can I get rid of that?
Most resources on NN that I've read indicate a substantially lower number of neurons. They usually state something like "equal to or less than the number of input variables". Why is that? Is a multidimensional Gaussian a pathological case?
If I need to be more intelligent with how I treat my network for a given number of neurons, what do I need to do? I tried retraining the network to see if it was a local minima issue, but I generally get a similar fit.
Anything else that may be remotely useful to this issue is appreciated!

Some things to take into account:

Try to apply appropriate input space transformations, e.g. convert to polar coordinates.
Despite the fact that a single hidden layer feedforward network can be a universal approximator, there is no guarantees about the number of neurons needed to approximate an arbitrarily complex function. Instead of having a single hidden layer and making it increasingly wide, try stacking more layers. This enables the network to model non linearities in an easier way.
Do not verify the performance of your model based on visual inspection (at least not only), but based on error measurements (e.g. MSE).


How to put multiple features into RNN input vector I am trying to code a recurrent neural network (LSTM) to create music in python and was considering using multiple features instead of just the note pitch as an input into the network. Initially I had just the note pitch so it was fed into the network by one-hot encoding it. The other two features I want to add are the note duration and the offset between the notes. How should the input vector be organised so that all the data is fed through the network?

I have tried combining all of the data into a long vector with all 3 features one-hot encoded and then concatenated but this caused the output to become 'NaN'. Any help would be appreciated.

Link a gist of my code:

https://gist.github.com/treutm/1b1f49e3d0a9de53cd67c136292f329f

A common way to input several features to an LSTM (or any RNN) is, as you did, to concatenate them in a vector. I suspect your NaN are related to a different issue in the code, and I recommend you to debug it and see when and why it happens.

A different way of combining several features is using embeddings for each feature and combine them via concatenation. For example, for the note pitch, you have an embedding for each note, for the note duration, the same, and so on. The input to the RNN would be a concatenation of that.

A third option would be to have an embedding to all possible combinations of the embeddings. That would be an embedding for each triplet of note pitch, note duration and offset. This representation can be also powerful.

My recommendation is that you start with the easiest to implement, and if the result is not satisfactory in terms of model performance, think about the next one.


How to compare supervised learning algorithm and it's technique ensemble learning algorithm? I have to compare Support Vector Machine and Random Forest algorithm , but i'm confused how it can be compared, like support vector machine is supervised learning algorithm and random forest is ensemble learning .
Help me out how i can compare it on which point like - in clasification , in regression .

TL;DR

Since both SVM and Random Forest are supervised algorithms, you can compare the two like you would compare any other two supervised algorithms.

The fact that a Random Forest is an ensemble classifier doesn't really matter as long as you treat all trees in the forest as a single model.

Comparing two supervised algorithms

The simplest way to compare supervised algorithms is with a train/test split:

Split all your data into two sets, namely a training and a testing set (a common ratio is 0.8/0.2).
Train both models independently with the data from the training set.
Use your models to predict the data from the testing set.
Give a score to the predictions by comparing what the model predicted vs the true value from the testing set. If you have a classification problem, you could use the F1 score. If you have a regression problem, you could use the R-square score.
Pick the model with the best score.

Other ways of comparing two supervised algorithms

Instead of a train/test split, you could look into cross-validation.
Instead of a random train/test split, you could look into stratified or time splits.
Instead of comparing two different algorithms, you could compare the same algorithm against itself but with different hyper-parameters (i.e. Hyper-parameter optimization).
Instead of F1 score or R-squared you could use a metric that best fits your business case.


Closest Supporting Hyperplane to Points How do I find the hyperplane (line) that touches a cloud of 2D points from one side? I used $w^Tx+b=0$ based line definition and implemented a gradient-descent based routine that would start the support line from a certain side of points, away from them. The loss function is $w^Tx_i+b$ for all points, then I take log sum. The use of hyperplanes is similar as in SVM but in this case there are no -1 or +1 labels, there are only points I am trying to approach as close as possible. I am hoping to do this from 4 sides of a point cloud to get 4 approximate / defining lines of this point cloud.

My solution was fine for some carefully selected step sizes, and iteration numbers, but the procedure can overshoot, trying to find another minima.

I am curious if there is a better way of defining this loss function. My implementation based on automatic differentiation package autograd, seen below:

import pandas as pd

def plot_sep(w):
    Q = np.array([[0, -1],[1, 0]])
    x = np.linspace(-20,20,3000)
    w2 = np.dot(Q,w[:2])
    m = w2[1]/w2[0]
    y = m*x + (-w[2]/w[1])
    plt.plot(x,y,'.')

df = pd.read_csv('in.csv')
df['1'] = 1.
df2 = np.array(df)

#w1_init = np.array([0.,1.,-6.]);eta  = 1e-3;iters = 30 # good
w1_init = np.array([0.,1.,-30.]);eta  = 1e-2;iters = 40 # bad

plt.plot(df2[:,0],df2[:,1],'.')
plot_sep(w1_init)
plt.xlim(0,15);plt.ylim(-10,40)
plt.savefig('out1.png')

import autograd.numpy as np
from autograd import elementwise_grad
from autograd import grad

def compute_loss(w1):
    tmp = np.dot(df2,w1)
    tmp2 = np.log(np.dot(tmp,tmp))
    return tmp2

gradient = grad(compute_loss)

w1 = np.copy(w1_init)

for i in range(iters):
    loss = compute_loss(w1)
    print "iteration %d: loss %f" % (i, loss)
    dw1 = gradient(w1)
    w1 += -eta*dw1
print "iteration %d: loss %f" % (i, loss)
print w1

plt.figure()
plt.plot(df2[:,0],df2[:,1],'.')
plot_sep(w1)
plt.xlim(0,15);plt.ylim(-10,40)
plt.savefig('out2.png')

Data

x,y
6.85483870968,11.875
8.06451612903,12.3958333333
7.37903225806,12.34375
8.18548387097,12.34375
8.83064516129,12.6041666667
9.43548387097,12.96875
10.0,13.0729166667
10.5241935484,13.1770833333
11.0483870968,13.2291666667
6.97580645161,10.9895833333
6.97580645161,10.4166666667
8.46774193548,10.15625
7.98387096774,10.15625
9.1935483871,10.15625
9.79838709677,10.15625
10.6048387097,10.0
11.1290322581,10.1041666667
11.1290322581,10.5208333333
10.9274193548,11.0416666667
10.9274193548,11.40625
10.9274193548,11.7708333333
10.8870967742,12.4479166667
10.0,12.7083333333
9.07258064516,11.9270833333
8.75,11.9270833333
7.86290322581,11.8229166667
7.33870967742,11.09375
7.9435483871,11.3541666667
9.15322580645,11.5104166667
9.39516129032,11.5104166667
8.50806451613,10.8854166667
9.47580645161,10.78125
9.91935483871,10.78125
10.1612903226,10.8333333333
10.1612903226,11.9270833333
9.91935483871,12.03125
9.83870967742,12.03125
9.63709677419,11.9270833333
10.564516129,11.3020833333
10.564516129,10.6770833333
9.11290322581,10.5208333333
8.02419354839,10.625

I pondered this question a little more, there might not be an optimal loss function.. But here is an idea which could utilize SVMs hyperplane seperation ability.

I take the point cloud, if I need an approximating line from the bottom, copy points below with user defined gap, and I flip the points, as in mirror image along the x-axis. I give both point groups opposing labels, then feed the whole thing to SVM. Ideally I should get something like this:

The top line is the line I am looking for.

Code

import pandas as pd
points1 = np.array(pd.read_csv('quadri.csv'))
plt.plot(points1[:,0], points1[:,1], 'o')
points2 = np.copy(points1)
points2[:,1] -= 7.
xmin = np.min(points2[:,0]); xmax = np.max(points2[:,0])
points2[:,0] = (xmax-points2[:,0])+xmin
points = np.vstack((points1,points2))
plt.plot(points[:,0], points[:,1], 'o')
plt.xlim(4,14); plt.ylim(0,15)


BERT has a non deterministic behaviour I am using the BERT implementation in https://github.com/google-research/bert for feature extracting and I have noticed a weird behaviour which I was not expecting: if I execute the program twice on the same text, I get different results. I need to know if this is normal and why this happens in order to treat this fact in one or another way. Why is the reason for this? Aren't neural networks deterministic algorithms?

I had the exact same problem using the pytorch implementation, until I realised I didn't set the model in eval mode. Hence, dropout was still activated. I guess this is also the source of your undeterministic behavior.

Fix using pytorch:

model.eval()

Using tensorflow with an Estimator, ensure you call estimator.evaluate or estimator.predict during testing. If you exported the model in some ways, then you should check the tensorflow documentation.


Identifying software and version range in a sentence I have sentences similar to the following format

  This vulnerability happened in Firefox 1.x before 1.8, Safari
  2.x before 2.8.

Given the above sentence, I want to extract a dictionary

{Firefox: 1.0-1.8, Safari: 2.0-2.8}

Problem is how should I identify the version range with the software they belong to, using NLP techniques?

If all your sentences are in the following format(atleast that what I assume), you can parse it with regex instead.

`([A-Z][a-z]+)\s+(\d+\.\d+)(?:\s+before\s+)(\d+\.\d+)`

Using this pattern with the given sentence finds all the software and the versions in it. re.findall(pattern, sentence) will give you a list of tuples like this
[('Firefox', '2.4', '2.5'), ('Safari', '6.0', '6.8')]


How to count grouped occurrences? My dataset has 2 columns:

1) Gender( that has 1 or 2)

2) Rating1(That has 3 ratings (1,2,3))

rating1: 1,2,3,3,2

gender: 2,1,2,2,1

I want output as:

I have tried this:

nutri.groupby(['Rating1','Gender']).nunique()

but I get an output which looks wrong.

You can do that by

>>> import pandas as pd
>>> m = pd.DataFrame({'gender': [1, 2, 2, 1, 1, 2, 1], 'rating': [3, 4, 2, 1, 3, 1, 5]})
>>> m.groupby(['rating','gender']).size().to_frame('count').reset_index()
   rating  gender  count
0       1       1      1
1       1       2      1
2       2       2      1
3       3       1      2
4       4       2      1
5       5       1      1

Hope this is what you want to pull.

Edit:
As mentioned, I did not account for zero values.

You need to do one more step additional to get what you want. Finding the combinations that are missing and then joining it. The one liner solution would be

>>> from itertools import product
>>> m.groupby(['rating', 'gender']).size().to_frame('count').reset_index().merge(
        pd.DataFrame(list(set([i for i in product(*[m.gender, m.rating])])), columns=['gender', 'rating']),
        on=['rating', 'gender'],
        how='right').fillna(value=0)
   rating  gender  count
0       1       1    1.0
1       1       2    1.0
2       2       2    1.0
3       3       1    2.0
4       4       2    1.0
5       5       1    1.0
6       2       1    0.0
7       4       1    0.0
8       3       2    0.0
9       5       2    0.0

Explanation

Get the original grouped counts as mentioned before the first edit but this time you need to join with the combinations that are missing to get the zero counts. Use itertools.product to get all combinations of gender and rating and right join it with original grouped frame on rating and gender to get merged DataFrame which has numpy.na values if no count is present and then use fillna method to fill it with zero. The only loophole is if a rating say 4 is not there in the original data, there is no combination that follows it.

Hope this helps.


Problem building a feature vector I am trying build a classifier for malware analysis for which basing in the instructions of an assembly code, such as push, mov,... I want to predict the compiler, and in a second time the optimization op, and I am having some troubles. My code is the following:

#pakages
import numpy as np
import pandas as pd
import json as j
import re
import nltk
from nltk.tokenize import word_tokenize

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import *
from sklearn.metrics import confusion_matrix, classification_report
from sklearn import svm

#for visualizing data
import matplotlib.pyplot as plt
import seaborn as sns; sns.set(font_scale=1.2)

%matplotlib inline

json_data = None;
with open('training_dataset.jsonl') as data_file:
    lines = data_file.readlines()
    joined_lines = "[" + ",".join(lines)+"]"

    json_data = j.loads(joined_lines)

data = pd.DataFrame(json_data)
data.head()

which gives:

now, when I look at:

len(data['instructions'])

I have as output :  30000

but if I do the following:

for value in data['instructions'].iteritems():
    myList = list(value[1]);

myList

opcodes = [instruction.split()[0] for instruction in myList]

len(opcodes)

I get as output : 151

Why don't I have an output 30000? I don't understand why I have less elements. I want to use the opcodes to build a feature vector, but don't understand why the  number of elements become so low.

Can somebody help me? Thank's in advance.

[EDIT] if it can be useful, if I do:

data['instructions']

I get as output:

Analyse this code

for value in data['instructions'].iteritems(): # Here we iter by line int the instruction colunm
    myList = list(value[1]); # Here you get 1 of the values and assing to your list

myList

opcodes = [instruction.split()[0] for instruction in myList]

len(opcodes)

your "myList" variable only holds 1 of the lines (the last one) of your data, which is probably the number of instructions in that line of your dataframe

While the 30k (I don't know how panda works here) is probably the number of lines in that column (or very unlikely the sum of all instructions in every line)


unable to parse XML in pig I have a XML file has this structure (not exactly a tree though)

<posthistory>
<row Id="1" PostHistoryTypeId="2" PostId="1"
RevisionGUID="689cb04a-8d2a-4fcb-b125-bce8b7012b88"
CreationDate="2015-01-27T20:09:32.720" UserId="4" Text="I just got a
pound of microroasted, local coffee and am curious what the optimal
way to store it is (what temperature, humidity, etc)" />

I am using apache pig to extract just the "Text" part using this code

grunt> A = load 'hdfs:///parsingdemo/PostHistory.xml' using
org.apache.pig.piggybank.storage.XMLLoader('posthistory') as(x:chararray);

grunt> result = foreach A generate XPath(x, 'posthistory/Text');

this returns "()" (null)

Upon examining the XML file, I learnt that my XML file should be in this format:

<root>
  <child>
    <subchild>.....</subchild>
  </child>
</root>

But my XML data file (stackoverflow data dump actually) is not in this format. Is there a way the tree structure can be imposed? what is wrong with my pig query?

This XPath will look for a tag called <Text> inside a tag called <posthistory>:

XPath(x, 'posthistory/Text');

You want to find the Text attribute of the row tag in posthistory tags.

An XPath something like this will do that: /posthistory/row/@Text

See example here: http://www.xpathtester.com/xpath/bac9874ec344f9d8ebcfb250633aaf65 and click "Test" to see results set.

Learn up on XPath notation for more.


Theoretical Question: Data.table vs Data.frame with Big Data I know that I can read in a very large csv file much faster with fread using the data.table library than with read.csv that reads a file in as a data.frame. However, dplyr can only perform operations on data.frame.

My questions are:

Why was dplyr built to work with the slower of the two data structures?
When working with big data is it good practice to read in as data.table then convert to data.frame to perform dplyr operations?
Is there another strategy I am missing?

The tidyverse also encompasses readr with faster functions for reading text files, e.g. read_csv (https://cran.r-project.org/web/packages/readr/README.html).

You are confounding two issues here, speed of operations of data.table vs. data.frame, and reading in such data.

I would venture a guess that read.csv and read.table is slow, because it does a lot of inefficient guessing and (by default) tries to convert strings to factors. When dealing with large data sets, you should always prefer to tell the routine what it is reading, rather than letting it guess.

Lastly, if you are dealing with very large data sets, specialized subroutines that are implemented in Rcpp or the like might be preferable, assuming your data is structured.


How to set up multi cluster spark without hadoop on Google Compute engine I'm new to apache spark.

Is it possible to configure multi cluster spark without hadoop?

If so, can you please provide the steps.
I would like to create clusters on Google Compute Engine  (1-master, 1-worker)

As explained in the comments by @Emre, there is a python script on github which you can use to set up a cluster of whatever size you like: Spark Cluster on Google Compute Engine | Ido Green.  It allows you to specify the instance type to use for the workers (slaves) and separately what to use for the master.

Note that it uses gcutil, which is deprecated in favor of gcloud compute


How I can interpret the attached tree? Is the total sample size = 30.891 and the overall percentage of “yes” = 11,3%? How I can describe the leaves predicting "yes" outcome in term of explicative variables and values?
What are the variables appearing in the second tree level?

Thank you in advance!

Initially in the uppermost node, you have 30 samples or data points that needs to be classified into yes/no category. Out of 30, nearly 27 are with no label and 3 are with yes label.

In the next level, it gets divided on the basis of poutcome value. Records in left side of the tree at level 2 are one with poutcome value is non-existent or fail while that on right hand side are with value success.


How to tune parameters batch by batch? As the title states, I am trying to cluster a huge dataset and cluster it by using sklearn.Birch to learn incrementally.

If it's a small dataset, I could just use gridsearchcv.

However, there's no built-in way to do that with large dataset in scikit-learn right now.

I'm curious if there is any suitable/general way to tune parameters batch by batch?

In Grid Search, Random Search, we trained different independent model i.e. irrespective of the outcome of previous models and later evaluate all these independent models.

  We have another automatic technique for hyper-parameter optimization,
  known as Bayesian  HyperOpt.

It took the references of previous model and try to identify the best new set of hyper param and trained a new model.
This two step process gets repeated, until we get the desired results.

You can get many online references for complete implementation.

Although I could not find any approach for batch by batch, I thought its worth to mention Bayesian approach.


How to quantify a tokenized user agent string for a neural network? I am currently experimenting with user agent strings. My current plan is to tokenize the user agent string and split it in its important parts. Like OS, Browser Name, etc.

After that, does it make sense to just create some sort of map where iOS = 0.1 and Windows = 0.2 where i map the strings to numbers for the operating system feature of my network?

Is that the current best approach to work with such data, or should I try a different approach? The network should predict which category the user will click, and than move that category to a more prominent place.

For example, I expect that an iOS user prefers an iPhone mobile case more than an android one.

Thank you for helping with my experiments.

iOS, Windows and Android should not be viewed as strings but as categories. Categories should be one-hot encoded, and should not share the same input as they are not related to each other.

So let's assume we work with two operating systems: iOS and Windows.

So for iOS, input is: 1 0 and for Windows input is: 0 1.

But now we want to add another OS: Android.

iOS becomes: 1 0 0, Windows 0 1 0, Android: 0 0 1. Note that by adding android, you do not actually change the input! You only add an extra 0 for iOS and Windows, and 0's don't get computed in neural networks 0*weight = 0.

So by adding an OS, you do not affect what the network has learned for the previously trained OS's.


Clustering Tweet Data using DBSCAN Algorithm I am doing a tweet clustering using DBSCAN algorithm. I use all the preprocessing steps and convert sentences to vector format before applying the algorithm. However, It always puts a lot of tweets in to the '0' class. The following is the plot showing eps with number of clusters.

The following are the parameters that I pass.
dbscan = DBSCAN(eps=0.15, min_samples=2, metric='cosine').fit(x)

The following are the resulting clusters.

label
-1     1221
 0     1349
 1        2
 2        2
 3        4
       ...
 67       3
 68       3
 69       2
 70       2
 71       2

What is the reason that class 0 getting a high number of tweets than any other classes?

Two things: eps and quantitative representation of text.
You see that there is only for eps=0.15 a lot of clusters. But for others a lot less. This is hyper parameter that needs to be optimised (and min_samples)
And the other thing thats more important is what you use quantitative representation of text. You said Bag of Words, TFIDF, Spacy Vectors and also, Word2Vec, but did you tune them? DId you tree embeddings etc etc. There is a lot of improvement here, and when its good dbscan will function a lot better.


Understanding Tensorflow LSTM models? I have some trouble understanding LSTM models in TensorFlow.
For simplicity, let us consider the Example program.

I use the tflearn as a wrapper as it does all the initialization and other higher level stuff automatically.

Till line number 42 net = tflearn.input_data([None, 200]) it's pretty clear what happens.

You load a dataset into variables and make it of standard lengths in this case 200. for both the input variables and also the 2 classes present in this case are converted to one hot vectors.

What I would like to know here is how the LSTM takes the input and across how many samples does it predict the output?

What do these parameters indicate: n_words=20000 & net = tflearn.embedding(net, input_dim=20000, output_dim=128)?

My goal is to replicate the activity recognition dataset in the paper.

For example, I would like to input a 4096 vector as input to the lstm and the idea is to take 16 of such vectors and then produce the classification result.

I think the code would look like this but I don't know how the input to the LSTM should be given

from __future__ import division, print_function, absolute_import

import tflearn
from tflearn.data_utils import to_categorical, pad_sequences
from tflearn.datasets import imdb

train, val = something.load_data()
trainX, trainY = train #each X sample is a (16,4096) nd float64
valX, valY = val #each Y is a one hot vector of 101 classes.

net = tflearn.input_data([None, 16,4096])
net = tflearn.embedding(net, input_dim=4096, output_dim=256)
net = tflearn.lstm(net, 256)
net = tflearn.dropout(net, 0.5)
net = tflearn.lstm(net, 256)
net = tflearn.dropout(net, 0.5)
net = tflearn.fully_connected(net, 101, activation='softmax')
net = tflearn.regression(net, optimizer='adam',
                         loss='categorical_crossentropy')

model = tflearn.DNN(net, clip_gradients=0., tensorboard_verbose=3)
model.fit(trainX, trainY, validation_set=(testX, testY), show_metric=True,
          batch_size=128,n_epoch=2,snapshot_epoch=True)

Arsenal,

We have to reshape the input data to fit in the LSTM's input layer like:

   trainX= np.reshape(trainX, (-1, 16, 4096))
   testX = np.reshape(testX, (-1, 16, 4096))

When one layer's output goes to the following layer's input, the earlier layer must produce a sequence like:

    net = tflearn.lstm(net,128, dropout=0.2, return_seq=True)
    net = tflearn.lstm(net,128)
    net = tflearn.fully_connected(net, num_classes, activation='softmax')

Hope this helps.