XGBoost Feature Importance, Permutation Importance, and Model Evaluation Criteria I have built an XGBoost classification model in Python on an imbalanced dataset (~1 million positive values and ~12 million negative values), where the features are binary user interaction with web page elements (e.g. did the user scroll to reviews or not) and the target is a binary retail action. My ultimate goal was not so much to achieve a model with an optimal decision rule performance as to understand which user actions/features are important in determining the positive retail action.

Now, I have read quite a bit in forums and literature about evaluating/optimizing an XGBoost model and subsequent decision rule, which I assume is required before achieving my ultimate goal. It seems that there are a lot of different ways to evaluate the decision rule part (e.g.  Area Under the Precision Recall Curve, AUROC, etc) and the model (e.g. log-loss). I believe that both AUC and log-loss evaluation methods are insensitive to class balance, so I don't believe that is a concern. However, I am not quite sure which evaluation method is most appropriate in achieving my ultimate goal, and I would appreciate some guidance from someone with more experience in these matters.

Edit: I did also try permutation importance on my XGBoost model as suggested in an answer. I saw pretty similar results to XGBoost's native feature importance. Should I now trust the permutation importance, or should I try to optimize the model by some evaluation criteria and then use XGBoost's native feature importance or permutation importance? In other words, do I need to have a reasonable model by some evaluation criteria before trusting feature importance or permutation importance?

So your goal is only feature importance from xgboost?
Then don't focus on evaluation metrics, but rather splitting.
I would suggest to read this. Using the default from tree based methods can be slippery.


30 Years Of Excel Test Data I am a CS intern at an industrial company that has 30 years of excel files that need to be analyzed. Looking at the data, only a fraction of the files need to be looked at and used. After those files are identified, I need to pull out values from specific columns. The real issue is that there is no standard excel format for the tests and each column name can be different (ex. 'Front Axial Temperature' vs 'axial temp front') but contains the same type of data.

So, the data is semi-structured, are there any python/java libraries which are good at pulling such data out? I could create some regex patterns but as you can see the above names can be drastically different (while still meaning the same thing).

I would also be interested in looking at services that do this already.

I see three main ways of comparing columns: automated comparisons of column names (e.g. regex,  Levenshtein distance), comparing content (e.g. compare mean and standard deviation of the data for the column; if the mean value of a column is 10,000 then it probably isn't recording Front Axial Temperature), and manual comparison. You can combine these, for instance clustering on column names and content, then manually looking at the contents of each cluster. The smaller the number of different column names, the more you can rely on manual examination. You may also be able to get other metadata sources, such as looking for documentation for whatever process generated the files.


Why can we not split train test data with 0.01 as parameter or 99% training data Most of the blogs mention about a good thumb rule to be 80-20 split for the train and test respectively.

My special case is a time series dataset and it is for the stock prices, which IMO is very time sensitive.

Question? Why can we not have a 99-1 train test split, for the model to learn all the information and time trends. Since my prediction will be in the future I will be ever increasing my test data set. I am using a neural network (rnn-lstm) for my prediction.

I am aware about cross_validation, froward_chaining which are better ways to train a time series dataset.

The idea of splitting 80%-20% is to use as much data as possible for training while keeping enough (labeled) data in the test set in order to reliably evaluate the performance. It's fine to use a smaller test set if it's big enough, but if it's too small the evaluation might not reflect the real performance of the model.

Your idea of using future test data is fine, it simply means that the reliable evaluation is delayed until you obtain enough test data. In the meantime you cannot be sure that your model performs well, so I would avoid using it for real trading on the stock exchange ;)


Some questions about feature hashing in the context of document classification I'm trying to understand feature hashing, specifically in the context of document classification.

I'm under the impression that it is useful because:

it allows us to easily deal with 'new' words/features/predictors that we haven't seen before
it is rather efficient because it allows us to exploit sparsity which is common in document/word data.

Some questions:

Say my word/feature set consists of 50,000 unique words and I use 16 hash bits $(2^{16}=65,536)$. In this setting, is it likely each word gets
mapped to one of the 65,536 indices?
How does this aid us in the situation where I build a model and try to make a prediction on a document that contains a word we haven't seen before?
Say my feature set is 100,000 words and I still use 16 hash bits. Does this effectively mean I have (on average?) $100,000-65,536 = 34,464$ words mapped randomly on to other words?

It seems I'm missing crucial point or idea here. Can anyone recommend some good introductory material?

Throughout the hashing process those words may map to the same indices and for the example you gave, i'll say it's extremely likely you'll have collision.  How the hashing table chooses to handle that is up to the implementation.

Feature hashing may purpose is when you have a large amount of data and that data tends to be sparse.  Words tend to be a good example because word matrix of counts tend to be rather sparse.

The hashing trick is useful for large datasets because the more traditional ways of handling text data is essentially by making two passes on the dataset.  You basically have to go over the dataset once to create a dictionary and then make your transformation.  For large data this can be rather expensive process.  Feature hashing allows for a one time pass of the data.  This also enables the ability to do online learning, which brings me to this point.  I wouldn't say that feature hashing helps with new words rather that it basically handles it for you.   In traditional approaches you have to ignore the new word or create a new dictionary.  Feature hashing will simply just map it to an existing index or a new one if there's space.

The last point is that feature hashing rarely if ever improves performance of a model.  It also hurts the ability to be introspective.  You are going to lose information, and in fact, you can view feature hashing as a way to introduce noise.  However, research indicates that in large sparse data, the noise does not impact performance in major ways.

Hashing Function

Large Scale Hashing

Short idea behind hashing


Clustering of Temporal Data I have been trying to find the correlation b/w the following type of temporal data for quite some time.

DataSet: A A B A B C B C B A B C X Y X Y Z X Y X Z Y Z A B A B C B (The actual dataset has 1000's of unique points)

I have to find some sort of similarity value b/w these points (my points I mean 'A', 'B', 'X', etc). For example similarity b/w 'A' and 'B' in above dataset is high so similarity('A', 'B') should return me a high value and similarity('A', 'X') should return a low value.

I am trying to calculate the similarity so that I can cluster the similar points. (For above dataset, possible good clusters are {A, B, C}, {X, Y, Z})

Till now I have tried the following approaches:

I tried to find segments of data whose correlation = len(segment)/unique(points in segment) value is high. If the value is high than that means in this segment the same type of points are coming back and forth and therefore highly correlated. The problem with this correlation formula is that it is not robust enough. For example Dataset is: X Y X X X X X Y X. Now the correlation value for subsegment with starting index = 3 and ending index = 7 is higher (correlation = 5/1) than the correlation value of the whole Dataset i.e 1 to end (correlation = 9/2). Now due to such problems I have to do a lot of manual settings, like select a segment having more unique points if correlation diff b/w those segments is less than some manually selected value (tradeoff b/w correlation value and unique points), and this does not work when I have 1000's of unique points.
Word2Vec: I ran this algorithm on the whole data as a single sentence (i.e merged the data in a single sentence separated by space). I don't exactly understand this algorithm but it seems to not work on a single sentence. It gives > 0.99 similarity b/w almost all points. I don't know why is this not working since it is supposed to learn the similarity b/w words based on how they are occurring in a sentence.

How should I approach this clustering problem? Can I tweak word2vec to fit for my problem? A better mathematical formula? Any other way?

Please correct me if I am doing something wrong.

Have a look at Deep Walk.  It's essentially word-to-vec applied to "sentences" created from a graph (nodes and edges).  If you can express your data as a graph, you can apply it.


How to study the effect of eps in sklearn.cluster.DBSCAN? I posted this question on stackoverflow.com and have not received any answer. In case I get an answer from one of them, I will inform on the other.

I have a dataset and is requested by my professor to study the effect of eps in sklearn.cluster import DBSCAN? Could you please suggest some ways to measure its effect? I generated eps = np.arange(start = .1, stop = 1, step = .1), but I don't know what should I compare.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.cluster import DBSCAN
%config InlineBackend.figure_format = 'svg' # Change the image format to svg for better quality
don = pd.read_csv('https://raw.githubusercontent.com/leanhdung1994/Deep-Learning/main/donclassif.txt.gz', sep=';')

kmeans = DBSCAN(eps = .2)
y_kmeans = kmeans.fit_predict(don)

plt.figure(figsize = 1.5 * np.array(plt.rcParams['figure.figsize'])) # This is to have bigger plot
plt.scatter(don['V1'], don['V2'], c = y_kmeans, cmap = 'viridis')

eps = np.arange(start = .1, stop = 1, step = .1)

The meaning of $\epsilon$ is that of the neighbourhood size. The neighbourhood of a point $p$, denoted by $N_{\epsilon}(p)$, is defined as the $N_{\epsilon}(p) = \{q \in D | dist(p,q) \leq \epsilon \}$. Here $D$ is a database of $n$ objects (points) and $q$ a query point.
So what you Professor probably wants you to do is to evaluate goodness of clustering algorithm (DBSCAN) based on its input Parameters Epsilon. This goodness of fit you can measure for example with Silhouette score


How do I find the minimum distance between zip codes in R? I have a dataset that lists all zip codes in the U.S., their types (standard, po box, university, etc). I want to replace po box and university zip codes with the next closest standard zip code. I broke down the dataset by state so that R wouldn't have to make as many calculations. In theory, I would like to have standard zip codes in the first column and zip codes that need replacement in the first row, and have the distance between the two be the intersection value.

For example,

                REP 1   REP 2   REP 3   REP 4
STD 1           0.215   0.152   0.025   0.124
STD 2           0.365   0.410   0.074   0.234
STD 3           0.234   0.201   1.322   0.683
STD 4           0.543   0.282   0.483   0.094
MINS            STD 1   STD 1   STD 2   STD 4

where STD 1 is a standard zip code with its own latitude and longitude, and REP 1 is a zip code that needs to be replaced (is a university/po box zip) with its own latitude and longitude. I only have about 5 weeks of experience in R, so please bear with me if something doesn't quite make sense to me immediately. I have tried to do this in excel and having a sheet with close to 10,000 columns by 40,000 rows crashes every time that I try to calculate all of the distances because there are just too many calculations.

I have a feeling that either the apply() or mapply() functions are needed here. I want to calculate the distances using a formula that considers the curvature of the earth, (euclidean, etc) like dist() or the geosphere package to maintain accuracy and be reproducible.

If there is anything else that would be helpful to add on here, let me know and I'll upload it asap. Here is my R code for Alaska, the first state in alphabetical order.

AK<-subset(db,STAABBRV.x=="AK")
AKPO<-subset(AK,ZipCodeType!="STANDARD",select=c("ZIP_CODE","ZipCodeType","Long","Lat"))
AKPO<-within(AKPO,{IS_PO=ifelse(ZipCodeType!="STANDARD",1,0)})
AKSTANDARD<-subset(AK,ZipCodeType=="STANDARD",select=c("ZIP_CODE","ZipCodeType","Long","Lat"))
AKSTANDARD<-within(AKSTANDARD,{IS_PO=ifelse(ZipCodeType!="STANDARD",1,0)})
table<-rbind(AKSTANDARD,AKPO)
table$ZipCodeType<-NULL
rm(AK,AKPO,AKSTANDARD)

This sets up a table that has column names "ZIP_CODE", "Long", "Lat", and "IS_PO". "IS_PO" is a numerical indicator for whether or not the zip code is standard or po/university. 1 indicates that the zip code is a po/univ zip and 0 indicates a standard zip. I did this because some functions required that the data in the dataset be the same type (numerical).

Here are some of my failed attempts at writing code to calculate the minimum distances.

 lapply(bit::chunk(1, nrow(zipcode), 1e2), function(ridx) {
  merge(zipcode, zipcode[ridx[1]:ridx[2]], by = "dum", allow.cartesian = T)[
    , dist := distGeo(matrix(c(longitude.x, latitude.x), ncol = 2),
                      matrix(c(longitude.y, latitude.y), ncol = 2))/1609.34 # meters to miles
    ][dist <= 5 # necessary distance treshold
      ][, dum := NULL]
}) %>% rbindlist -> zip_nearby_dt

DOESITWORK<-apply(db, 1, function(x) spDistsN1(matrix(x[3:4], nrow=1),
                                    x[5:6],
                                    longlat=TRUE))

mins<-apply(Lat,1,function(x)return(array(which.min(x))))
mins<-data.frame(row=names(mins),col=mins)
Lat$mins<-apply(mins,1,FUN=function(x)return(paste(x["row"],colnames(Lat[as.numeric(x["col"])]),Lat[x["row"],as.numeric(x["col"])],sep="/")))

I think I have read your question correctly it looks like you need a nearest neighbor implementation. If you are unfamiliar with the concept you can find the wiki article here https://en.wikipedia.org/wiki/Nearest_neighbor_search.

I went ahead a wrote an example implementation you can use as a guide. Please note that this is a brute force method and not useful for big data sets. Once you have a grasp of the material I suggest checking out some libraries like RANN that have "real" implementations.

Read in some random test data and clean
   For this test let us assume that we want to find the
   closest AMERICAN city for each location

coord_data = read.csv("~/Downloads/SalesJan2009.csv", stringsAsFactors = F)
coord_data$id = c(1:nrow(coord_data))
coord_data$is_usa = ifelse(coord_data$Country == "United States", 1, 0)
coord_data = coord_data[ , c("id", "Latitude", "Longitude", "is_usa")]
names(coord_data) = tolower(names(coord_data))

Define your distance function.
         Here we have geo-coordinates over long distance so Euclidean will not do.
         I am using the Law of Cosines to calculate great circle distance
        but Haversine and Vincenty should be considered given your needs.
        To learn more start here: https://en.wikipedia.org/wiki/Great-circle_distance.

greatCircleDistance = function(latAlpha, longAlpha, latBeta, longBeta, radius = 6371) {
  ## Function taken directly from Wikipedia
  ## Earth radius in km is default (6371)
  ## Long/Lats are in degrees so need helper function to convert to radians
  degreeToRadian = function(degree) (degree * pi / 180)
  deltaLong = degreeToRadian(longBeta) - degreeToRadian(longAlpha)
  sinLat = sin(degreeToRadian(latAlpha)) * sin(degreeToRadian(latBeta))
  cosLat = cos(degreeToRadian(latAlpha)) * cos(degreeToRadian(latBeta))
  ## acos is finicky with precision so we will assume if NA is thrown
  ## the argument was very close to 1 and therefore will return 0
  ## acos(1) == 0
  acosRaw = suppressWarnings(acos(sinLat + cosLat * cos(deltaLong)))
  acosSafe = ifelse(is.na(acosRaw), 0, acosRaw)
  acosSafe * radius
}

Distance between Basildon, UK and Parkville, US

greatCircleDistance(coord_data$latitude[1],
coord_data$longitude[1],
coord_data$latitude[2],
coord_data$longitude[2])

 Returns [1] [1] 6929.351 km.

It matches Google's calc so we are good to go!

Brute Force Example:
     As you noticed in your Excel sheet this will blow up quickly as data set gets larger. There are much more efficient ways of implementing the search. One idea is to start with the the geo-data structure itself and write an R-Tree, but I'll leave that for you.

 bruteForceNearestNeighbor = function(geoData) {
      makeCoordinate = function(idx) {
        c("id" = idx, "latitude" = geoData$latitude[idx], "longitude" = geoData$longitude[idx])
      }
      singleCoordMinDistance = function(coordinate, locations) {
        locationsUS = locations[locations$is_us == 1 & locations$id != coordinate["id"], ]
        distances = mapply(greatCircleDistance,
              latAlpha = coordinate["latitude"],
              longAlpha = coordinate["longitude"],
              latBeta = locationsUS$latitude,
              longBeta = locationsUS$longitude)
        closestIndex = which(distances == min(distances))
        locations[closestIndex, "id"]
      }
      nearestNeighbors = vector("numeric", nrow(geoData))
      for ( i in 1:nrow(geoData) ) {
        coord = makeCoordinate(i)
        nearestNeighbors[i] = singleCoordMinDistance(coord, geoData)
      }
      nearestNeighbors
    }

    coord_data$nearest_neighbor = bruteForceNearestNeighbor(coord_data)


How to segment old digitized newspapers into articles I'm working on a large corpus of french daily newspapers from the 19th century that have been digitized and where the data are in the form of raw OCR text files (one text file per day). In terms of size, one year of issues is around 350 000 words long.

What I'm trying to achieve is to detect the different articles that form a newspaper issue. Knowing that an article can be two or thee lines long or very much longer, that there is no systematic typographic division and that there are a lot of OCR errors in each file.
I should also mention that I don't have access to others OCR data like document layout in XML or so.

I've tried the TexTiling algorithm (the nltk implementation) but the results were not really conclusive.

Before diving deeper by myself I was wondering if maybe some of you would have some hint about a task like this one : train a machine learning model, try others algorithms ?

As far as I know topic segmentation is not a particularly easy task with clean data, so it's likely to be challenging with noisy old French.

It's not exactly the same problem so I'm not sure if this is useful but you might want to look into using stylistic features in order to help the model detect the changes between articles. There has been a fair amount of work on the task of style change detection as part of the PAN series (the task has run for 3 years, results and papers are available from the previous years).

Hope this helps.


Grouping by similarity I would like to find a way/algorithm to group people into, say, four groups by their answer similarity to yes/no questions.
So, each pair of people in one group would have given the same answers for a big part of the questions – (mostly) bigger than compared to people from the other three groups.
Would appreciate any pointers to the right direction.

I would start with the simple option: represent every person as a boolean vector in which every position represents the answer to a particular question (the length is the total number of questions). Then you can apply any standard clustering algorithm, such as K-means (hierarchical clustering would probably also work with data like this).


How to optimize hyperparameters in stacked model? I was wondering whether somebody could explain how to optimize hyperparameters for the base learners and meta algorithm when stacking? In many tutorials they seem to be plucked out of thin air!

Thanks,

Jack

I believe the most common way involves some slight data leakage during the training step that is often ignored. The "correct" way involves giving up more training data but many have empirically realized that giving up more training data is often worse.

Split your data into training and testing.
Split the training set into k-folds.
Train your base models using all of the k folds. Save the predictions from the best (optimized in terms of hyperparameters) base models, for each of the k folds.

Example: suppose we use 3 fold cross validation and have two base models. For each base model, we find the best hyperparameters that on average optimize some loss function over these three folds. We then save the predictions made by this optimal base model over these same three folds (i.e. we save the exact same predictions that you must have made when scoring your model over all three folds).

In essence, you are transforming your original training set with your base models in pieces and reconstructing it back together.

Using your reconstructed training set of predictions from folds 1, 2 and 3, use k fold cross validation again to train your combiner (i.e. find optimal hyperparameters again but for the combiner). You can choose to use the same splits as you did in step three but it doesn't matter. This is where the problem lies with data leakage. No matter how you split your data here you will have variables in your test set that were directly created from observations in your training set. For example, suppose fold 1 and fold 2 in the previous example make up the training set, and therefore fold 3 is used for validation. Since both fold 1 and fold 2 were created using fold 3 observations there is a chance that you overfit here due to optimism in your validation scores. In Kaggle competitions this is often ignored. My guess is probably because if you are stacking you have a large dataset, and if you have a large dataset minor data leaks become less of an issue (in general).
Fit your base models using your entire training dataset, using the hyperparameters found in step 3. Predict the training set to generate the required meta features for your combiner, and also predict the test set created in step 1.
Fit your combiner using the training set generated in the previous step, using the hyperparameters found in step 4.
Predict the test set that you had created from your base models in step 5, using the combiner model from the prior step. Score your loss and report this score as your models performance.


Having trouble scaling scores of logistic regression I am constructing a credit scorecard using logistic regression, similar to the one shown here. However, when trying to convert the coefficients of logistic regression into score representation (by scaling the values using the provided formula) I am getting numbers that dont make much sense.
Formula used for calculating scores:
Score_i= (βi × WoE_i + α/n) × Factor + Offset/n

where βi is the coefficient of the logistic regression (of variable i),

WoE_i is the weight of evidence of corresponding variable,

α is the intercept of the logistic regression,

Factor is calculated as PDO / ln(2),

Offset is calculated as target_points - (factor * ln(target_odds))

n is number of variables used in regression

In my case PDO = 50, target_odds = 2, target_points = 500, n = 81, and intercept is -0.12686514.
Here is an example of one of my features:

As it can be seen, both WoE and the coefficients increase in value as the "revenue" variable increases. The score however does not act in this way. Initially I just assumed the scores acted in inverse proportion to those two values so I added a negative sign in front of the formula:
Score_i= -(βi × WoE_i + α/n) × Factor + Offset/n
But for some other features, the score is proportional to the other values:

And adding the negative sign would now make this feature have a score that does not make much sense.
How can I keep the scaling consistent? What am I doing incorrectly here?

Turns out I was using a wrong formula for scaling the scores. The formula
Score_i= (βi × WoE_i + α/n) × Factor + Offset/n
is applicable only if the logistic regression is done after doing WoE Transform, meaning for all features (not for all bins!) a WoE value is assigned and then the classifier is fit.
The way I was doing it was binning first, and feeding columns with dummy values into the classifier.
Reference to correct approach


Machine learning toolkit for Excel Do you know of any machine learning add-ins that I could use within Excel? For example I would like to be able to select a range of data and use that for training purposes and then use another sheet for getting the results of different learning algorithms.

As far as I know, currently there are not that many projects and products that allow you to perform serious machine learning (ML) work from within Excel.

However, the situation seems to be changing rapidly due to active Microsoft's efforts in popularizing its ML cloud platform Azure ML (along with ML Studio). The recent acquisition of R-focused company Revolution Analytics by Microsoft (which appears to me as more of acqui-hiring to a large extent) is an example of the company's aggressive data science market strategy.

In regard to ML toolkits for Excel, as a confirmation that we should expect most Excel-enabled ML projects and products to be Azure ML-focused, consider the following two projects (the latter is an open source):

Excel DataScope (Microsoft Research): https://www.microsoft.com/en-us/research/video/excel-datascope-overview/
Azure ML Excel Add-In (seems to be Microsoft sponsored): https://azuremlexcel.codeplex.com


Training model on a Balanced vs Imbalanced dataset? Let's say that I have a 2-class classification problem where classes A & B have 10*N and N observations respectively.

I am pretty sure that the answer to my question depends on the specific classification problem and on the features of my dataset etc.. Still there are general analysis that can be done on my question.?

Something that I could clarify is that I am interested in having high recall in both classes ("macro-average recall"); not primarily in having the highest possible recall in the minority class as in imbalanced dataset classification problems such as spam detection, financial fraud detection or disease detection.

So for this, generally speaking, is it better to train my model on:

1) A: 10*N observations, B: N observations

2) A: 5*N observations, B: N observations

3) A: N observations, B: N observations

I am having an impression, that assuming we start with a balanced dataset then the more data you add to one class then the better the macro-average recall because of the new information added but after one point the dataset becomes so imbalanced that the performance of the model on the minority class probably deteriorates and hence the macro-average recall falls.

Am I right on this?

It's an interesting problem, but I think the answer might be disappointing:

Usually problems about maximizing recall are considered in the context of a trade-off with precision, i.e. the goal is to sacrifice some precision by predicting more positives, whether true (TP) or false (FP). Usually in a binary problem this is possible because we focus on one class of importance and consider the other as irrelevant, so its performance doesn't matter.

Here we have a binary problem where we want to maximize recall for both classes, so we cannot sacrifice one for the sake of the other: any gain in recall for a class is likely to cause a loss in recall for the other class, since predicting more instances as positive for class A would mean more negative instances for class B, and conversely. If we were talking micro-recall we could still use the higher importance in proportion of class A, but with macro-recall we can't. Note that this is a typical case where accuracy could be used, since it would give the same weight to both classes and would be a much simpler metric.

So the only way to improve macro-recall is to increase true positives. Let's look at the options:

  1) A: 10*N observations, B: N observations

Pro: performance for class A is maximized. Cons: class B proportionally disadvantaged, so possible loss in its performance.

  2) A: 5*N observations, B: N observations

Cons: class A a bit less advantaged; pro: class B a bit less disadvantaged.

  3) A: N observations, B: N observations

Pro: maximum performance for class B; cons: class A underperforms.

Actually the best option is probably to use all of N*10 instances for A and repeat 10 times the N instances for B, so that:

the learning method can benefit from all the available training data for A
class B is not proportionally disadvantaged.

But my guess is that it's unlikely to make a very big difference anyway. When it's a matter of increasing true positives, it's usually the features and the ML method which can have an impact.


What are the implications for training a Tree Ensemble with highly biased datasets? I have a highly biased binary dataset - I have 1000x more examples of the negative class than the positive class. I would like to train a Tree Ensemble (like Extra Random Trees or a Random Forest) on this data but it's difficult to create training datasets that contain enough examples of the positive class.

What would be the implications of doing a stratified sampling approach to normalize the number of positive and negative examples? In other words, is it a bad idea to, for instance, artificially inflate (by resampling) the number of positive class examples in the training set?

I would recommend training on more balanced subsets of your data. Training random forest on sets of randomly selected positive example with a similar number of negative samples. In particular if the discriminative features exhibit a lot of variance this will be fairly effective and avoid over-fitting. However in stratification it is important to find balance as over-fitting can become a problem regardless. I would suggest seeing how the model does with the whole data set then progressively increasing the ratio of positive to negative samples approaching an even ratio, and selecting for the one that maximizes your performance metric on some representative hold out data.

This paper seems fairly relevant http://statistics.berkeley.edu/sites/default/files/tech-reports/666.pdf it talks about a weighted Random Forest which more heavily penalizes misclassification of the minority class.


how to pass parameters over sklearn pipeline's stages? I'm working on a deep neural model for text classification using Keras. To fine tune some hyperparameters i'm using Keras Wrappers for the Scikit-Learn API.
So I builded a Sklearn Pipeline for that:

def create_model(optimizer="adam", nbr_features=100):
    model = Sequential()
    model.add(Dense(512, activation='relu', input_shape=(nbr_features,)))
    ...
    model.compile(loss='binary_crossentropy', optimizer=optimizer,metrics=["accuracy"])
    return model

estimator = Pipeline([("tfidf", TfidfVectorizer()),
                      ('norm', StandardScaler(with_mean=False)),
                      ("km", KerasClassifier(build_fn=create_model, verbose=1))])
grid_params = {
     'tfidf__max_df': (0.1, 0.25, 0.5, 0.75, 1.0),
     'tfidf__max_features': (100, 500, 1000, 5000,),
      ... }

gs = GridSearchCV(estimator,
                   param_grid,
                   ...)

I want to pass max_features parameters from tfidf stage to km stage as nbr_features. Any Hack/Workaround  to do that ?

I figured out how to do that by monkey patching ParameterGrid.__iter__ and GridSearchCV._run_search methods.

ParameterGrid.__iter__ iterates over all possible combinations of hyerparameters (dict of param_name: value). so i modified what it yields (one configuration of hyperparameters params) by adding "km__nbr_features" equal to 'tfidf__max_features':

params["km__nbr_features"] = params['tfidf__max_features']

Important: "km__nbr_features" must be missing from grid_params so the trick works.

Here is some code:

from sklearn.model_selection import GridSearchCV, ParameterGrid
import numpy as np
from itertools import product

def patch_params(params):
    # Updates a configuration of possible parameters
    params["km__nbr_features"] = params['tfidf__max_features']
    return out

def monkey_iter__(self):
    """Iterate over the points in the grid.

    Returns
    -------
    params : iterator over dict of string to any
        Yields dictionaries mapping each estimator parameter to one of its
        allowed values.
    """
    for p in self.param_grid:
        # Always sort the keys of a dictionary, for reproducibility
        items = sorted(p.items())
        if not items:
            yield {}
        else:
            keys, values = zip(*items)
            for v in product(*values):
                params = dict(zip(keys, v))
                yield patch_params(params)

# replacing address of "__getitem__" with "monkey_getitem__"
ParameterGrid.__iter__  = monkey_iter__

def monkey_run_search(self, evaluate_candidates):
    """Search all candidates in param_grid"""
    evaluate_candidates(ParameterGrid(self.param_grid))

# replacing address of "_run_search " with "monkey_run_search"
GridSearchCV._run_search = monkey_run_search

Then i preformed Grid Search normaly:

def create_model(optimizer="adam", nbr_features=100):
    model = Sequential()
    model.add(Dense(512, activation='relu', input_shape=(nbr_features,)))
    ...
    model.compile(loss='binary_crossentropy', optimizer=optimizer,metrics=["accuracy"])
    return model

estimator = Pipeline([("tfidf", TfidfVectorizer()),
                      ('norm', StandardScaler(with_mean=False)),
                      ("km", KerasClassifier(build_fn=create_model, verbose=1))])
grid_params = {
     'tfidf__max_df': (0.1, 0.25, 0.5, 0.75, 1.0),
     'tfidf__max_features': (100, 500, 1000, 5000,),
      ... }

# Performing Grid Search
gs = GridSearchCV(estimator,
                   param_grid,
                   ...)

Update:
In case you used RandomizedGridSearchCV you must monkey patch ParameterGrid.__getitem__ insted.

def monkey_getitem__(self, ind):
    """Get the parameters that would be ``ind``th in iteration
    Parameters
    ----------
    ind : int
        The iteration index
    Returns
    -------
    params : dict of string to any
        Equal to list(self)[ind]
    """
    # This is used to make discrete sampling without replacement memory
    # efficient.
    for sub_grid in self.param_grid:
        # XXX: could memoize information used here
        if not sub_grid:
            if ind == 0:
                return {}
            else:
                ind -= 1
                continue

        # Reverse so most frequent cycling parameter comes first
        keys, values_lists = zip(*sorted(sub_grid.items())[::-1])
        sizes = [len(v_list) for v_list in values_lists]
        total = np.product(sizes)

        if ind >= total:
            # Try the next grid
            ind -= total
        else:
            out = {}
            for key, v_list, n in zip(keys, values_lists, sizes):
                ind, offset = divmod(ind, n)
                out[key] = v_list[offset]
            return patch_params(out)

    raise IndexError('ParameterGrid index out of range')

ParameterGrid.__getitem__ = monkey_getitem__


how to avoid tokenizing w/ sklearn feature extraction I'm trying to analyze some machine log files and the column I'm looking at can have values like 'Part.C1.11.Reading Status'. I want to treat the complete string as one token and I don't want it to be split into 'Part', 'C1', '11' and 'Reading# and 'Status'.

I've got the vague feeling that the token_pattern is the parameter I need to adjust so I tried to specify the beginning and the end of a string like so:

from sklearn.feature_extraction.text import CountVectorizer
cvo = CountVectorizer(token_pattern='^$',lowercase=False)
OriginCV = cvo.fit_transform(log['Message_Origin']).toarray()

However, the last line throws an error:
ValueError: empty vocabulary; perhaps the documents only contain stop words

I've also tried to explictly include dot and space in the token_pattern like so:

cvo = CountVectorizer(lowercase=False, token_pattern=r"(?u)\b\w\w+\b|\.|\s")

Throws no errors but does not do the trick (no change except for an additional token '.')

Not changing the default token_pattern does split the string at the spaces and colons though. I found this solution, which however modifies the string by removing e.g. the colons.
Any other idea how to solve this?

Thanks,
Mark

You can change the tokenizer instead of the token pattern. The token pattern is used by the tokenizer. You can set as tokenizer any function that returns a list of string. For example, str.split

tokenizer=str.split

From the official documentation,

  tokenizer callable, default=None

  Override the string tokenization step while preserving the preprocessing and n-grams generation steps. Only applies if analyzer == 'word'.


Batch Normalization will disrupt multi-threading? Question:

In a Feed Forward network, assume we have a Mini-batch of 64 examples.
Our layer $l$ contains 20 neurons
Because every neuron in layer will require Mean and Variance from the whole minibatch, we must  to run all 64 'examples' in parallel (is this true?). We must wait until each such 'example' will have the appropriate sum (for that particular neuron!) available across all these minibatches.

Wouldn't this break parallelism? Previously, each of our example-threads could have computed everything, up to the activation values for the entire layer $l$. However, now we have to:

Wait for all example-threads to finish computing the sums for our layer $l$ (hence, our speed slows down to match the slowest example-thread)
Pause multithreading (!) and compute mean & variance for the layer $l$
Apply it to the sums, resume example-threads, so each thread completes the job, by activating its corresponding adjusted sums

Do I miss a part of the puzzle, or it really has to be done like this?

$$$$
my understanding of Batch Norm:

Batch norm (link to original paper) makes the search space more even & simpler to navigate during forward & backward passes.

It is usually applied to the sum $Z$ going into activation function (although there is a lot of debate (1) (2)). This is done for every neuron in a layer.

in usual settings, the neuron's output on layer $l$ is $$H_{i} = \phi(W\cdot X +b)$$  where W is a vector of weights leading into our $i$'th unit, $X$ is a vector of outputs from previous layer $l_{-1}$ and $b$ is a bias

With Batch norm, we still have our usual non-activated sum for neuron $i$:

$$Z_i = W\cdot X$$

but also have our mean for that neuron across all minibatches (Just for that neuron!)

$$\mu_i = \frac{1}{m} \sum_{p}^m Z_{ip} $$

Similar, for our Variance as:

$$\sigma^2_i = \frac{1}{m}\sum_{p}^m (Z_{ip} - \mu_i)^2$$

Our non-activated normalized sum for neuron $i$ (here, $\epsilon$ is a tiny number, in case if variance is too small, to avoid division by zero):

$$Z_{norm} = \frac{Z_i - \mu_i}{\sqrt{\sigma^2_i + \epsilon}}$$

We give a network a chance to learn the actual distribution it needs, so it's different from default distrib. We introduce 2 learnable parameters, $\gamma$ and $\beta$, and network can use it to 'squash and shift' the distributions as it needs (even canceling-out the batch-norm if required).

$$Z_{adjusted} = {\gamma}Z_{norm} + \beta $$

Finally neuron's output is computed as:

$$H_i =  \phi (Z_{adjusted}) $$

Bias isn't included, because we already have $\beta$ for the whole layer $l$.

My experience with batch and layer normalisation is that it is around 3 to 5 times slower per step than without it. Although the model may converge in less number of steps (which is what the authors report in the article) it takes much longer to calculate each step, and hence the whole training takes longer.

This has been reported in a few places like:

https://www.reddit.com/r/MachineLearning/comments/4ufmxy/layer_normalization_implemented_in_tensorflow/

https://github.com/cooijmanstim/recurrent-batch-normalization/issues/2

https://github.com/keras-team/keras/issues/1309


Classification model accuracy with ensemble methods I came through this statement in a Machine Learning text book based on law of large numbers:
Suppose you build an ensemble containing 1,000 classifiers that are individually correct only 51% of the time (barely better than random guessing). If you predict the majority voted class, you can hope for up to 75% accuracy!

I understand the analogy if we consider average over 1000 predictions but how majority votes lead to 75% accuracy from 51% (individual)?

This comes from the Binomial distribution, where you have $n=1000$ independent trials (models), $p=0.51$ of each model being right and since you care about the majority vote you want to have at least $k=500$ successful trials. That leads to:
$$\text{Pr}(k\geq500 \text{ models are right}) = \sum^{1000}_{k=500}\binom{1000}{k}0.51^{k}(1-0.51)^{1000-k}=0.74675\approx0.75$$
Here is how I calculated it:
    import numpy as np
    from scipy.stats import binom
    np.sum([binom.pmf(k,1000,0.51) for k in range(500,1001)])


Creating dataset - imbalanced or balanced? I'm trying to make an image classification model and I have 5 classes - A, B, C, D, E. The goal is to get the highest possible classification accuracy.
I have a database of images and I'm selecting the number of images I will use for each class for my model. I'm trying to figure out how many images I should pick for each class if the distribution of available data is as shown below.
Should I randomly choose something like 8,000 images for each class, in order to avoid class imbalance? Or should I just use as many images per class?
Images available per class:

A - 100,000
B - 70,000
C - 40,000
D - 10,000
E - 8,000

The easiest option to close your approach is too just run the model twice in both variations. But in my experience this specific data is bound to be imbalanced.
Your idea to fix the class imbalance is technically known as under-sampling. This is one technique which is used. You can try out a few other techniques which are mentioned here: https://www.kdnuggets.com/2017/06/7-techniques-handle-imbalanced-data.html


How to validate different method of clustering I have applied K-means and Hierarchical Agglomerative clustering method on some data and clustered them into 5 groups. For validation (agreement) purpose I used this formula:

c=(# of cluster shared/ Total cluster)*100 %

I feel its a wrong validation technique.

Can I use entropy, Rand index, Dunn number etc to measure the validation?

Could always try a silhouette score, although it may not perform well on high dimensional data.

Silhouette scores compare a datapoint's placement relative to its own cluster to its placement relative to other cluster/clusters.  The values returned will range between -1 (indicating a poor clustering) to +1 (indicating "good" clustering).  Using Wikipedia's example, use 2 clusters and assume there is an observation i that has been placed in Cluster 1.  The Silhouette score uses distance measures (Euclidean, for example, but others can also be used) to determine the average distance of i' to every other point in Cluster 1.  Then it determines the average distance of i to every point in Cluster 2.  The silhouette formula for point i would be:

silhouette(i) = $\frac{b(i)-a(i)}{Max\left \{ a(i), b(i) \right \}}$

Remembering for this example that i is in Cluster 1, then:

If an object is closer, on average, to the points in Cluster 1 than
Cluster 2, then a(i) < b(i) and silhouette score is positive for that
point.
If an object is closer, on average, to the points in Cluster 2 than
Cluster 1, then a(i) > b(i) and silhouette score is negative for that
point.

Run this comparison across all points and all clusters, then average, and you will obtain a global silhouette score.

You will see most people capture silhouette score in a for loop where they vary the number of clusters; this allows quick comparison of a range of clusters.


Logitic Regression cost function - what if ln(0)? I am building logistic regression from scrap.

The simplified cost function I am using is (from machine learning course on coursera):

in specific case during learning,
one observation in training set y is 0 - but the specific choice of betas in:

makes g(z) = h(x) = 1 , because e.g. z > 50.

in this case my right side od J is (1 - 0) * log(1 - 1) what is -inf (I am doing my calculations in python)

I understand that in this case value of cost function should be high because the probability of y = 1 is very big while the truth is that it actually is 0.

Is the problem approximation of g(50) being 1 instead of something like: 0.999999? Or there is some more fundametal error with my logic?

because this specific example the summation of cost of all observations is nan (not a number) in my code.

In practice, an offset is used to avoid log explosion due to values close to zero. For example $\hat{\text{log}}(x)=\text{log}(x + \text{1e-6})$.


multiplying two array in python3.7 I am trying to multiply two array in python 3.7 using numpy by using the following syntax:

array1 = np.array([{1,2,3,4},{5,6,7,8}])
print (array1)
array2=array1*array1
print(array2)

but this error arises

TypeError: unsupported operand type(s) for *: 'set' and 'set'

You are simply defining your array so that it is made of python sets. That is a different data structure which is not able to be  multiplied, unlike an array.

Just change your code to this:

array1 = np.array([[1,2,3,4],[5,6,7,8]])

The only difference is using square brackets instead of curly ones. These are python list objects (or standard arrays).

array2=array1*array1
print(array2)

[[ 1  4  9 16]
 [25 36 49 64]]


Question about "1x3 and 3x1 conv is equivalent to 3x3 conv" I see a lot of sites talk that we can substitute 1x3 conv + 3x1 conv for 3x3 conv.

In order to demonstrate easily, we use a 3x3 image as an example.

From the point of view of parameters, I know that we will get fewer parameters and end up with the same result.
If the image(3x3) perform a 3x3 conv, we will get a 1x1 scalar, and this kernel size is 3x3, means we have 9 parameters.
On the other hand, if the image first performs a 1x3 convolution, and then performs a 3x1 convolution on its output. It will intuitively get the same result, a 1x1 scalar, and this kernel size is 2x3x1 = 6, which means we have 6 parameters.

However, from the point of view of calculation time, I don't think 3x3 should be substituted by 1x3 & 3x1.

For example, if we use the image to perform a 3x3 kernel, we should do 9 times of pixel-wise multiply operations, and then one time of addition. Totally 10 operations to get the result.

But, if we use the image(3x3) to perform a 3x1 kernel, we have to implement [3(multiply) + 1(add)] * 3 times to get a 1x3 feature map, then we still need to perform a 1x3 kernel. It will then takes 3(multiply) + 1(add) operations. Totally 16 operations throughout the whole process.

What I want to show is that although we reduce the parameters from replacing 3x3 conv to 3x1 & 1x3 conv, we still need more time to calculate during the training process.

Training spends lots of time and implements 3x1 & 1x3 doesn't save our time. So, should we substitute 1x3 conv + 3x1 conv for 3x3 conv?

A 3x3 convolution can do things a 1x3 followed by a 3x1 cannot do, for instance diagonal filtering.

As you said, it's all about the number of parameters, the first convolution has far more capabilities than the second solution.


How can I access dataset from Nasa website So Now I was working on my project ("Failure prediction models in FoF") and the Tool I will be using is RapidMinder but I need to have a dataset to run the tool on, I found a nice description of dataset that could be helpful on Nasa website https://data.nasa.gov/external-dataset?datasetId=dc5d-zf5v , However I do not know how to access the data itself in form of CSV file.

It seems that there is no downloadable data (I clicked Show the complete Project library), but the project was completed in 2012, so you could ask the listed contact, Gary Jahns, explaining your motivation, to be sure: gary.c.jahns@nasa.gov


word/sentence alignment for English document I have a English document, which is preprocessed into two versions. I want to align words or sentences from these two versions of document. A simple example is as below:

I don't want to go there. My e-mail address ok.

should be aligned with

i do n't want to go there my e mail address ok

Or the tokens from the first sentence is aligned correctly with the tokens in the 2nd sentence. I have tried some sequence alignment methods, which do not perform well. Considering there are alignment tools for machine translation, automatic alignment for English should be easier right?

Use kaldi's align-text which align two sentences using Levenshtein distance.

Code: https://github.com/kaldi-asr/kaldi/blob/master/src/bin/align-text.cc


How much data to use for feature selection? Working on my master's thesis, this is a problem I'm unable to find good resources about.

I'm working with data of 18 participants, who are either active or passive. Each participant is then subjected to a 3 x 3 experiment and results in a total of around 676 trials per participant (around 12.168 trials in total). There are 100 data points in each trial but cannot be used separately from the trial (since its an EEG epoch).

My data consists of 579 features, so I need some sort of feature selection as literature shows that most of them are irrelevant, but I want to use a bottom-up machine learning approach (to verify this).

Is there a rule of thump/literature to use for the amount of data needed for feature selection?

Have you considered an approach that wouldn't require you to do feature reduction? For example, if you were to use a neural network, the need for feature reduction is greatly reduced. Now, obviously I haven't seen your data so I don't want to suggest that neural networks should be your only answer. But with that many features I would be more concerned with how I'm actually going to reduce them rather than the number of data points that I need to collect.


What can be done to increase the accuracy of a biological dataset? I have a biological unbalanced dataset on which I have applied deep learning, Support Vector Machine (all the kernel functions) and Artificial Neural network for multiclass classification (size: 139 samples , 5 attributes) in python. Unfortunately the accuracy is not exceeding 55%. What can be done to increase the accuracy? If a dataset cannot ever go beyond such an average accuracy, what is the solution?

There are many possible explanations.
Maybe just mistakes in the code or maybe there is just no better separation possible.
When usign SVM the features are extremely important and you should most of your time invest in choosing/designing them.
The statement "I have applied Deep learning" ist just to vague and therefore I can not give any tip.
Third when dealing with unbalanced data you have many possible actions.
I recommend reading:
Oversampling and undersampling in data analysis


How to visualize data of a multidimensional dataset (TIMIT) I've built a neural network for a speech recognition task using the timit dataset. I've extracted features using the perceptual linear prediction (PLP_ method. My features space has 39 dimensions (13 PLP values, 13 about first order derivative and 13 about second order derivative).

I would like to improve my dataset. The only thing I've tried thus far is normalizing the dataset using a standard scaler (standardizing features with mean 0 and variance 1).

My questions are:

Since my dataset has high dimensionality, is there a way to visualize? For now, I've just plotted the dataset values using a heat map.
Are there any methods for separating my sample even more, making it easier to differentiate between the classes?

My heat map is below, representing 20 samples. In this heatmap there are 5 different phonemes, related to vowels, in particular, uh, oy, aw, ix, and ey.
As you can see, each phoneme is not really distinguishable from the others. Does anyone know how could I improve it?

Like I said in the comment, you'll need to perform dimension reduction, otherwise you'll not be able to visualize the $\mathbb{R}^n$ vector space and this is why :

Visualization of high-dimensional data sets is one of the traditional applications of dimensionality reduction methods such as PCA (Principal components analysis).

In high-dimensional data, such as experimental data where each dimension corresponds to a different measured variable, dependencies between different dimensions often restrict the data points to a manifold whose dimensionality is much lower than the dimensionality of the data space.

Many methods are designed for manifold learning, that is, to find and unfold the lower-dimensional manifold. There has been a research boom in manifold learning since 2000, and there now exist many methods that are known to unfold at least certain kinds of manifolds successfully.

One of the most used methods for dimension reduction is called PCA or Principal component analysis. PCA is a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components. You can read more on this topics here.

So once you reduce your high dimensional space into a ${\mathbb{R}^3}$ or ${\mathbb{R}^2}$ space you will able to project it using your adequate visualization method.
References :

Information Retrieval Perspective to Nonlinear Dimensionality
Reduction for Data Visualization - Jarkko Venna

EDIT: To avoid confusion for some concerning PCA and Dimension Reduction, I add the following details :

PCA will allow you compute the principal components of your vector model, so the information are not lost but "synthesized".

Unfortunately there is no other imaginable way to display 39 dimensions on a 2/3 dimension screen. If you wish to analyze correlations between your 39 features, maybe you should consider another visualization technique.

I would recommend a scatter plot matrix in this case.


Improve Convolutional Autoencoder I just built a Convolutional Autoencoder to try to reconstruct a time series with shape (4000, 10, 30). This is the code, I used a batch size of 32, but I think it is overfitting since it performs well on the training set but starts not reconstruction well in the validation and test set. What are the steps I can do to improve it? How can I define the right number of filters for each layer?
kernel_size = 7
stride = 1
model = Sequential()
model.add(Conv1D(filters=128, kernel_size=kernel_size, activation='relu', padding="same", strides=stride,
                 input_shape=(TIME_STEPS, n_features)))
model.add(Conv1D(filters=64, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(Conv1D(filters=32, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(MaxPooling1D(pool_size=2))
model.add(Conv1D(filters=16, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(Conv1DTranspose(filters=16, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(UpSampling1D(size=2))
model.add(Conv1DTranspose(filters=32, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(Conv1DTranspose(filters=64, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(Conv1DTranspose(filters=128, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(Conv1DTranspose(filters=n_features, kernel_size=kernel_size, padding="same"))

This is the optimizer:
optimizer = tf.keras.optimizers.Adam(learning_rate=1e-04)
model.compile(optimizer=optimizer, loss="mse",
              metrics=[tf.metrics.MeanAbsoluteError(), 'accuracy'])

EDIT: Dropout version:
model = Sequential()
model.add(Conv1D(filters=20, kernel_size=kernel_size, activation='relu', padding="same", strides=stride,
                 input_shape=(TIME_STEPS, n_features)))
model.add(Dropout(0.2))
model.add(Conv1D(filters=15, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(MaxPooling1D(pool_size=2))
model.add(Conv1D(filters=10, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(Conv1DTranspose(filters=10, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(UpSampling1D(size=2))
model.add(Conv1DTranspose(filters=15, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(Dropout(0.2))
model.add(Conv1DTranspose(filters=20, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(Conv1DTranspose(filters=n_features, kernel_size=kernel_size, padding="same"))

After taking a look at your code, it seems that you've not employed any kind of regularization. You may want to use dropout. Moreover, in convolutional autoencoders, in the decoder part, there is a well-known artifact called checkerboard. I don't know how this can be a problem for your task since you're using one-dimensional convolution in the decoder. By the way, I guess using dropout will suffice. Try to use both in decoder and encoder.


Scalable Outlier/Anomaly Detection I am trying to setup a big data infrastructure using Hadoop, Hive, Elastic Search (amongst others), and I would like to run some algorithms over certain datasets. I would like the algorithms themselves to be scalable, so this excludes using tools such as Weka, R, or even RHadoop. The Apache Mahout Library seems to be a good option, and it features algorithms for regression and clustering tasks.

What I am struggling to find is a solution for anomaly or outlier detection.

Since Mahout features Hidden Markov Models and a variety of clustering techniques (including K-Means) I was wondering if it would be possible to build a model to detect outliers in time-series, using any of this. I would be grateful if somebody experienced on this could advice me

if it is possible, and in case it is
how-to do it, plus
an estimation of the effort involved and
accuracy/problems of this approach.

I would take a look at t-digest algorithm. It's been merged into mahout and also a part of some other libraries for big data streaming. You can get more about this algorithm particularly and big data anomaly detection in general in next resources:

Practical machine learning anomaly detection book.
Webinar: Anomaly Detection When You Don't Know What You Need to Find
Anomaly Detection in Elasticsearch.
Beating Billion Dollar Fraud Using Anomaly Detection: A Signal Processing Approach using Argyle Data on the Hortonworks Data Platform with Accumulo


Scatter Plot with Conditions Suppose I have a data frame

name = ['A', 'B', 'C']
score = [2,4,6]

I want to create a scatter plot with the following conditions, color the bubble as green if the score is greater than 3 and red otherwise. I'd also like to label the bubble with its respective name.

I'm only able to create a scatter plot with the bubble having the respective name.

This type of scatter plot is best performed in seaborn for Python.

# example from: https://python-graph-gallery.com/?s=scatter
# library & dataset
import seaborn as sns
df = sns.load_dataset('iris')

# Use the 'hue' argument to provide a factor variable
sns.lmplot( x="sepal_length", y="sepal_width", data=df, fit_reg=False, hue='species', legend=False)

# Move the legend to an empty part of the plot
plt.legend(loc='lower right')

The keyword hue will designate which variable is the one which is used for labeling the data points. Also, have a look at the link provided for more examples.


How to protect data from internal data scientists? In our company we want to protect data privacy internally. Meaning, we want to find a way to anonymize the data so the data science team members cannot expose it and yet still can use it for modelling.

I googled and read about Pseudonymization. But I mean, is it destroying the data? I didn't find any reliable source using it practically.

You could checkout the OpenMined Pysyft library which is a library for encrypted, privacy preserving deep learning built over Pytorch. PySyft decouples private data from model training.

Github link to the Pysyft library - https://github.com/OpenMined/PySyft


Recall score for each sample in multilabel classification Does it make sense to calculate the recall for each sample in a multilabel classification problem?
Suppose I have 3 data samples, each having its own true set of labels and predicted set of labels.

I want to see the match between the true set of labels and the predicted set of labels. I do not care for the true negatives or false positives in each prediction, so this translates to recall score for me. Programmatically, I would do an AND operation between y_predicted and y_true to get the number of true positives and divide it by the total number of true labels for each sample. (in other words, true positives/(true positives+false negatives))
My question is -

Is calculating recall per sample (not per label), usually done?
Is my thought process correct?

I've seen articles where a single recall is calculated for the whole matrix of y_true and y_predicted or recall is calculated for a single label.

This metric is usually referred to as a sample-based or example-based score and can be applied in multi-label cases (to recall and other scores too). You can find a brief explanation here.

Scikit learn has an implementation for it (see here for recall) when setting  average='samples' for recall_score:

  'samples': Calculate metrics for each instance, and find their average (only meaningful for multilabel classification where this differs from accuracy_score).


Implementation of BERT using Tensorflow vs PyTorch BERT is an NLP model developed by Google. The original BERT model is built by the TensorFlow team, there is also a version of BERT which is built using PyTorch. What is the main difference between these two models?

There are not only 2, but many implementations of BERT. Most are basically equivalent.

The implementations that you mentioned are:

The original code by Google, in Tensorflow. https://github.com/google-research/bert
Implementation by Huggingface, in Pytorch and Tensorflow, that reproduces the same results as the original implementation and uses the same checkpoints as the original BERT article. https://github.com/huggingface/transformers

These are the differences regarding different aspects:

In terms of results, there is no difference in using one or the other, as they both use the same checkpoints (same weights) and their results have been checked to be equal.
In terms of reusability, HuggingFace library is probably more reusable, as it is designed specifically for that. Also, it gives you the freedom of choosing TensorFlow or Pytorch as deep learning framework.
In terms of performance, they should be the same.
In terms of community support (e.g. asking questions in github or stackoverflow about them), HuggingFace library is better suited, as there are a lot of people using it.

Apart from BERT, the transformers library by HuggingFace has implementations for lots of models: OpenAI GPT-2, RoBERTa, ELECTRA, ...


Applying Sci-kit Learn's kNN algorithm to Fresh Data While I was studying Scikit-learn's kNN algorithm, I realized that if I use sklearn.model_selection.train_test_split, the provided data gets automatically split into the train data and the test data set, according to the proportions provided as parameters.
Then based on the train data, the algorithm looks at the k-nearest neighbor points closest to the test data points to determine whether the test data points belong to a certain criteria or not.
I was wondering whether there was a way to predict the criteria NOT for the test data sets, which were already a part of the provided data set, but brand new data that were not provided during the whole process.
Is there a way to do that using sci-kit learn?

KNN is not fitted to "the k-nearest neighbor points closest to the test data points". You specify the fit option, like:
neigh = KNeighborsClassifier(n_neighbors=3)
neigh.fit(X, y)

Usually this will be xtrain, ytrain, while you test the model performance using "new" (unseen) data and compare the true targets to the prediction.
neigh.predict(xtest)

or
neigh.predict_proba(xtest)

See docs: https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html


How to deploy a deep learning model in Flask (Python)? I am still an amateur with respect to deploying deep learning models as a flask app. I want to deploy a VQA model based on MMF (Pythia) in a flask app. Github repository for reference. Can someone explain in brief as to how to go about it? Thanks in advance.

You need to write a simple web application with one endpoint. I've checked the mmf doc you can load a model using it's keys:

from flask import Flask, jsonify, request
from mmf.common.registry import registry

model_cls = registry.get_model_class("<DESIRED_MODEL_KEY">)
model = model_cls.from_pretrained("<DESIRED_PRETRAINED_ZOO_KEY>")

@app.route('/predict', methods=['POST'])
def predict():
    data = request.get_data()
    prediction = model(data)
    return jsonify(list(prediction))

app.run(host='0.0.0.0', port=8000)

I haven't ran that code but I know for a fact that this won't run out of the box. data is a dictionary and you need to get the right key and convert the value
to a numpy array or Torch tensor. And predict will be a numpy array (or Torch tensor - but can be converted back to numpy) and you will need to convert everything back to a standard python format (because numpy or torch data types are not json serialisable).


How to train the same RNN over multiple series? I have multiple separate time series and would like to train the same LSTM network on them. How to do in this situation? I can't just concatenate timeseries (along time), because I am afraid network will be confused by jumps at the points of concatenation.

How to overcome?

just re-use

  model.fit()

on the fresh datasets using the already trained model, simple as that :) !

(given that you do it in Keras)


What is the reward function in the 10 armed test bed? The Sutton & Barto book on reinforcement learning mentions the 10 armed test bed in chapter 2, Bandit Problems:

  To roughly assess the relative effectiveness of the greedy and
  ε-greedy methods, we compared them numerically on a suite of test
  problems. This was a set of 2000 randomly generated n-armed bandit
  tasks with n = 10. For each bandit, the action values, $q_∗(a), a = 1, . . . , 10,$ were selected according to a normal (Gaussian) distribution with mean 0 and variance 1. On tth time step with a given bandit, the actual reward $R_t$ was the $q_∗(A_t)$ for the bandit (where $A_t$ was the action selected) plus a normally distributed noise term that was mean 0 and variance 1 [. . . .] We call this suite of test tasks the 10-armed testbed.

What is the reward function in the 10 armed test bed ?
I interpreted it as something like q*(a) + some normally random value.
where q*(a) is the true value of action a.

Why is the reward function chosen this way in the test bed and how does the reward function affect the value estimations and the graphs ?

I guess the reason i'm asking this question is because i'm not completely clear how a reward looks like in the real world where i don't know anything about q*(a).

The reward function in the Chapter 2 test bed is simply the "true" mean value for the chosen action, plus a "noise term" which is normal distribution with mean 0, standard deviation 1.

The noise has the same distribution as the initial setting of "true" values. The difference is you set the true values at the start and do not change them, then add noise on evaluation of each reward. The goal for the learner is then to find the best "true" value whilst only seeing the reward.

This matches your understanding as I read it from the question. You could write it like this:

Initialisation:

$\forall a \in A: q_*(a) \leftarrow N(0,1)$

Evaluation:

$R_t = r(A_t) = q_*(A_t) + N(0,1)$

Where $N(\mu,\sigma)$ is a sample from the normal distribution, mean $\mu$, standard deviation $\sigma$

  Why is the reward function chosen this way in the test bed and how does the reward function affect the value estimations and the graphs ?

For a bandit problem to be non-trivial, the reward function needs to be stochastic, such that it is not possible to immediately discover the best action, there should be some uncertainty on what the best action to take is, even after taking many samples.

So the noise is there to provide at least some difficulty - without it finding the best action would be a trivial $argmax$ over the 10 possible actions. The noise does not represent uncertainty in the sensing (although that could also be a real world issue), but variability of the environment in response to an action. The test examples could have almost any distribution (e.g. $p(-1.0|a=1) = 0.9, p(9.0|a=1) = 0.1$ for $q_*(a=1) = 0.0$), the authors made a choice that was concise to describe and useful for exploring the different techniques in the chapter.

The specific reward function will affect the learning graphs. The test bed has been chosen so that ratio of noise to magnitude of "true" values is high. In turn, this means that value estimates will converge relatively slowly (as a ratio to the true values), and this exposes differences between different sampling and estimation techniques when they are plotted by time step.

To answer your concern:

  I guess the reason i'm asking this question is because i'm not
  completely clear how a reward looks like in the real world where i
  don't know anything about q*(a).

In the real world you may need to sense or receive the reward from the environment. Obviously that complicates the test scenario, and doesn't add anything to the understanding of the maths, so the test bed just generates some imaginary distribution for the environment inside the problem. The "sensing" in the test is just assumed with a reward amount defined by the test.

To qualify as a simple (and static) bandit problem, the reward has to be immediately apparent on taking the action, and have no dependency on current state or history. That constrains the problem somewhat - it is not the full reinforcement problem. So real-world examples tend to be about gambling with limited choice on independent repeatable events.


Should a bias neuron be connected to previous neurons? I'm trying to add a bias neuron to my neural network that uses the backpropagation algorithm. I'm trying to figure out how I should go about this, should I treat the bias neuron as a regular neuron? which means it's connected to the neurons on the previous layer?

No, bias neurons are not connected to any previous neuron. This is visualized like this:

(source)


How do I load FastText pretrained model with Gensim? I tried to load fastText pretrained model from here Fasttext model. I am using wiki.simple.en

from gensim.models.keyedvectors import KeyedVectors

word_vectors = KeyedVectors.load_word2vec_format('wiki.simple.bin', binary=True)

But, it shows the following errors

Traceback (most recent call last):
  File "nltk_check.py", line 28, in <module>
    word_vectors = KeyedVectors.load_word2vec_format('wiki.simple.bin', binary=True)
  File "P:\major_project\venv\lib\sitepackages\gensim\models\keyedvectors.py",line 206, in load_word2vec_format
     header = utils.to_unicode(fin.readline(), encoding=encoding)
  File "P:\major_project\venv\lib\site-packages\gensim\utils.py", line 235, in any2unicode
    return unicode(text, encoding, errors=errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xba in position 0: invalid start byte

Question 1 How do I load fasttext model with Gensim?

Question 2 Also, after loading the model, I want to find the similarity between two words

 model.find_similarity('teacher', 'teaches')
 # Something like this
 Output : 0.99

How do I do this?

Here's the link for the methods available for fasttext implementation in gensim fasttext.py

from gensim.models.wrappers import FastText

model = FastText.load_fasttext_format('wiki.simple')

print(model.most_similar('teacher'))
# Output = [('headteacher', 0.8075869083404541), ('schoolteacher', 0.7955552339553833), ('teachers', 0.733420729637146), ('teaches', 0.6839243173599243), ('meacher', 0.6825737357139587), ('teach', 0.6285147070884705), ('taught', 0.6244685649871826), ('teaching', 0.6199781894683838), ('schoolmaster', 0.6037642955780029), ('lessons', 0.5812176465988159)]

print(model.similarity('teacher', 'teaches'))
# Output = 0.683924396754


Machine learning goal: given a population of 100,000 students, predict a group of 3,000, and minimize the median grade of that group In other words, I am looking to predict students that will fail out of school before it happens. The data includes socioeconomic status and other related variables.
I have tried an XGB binary classification (both tree and forest), but the problem is that it doesn't penalize severely wrong answers (predicting that a student will be in the bottom 3% in terms of grades, but they're actually A+ students). The result is that the average grades of the predicted students is quite low, but the median grades aren't actually that bad - there are a few extremely bad students that pull down the average but not the median.
I have tried a XGB regression (both tree and forest), but the problem is that I can't get the model to focus on the bottom 3%. It seeks to reduce error for all predictions. I couldn't care less about telling the difference between an A student and a B student, I only need to consistently identify the bottom 3%ile.
I was thinking that perhaps this could lend itself to reinforcement learning instead of supervised, but I know nothing about reinforcement... WOuld it be possible to make a reinforcement model where the goal is to minimize the median grades of the 3% of students predicted? Or are there any other machine learning techniques that would work?

Try writing a custom loss function for a regression model!
Keras' neural networks support this, for example. See https://stackoverflow.com/q/43818584/745868
(But many other libraries give support for this as well)
The only special thing about your custom loss function is that it doesn't add up the error of a datapoint if min(pred_y, actual_y) >= THRESHOLD


Which algorithm to use for predicting late deliveries at warehouse? I am trying to predict risk labels of late deliveries for the upcoming 1-2 months based on 3 years of historic delivery data.
The target variable is a categorical output based on how many days late a delivery was (on time, 1-2 days late, one week late, more than one week late).
Input variables include things like material information, suppliers location & distance to warehouse, # of times delivery was marked as delayed and suppliers monthly quality ratings as well as information on known quality issues registered with supplier over time etc.

So looking at the information above and that I have records per year of around 500.000 delivieries from 300 suppliers I believe random forests would be a good choice to start.

However, I´m thinking it would not capture important time sensitive trends, such as a supplier all of the sudden delivering late while he was reliable the last 3 years.
So what is a good way to capture time sensitive trends in RF or should I choose a different algorithm form the time-series domain?
Or is there a way to combine multiple approaches?

Thanks for you input!

Yes you could add a time component if you have dates in your data. Use the functions found in this link, for your work.

Essentially you would want to see if there are day based effects such as Are deliveries late on Fridays / Friday evenings / Early mornings / Late nights / are etc.

What you'd have to do for that is to strip out the day of the week from the relevant date variables and add it as a ordinal predictor to your data. If you want to strip out individual weekday effects, use an if else function to classify the day of the week as Monday / Not Monday, Tuesday / Not Tuesday, etc. You could do something similar to month of the year, week of the year, season, etc.


How can I count the number of occurrences of a category in dataset as part of an Sklearn Pipeline Let us say we have a dataset with a feature such as Surname.

arr['Surname'] = ['Smith', 'Jones', 'Johnson', 'Smith']

And I want to encode this categorical info as a new feature like

arr['Surname_Count'] = [2, 1, 1, 2]

with the caveat that it is done within an Sklearn pipeline. Are there quick ways to do this that do not involve rolling my own partition counting transformer?

You can check out Featuretools, which an open source python framework for automated feature engineering. Specifically for you, it can generate aggregation features such as count for your dataset.

After generating the new feature matrix with the desired column, you can use the matrix as you normally would within an Sklearn pipeline.


Counting how many rows contain chinese in mixed value dataset using python I'm wanting to count how often Chinese is used, I've tried langdetect but the issue is the values are a mix of numbers, text and even URLs which langdetect doesn't seem to support

import pandas as pd
from langdetect import detect

df = pd.read_csv("annotation_sample.csv", nrows=10)
df = df.drop(['id', 'profile_id', 'created', 'document_id'], axis=1)
df = df[pd.notnull(df['txt'])]
df

df.txt.apply(lambda x: detect(x))

Above works (although I've not figured out how to count a specific language) although it only works as its limited to 10 rows, when using the entire 10,000 it throws the 'No features in text' error

Pic of the data:

What I need help with is;
How do I avoid running this on rows that contain just numbers, urls etc? Is there a way to skip error'd ones and continue?

Secondly, how do I then count how many have come back as a specific language?

That should be feasible using some more python and pandas features:
- errors can be caught when you wrap the detect in your own function and return None (or what you prefer) instead of a language code
- results of the detect can be added to the df and counted, filtered, you name it.

def mydetect(txt):
    try:
        return detect(txt)
    except:
        return None

Pandas allows you to add the results to a second column of the dataframe, but I do not have the command handy at the moment, please check a tutorial.
Then you can filter the dataset or count occurences of a language.

You can also limit the except clause to certain errors if you think that would make more sense.


Why do we calculate partial derivative of Error w.r.t output of Neural Network during backpropagation? As seen in this image we calculate partial derivative of Error w.r.t output of the output neuron. Shouldn't it be normal derivative? Does not that particular error is determined by only that output?

The partial derivative is used precisely because it separates concerns about how the value is calculated (from all the other parameters and outputs in the network) from how the value affects the output. This is purely by definition so you can do the calculation.

The "normal" derivative of the error function can be expressed as the sum of any "complete" set of independent partial derivatives. There are a few such sets possible - in a simple feed-forward network one for each layer plus one for all the weights combined.

So in essence you are calculating the "normal" derivative w.r.t. the network weights, but due to the nature of the problem this is done by calculating the partial derivatives in multiple steps.

Caveat: I am probably not using those maths terms 100% accurately. For instance, I am ignoring the role of the training data and treating it as a constant.


Whether AUC can be calculated as average of sensitivity and specificity or not? I have gone through few of the paper for AUC calculation, following paper has mentioned AUC can be calculated as

AUC = (Sensitivity +Specificity)/2.

A systematic analysis of performance measures for classification tasks

Please let me know that whether it is correct or not.

The article you mentioned is referring to the AUC of Receiver Operating Characteristic Curve (ROC-Curve), that is different from the AUC of the Precision-Recall Curve (PR-Curve). See this post to know about the differences between the PR-Curve and the ROC-Curve. If what you want is that same value for the PR-Curve you should look at the F1-Score, see this wikipedia article for an explanation of the relation between the AUC of the ROC-Curve, the Youden Index and the F1-Score. Also a good indicator of the AUC of the PR-Curve is the Average Precision (AP) as explained here.


Clustering of documents using the topics derived from Latent Dirichlet Allocation I want to use Latent Dirichlet Allocation for a project and I am using Python with the gensim library. After finding the topics I would like to cluster the documents using an algorithm such as k-means(Ideally I would like to use a good one for overlapping clusters so any recommendation is welcomed). I managed to get the topics but they are in the form of:

0.041*Minister + 0.041*Key + 0.041*moments + 0.041*controversial + 0.041*Prime

In order to apply a clustering algorithm, and correct me if I'm wrong, I believe I should find a way to represent each word as a number using either tfidf or word2vec.

Do you have any ideas of how I could "strip" the textual information from e.g. a list, in order to do so and then place them back in order to make the appropriate multiplication?

For instance the way I see it if the word Minister has a tfidf weight of 0.042 and so on for any other word within the same topic I should be to compute something like:

0.041*0.42 + ... + 0.041*tfidf(Prime) and get a result that will be later on used in order to cluster the results.

Thank you for your time.

Assuming that LDA produced a list of topics and put a score against each topic for each document, you could represent the document and it's scores as a vector:

Document | Prime | Minister | Controversial | TopicN | ...
   1       0.041    0.042      0.041          ...
   2       0.052    0.011      0.042          ...

To get the scores for each document, you can run the document. as a bag of words, through a trained LDA model. From the gensim documentation:

>>> lda = LdaModel(corpus, num_topics=100)  # train model
>>> print(lda[doc_bow]) # get topic probability distribution for a document

Then, you could run the k-means on this matrix and it should group documents that are similar together.
K-means by default is a hard clustering algorithm implying that it classifies each document into one cluster. You could use soft clustering mechanisms that will give you a probability score that a document fits within a cluster - this is called fuzzy k-means.
 https://gist.github.com/mblondel/1451300 is a Python gist showing how you can do it with scikit learn.

ps: I cant post more than 2 links


Cascaded Error in Apache Storm Going through the presentation and material of Summingbird by Twitter, one of the reasons that is mentioned for using Storm and Hadoop clusters together in Summingbird is that processing through Storm results in cascading of error. In order to avoid this cascading of error and accumulation of it, Hadoop cluster is used to batch process the data and discard the Storm results after the same data is processed by Hadoop.

What is the reasons for generation of this accumulation of error? and why is it not present in Hadoop? Since I have not worked with Storm, I do not know the reasons for it. Is it because Storm uses some approximate algorithm to process the data in order to process them in real time? or is the cause something else?

Twitter uses Storm for real-time processing of data.  Problems can happen with real-time data.  Systems might go down.  Data might be inadvertently processed twice.  Network connections can be lost.  A lot can happen in a real-time system.

They use hadoop to reliably process historical data.  I don't know specifics, but for instance, getting solid information from aggregated logs is probably more reliable than attaching to the stream.

If they simply relied on Storm for everything - Storm would have problems due to the nature of providing real-time information at scale.  If they relied on hadoop for everything, there's a good deal of latency involved.  Combining the two with Summingbird is the next logical step.


Working with few instances of specific target feature over large dataset I have data over a single, a machine includes different components, all the parts are interacting, the data are tracked for those parts, it tracks power consumption and many other relevant feature measurements. The longevity of the machine, from the seller, is specified to be around 2000-15000 hours. Due to this large range, it would be ideal to constrain the target for machine failure, with machine learning. Because we have tracked all the data, with the different components in the system. One specific component fails.  Over a period of 9 months, the machine fails 5 times. Monthly average about 3-4 month. It is this, specific target we want to predict when the chances for failure are high.

Thus the specific instances for failure are limited. How do we deal with this, from a Machine Learning perspective, as opposed to giving a monthly average, it is not a safe bet? How do we train on limited instances for failure? This problem might not have one right answer, but many possible advice.

When you lack data you can use some theoretical knowledge on what should happen before the failure. For example for bearings a vibration in the frequency of the bearing should rise exponentially. The same goes for temperature, but there are too many variables that influence temperatures and it is not that easy to separate it. There are temperature sensors in some bearings. That should be more precise. Microphones can hear the problems like humans do. Ultrasound tools allow humans to hear the problems earlier.

You can read about some of these patterns in An introduction to predictive maintenance by R. Keith Mobley


Predicting financial data (choosing a model) it is my first time doing something with financial data. I have a dataset with account numbers and some other information about each client (some clients span more than one row since we have info for each month in a different row).
I managed to clean and create some models, here are the confusion matrices, the classification reports and AUC:
Logistic regression
[[185847  62897]
 [     1   1061]]

             precision    recall  f1-score   support

     not buy       1.00      0.75      0.86    248744
         buy       0.02      1.00      0.03      1062

    accuracy                           0.75    249806
   macro avg       0.51      0.87      0.44    249806
weighted avg       1.00      0.75      0.85    249806

AUC train = 0.9168592981611143
AUC test = 0.9150300677458543

Random Forest Classifier:
[[245503   3241]
 [   960    102]]

              precision    recall  f1-score   support

     not buy       1.00      0.99      0.99    248744
         buy       0.03      0.10      0.05      1062

    accuracy                           0.98    249806
   macro avg       0.51      0.54      0.52    249806
weighted avg       0.99      0.98      0.99    249806

AUC train = 0.9996866568080237
AUC test = 0.9139101966925902

Gradient Boosting Classifier:
[[184940  63804]
 [     3   1059]]

              precision    recall  f1-score   support

     not buy       1.00      0.74      0.85    248744
         buy       0.02      1.00      0.03      1062

    accuracy                           0.74    249806
   macro avg       0.51      0.87      0.44    249806
weighted avg       1.00      0.74      0.85    249806

AUC train = 0.8800353734759541
AUC test = 0.8657829269466372

Voting Classifier (from all the three above):
[[211316  37428]
 [   213    849]]

              precision    recall  f1-score   support

     not buy       1.00      0.85      0.92    248744
         buy       0.02      0.80      0.04      1062

    accuracy                           0.85    249806
   macro avg       0.51      0.82      0.48    249806
weighted avg       0.99      0.85      0.91    249806

AUC train = 0.9987531510931085
AUC test = 0.9160262741936392

Since I do not have any experience I am not sure which model is producing better results. Can you help me understand which one and why? Thank you!

Well, evaluating your model just by comparing matrixes can be pretty hard.
In your case, your matrixes are not done with the same number of row classified 1 or 0, so comparing them is really hard. Let's take an example : your logistic regression classifies around 64000 as 1, while your RandomForest only classifies around 4500, so comparing these data by this matrix is pretty hard.
I'd suggest you to use the ROC AUC metric, that is really useful to compare models. You might find many info on this subject in the Internet. The closer your AUC is than 1, the better your model is. If it's 0.5 or less, your model is less efficient than a random classifier.


Is GLM a statistical or machine learning model? I thought that generalized linear model (GLM) would be considered a statistical model, but a friend told me that some papers classify it as a machine learning technique. Which one is true (or more precise)? Any explanation would be appreciated.

A GLM is absolutely a statistical model, but statistical models and machine learning techniques are not mutually exclusive. In general, statistics is more concerned with inferring parameters, whereas in machine learning, prediction is the ultimate goal.


How should we manage large database We currently manage a large volume of economic data in Excel in my organisation. All of the data is downloaded from different online databases into Excel spreadsheets (one for each data frequency including annual, monthly, quarterly) - and then one main spreadsheet organises everything and creates tables that we need regularly. By organise, I mean that many of the things we need are simply identities ($Z=X+Y$ where we would have only downloaded data on $X$ and $Y$)

My view is that this could be done much more efficiently in R - where we'd automate the updating of the data and then spit out the tables that we need. But I am not trained at all in data management.

Would you all recommend a better way of doing this, or are there pitfalls to using R that I am not considering.

As mentioned in another answer, you should consider using a database. In the long run it will make your life easier.

But, R can help you automate things. You can store excel files in specific folders and/or with specific names (or patterns of names ex. file1_day1, file1_day2 etc) and then create R scripts that process them and produce the report you want.

Since you are using excel files (nothing personal with excel, it is a great program) I am inclined to think that your data will fit in a decent computer running R. In any case, if you end up using R, you should check out data.table package.

Furthermore, R can help you create more complicate reports. Plus, its learning path is not too steep.


Time series classification I am looking at time series security attack data where a given IP can either be labeled as (1) attack or (0) no attack.
In total we will have thousands of IPs and roughly an equal number of attacks and non attacks. The data is rather noisy and every time series sequence can have a different length.

I am looking for advise on state of the art approaches to time series classification. I am past the stage of simple things like moving averages and I am looking for ways to improve my current methods or new things to try.

I have currently implemented a few different techniques:

K-nearest neighbor with DTW. I am successfully using http://www.cs.ucr.edu/~eamonn/UCRsuite.html which provides state of the art performance.
Logical shapelets (http://www.cs.ucr.edu/~mueen/LogicalShapelet/). This seems promising but have not been able to get any existing code base to work.

Can anyone suggest different technique to try? I have seen papers about discords and motifs but still need to investigate if they are relevant for my problem.

I would suggest Recurrent Neural Nets. They are good for time series, however they need a huge dataset to get good performance. Here you can find an implementation in torch.


The implementation of overlapping clustering (NEO-K-Means) I am researching on overlapping clustering (Clusters are non-disjoint).I found that Neo-K-Means is probably the state-of-the-art now.But, when I tried  implementing the algorithm with the multi-label data set (music-emotion/scene).I hadn't got the high result as declared in the paper (My results are around 0.4 F-measure , the paper declare 0.55 for music and 0.626 for scene ). In spite of , I had initialized the experiments with the best situation for K-Means (centroid seeds are means of each class and the total cluster assignments are equal to reality) . I wonder what wrong with my implementation method , orDo I have to do something extra for getting higher result ?

PS. I have found the further research of Neo-Kmeans , but I think I should clear this point before go further.

This is my code

while (count < TIMES) {

            DC = new ArrayList<DistanceCollection>();

            for (int i = 0; i < K; i++) {

                cluster[i] = new Cluster();

            }

            for (int i = 0; i < dataList.size(); i++) {

                for (int j = 0; j < K; j++) {

                    DistanceCollection dist = new DistanceCollection();
                    dist.dataNum = dataList.get(i).dataNum;
                    dist.clusterNum = j;
                    dist.distanceFromCluster = euclidean(centroids[j], dataList.get(i));
                    DC.add(dist);

                }

            }

            // sort the distances for argmin(i,j) checking
            Collections.sort(DC, new DistanceCollectionComparator());

            int totalAssignment = 2585;
            int assignedCluster = -1;
            int assignmentCount = 0;
            int[] dataSelectionCheck = new int[2407];
            int dataMatrix[][] = new int[6][2407];

            for (int i = 0;  assignmentCount < dataList.size(); i++) {

                int clusterNum = DC.get(i).clusterNum;
                int dataNum = DC.get(i).dataNum;

                if (dataMatrix[clusterNum][dataNum] == 0 && dataSelectionCheck[dataNum] == 0) {

                    cluster[clusterNum].dataMembers.add(dataList.get(dataNum));
                    dataMatrix[clusterNum][dataNum] = 1;
                    dataSelectionCheck[dataNum] = 1;

                    assignmentCount++;

                }

            }

            for (int i = 0; assignmentCount < totalAssignment; i++) {

                int clusterNum = DC.get(i).clusterNum;
                int dataNum = DC.get(i).dataNum;

                if (dataMatrix[clusterNum][dataNum] == 0) {

                    cluster[clusterNum].dataMembers.add(dataList.get(dataNum));
                    dataMatrix[clusterNum][dataNum] = 1;

                    assignmentCount++;

                }

            }

            for (int i = 0; i < K; i++) {

                if (cluster[i].dataMembers.size() > 0) {

                    for (int j = 0; j < centroids[i].features.length ; j++) {

                        double accumFeaturesValue = 0;

                        for (int k = 0; k < cluster[i].dataMembers.size(); k++) {

                            accumFeaturesValue += cluster[i].dataMembers.get(k).features[j];

                        }

                        centroids[i].features[j] = accumFeaturesValue / cluster[i].dataMembers.size();

                    }
                }

            }

            count++ ;

        }

I have never heard of NEO-k-means. So I wouldn't call it "state of the art": which tools does include it?

Plenty of methods are published every week and have zero impact because they are redundant, badly written, only work on a single data set, hard to parameterize, or outright irreproducible (As I have not read NEO-k-Means, I'm not claiming any of this applies to this particular algorithm!)

Nevertheless, it is not the first to do non-overlapping clustering.

Fuzzy c-means (FCM): a soft-asssignment k-means variant
EM (Gaussian Mixture Modeling) is even more powerful, and has a sound theorertical backing
Subspace and correlation clusterings are usually overlapping, too.

I'd start with GMM and FCM, as these have been successfully used by many people.


Equivariance vs Invariance in Convolutional Neural Networks Could someone please explain to me in details (possibly from mathematical point of view) what is the role of Equivariance and Invariance in Convolutional Neural Networks, and how are they actually achieved? I found a similar question here but I did not get the full picture.

Several answers have been provided to the role of equivariance or invariance question "What is the difference between “equivariant to translation” and “invariant to translation”.
Depending on how local they are, scalar and Convolution operators tend to be equivariant, max/min or range are more invariant, and subsampling/pooling can somewhat link those behavior.