Is Clustering used in real world systems/products involving large amounts of data? How are the nuances taken care of? I am working with a real dataset of around 2.5 million short text data (posts, around 10 words in an average data element). I wanted to group similar posts together. So, I applied KMeans clustering on the dataset after doing the standard pre processing. After vectorization (I did TfIdf), the matrix was huge (2.5 million rows and around 60 k columns). I did an SVD + LSA transformation which made it smaller (2.5 million x 150 ). I gave 200 as the number of clusters and applied K Means. I must say that the results are not very bad. There are clusters which have similar posts (not the semantics, but the words similar), but there are those clusters which are randomly placed too.

Because of the large size of the data, I am not able to effectively visualize the results or measure the goodness of the clusters. Though I tried implementing silhouette coefficient, elbow methods for finding the optimal number of clusters, I was getting continuous memory errors. I don't know if 150 is the right number to use during dimensionality reduction. Could you please suggest any other methods for solving these issues?

When I reached this point, I started doubting the effectiveness of clustering as a method. I am curious to know if anybody is using clustering in large production scale systems, if so, what kind of techniques do people use?

Please find the code below:

`

 tfidf_vectorizer = TfidfVectorizer(max_df=0.5, max_features=200000,
                                 min_df=0.001,decode_error='ignore', stop_words='english',strip_accents='ascii',use_idf=True, ngram_range=(1,2),tokenizer=tokenizeAndFilter,lowercase=True)`

`    tfidf_matrix = tfidf_vectorizer.fit_transform(tokens.values()) #fit the vectorizer to synopses

    svd = TruncatedSVD(150)
    normalizer = Normalizer(copy=False)
    lsa = make_pipeline(svd, normalizer)

    X = lsa.fit_transform(tfidf_matrix)
    feature_names = tfidf_vectorizer.get_feature_names()

    with open(path+'terms.csv','w') as f:
       for col in tfidf_matrix.nonzero()[1]:
          f.write(feature_names[col]+','+str( tfidf_matrix[0, col]))
          f.write('\n')

        sil_dict={}
        for n_clusters in range(50,500,10):
            km=KMeans(n_clusters=n_clusters,init='k-means++',max_iter=500,n_init=1)
        km.fit(X)
            label = km.labels_
            sil_coeff = silhouette_score(X, label, metric='cosine',n_jobs=10)
    #        print("For n_clusters={}, The Silhouette Coefficient is {}".format(n_cluster, sil_coeff))
        sil_dict[n_clusters]=sil_coeff
            with open('kmeans_model'+str(n_clusters)+'.pkl','wb') as f:
               cPickle.dump(km,f)

    original_centroids=svd.inverse_transform(km.cluster_centers_)
    order_centroids=original_centroids.argsort()[:,::-1]

    terms=tfidf_vectorizer.get_feature_names()
    clusters=collections.defaultdict(list)
    k=0
    for i in km.labels_:
        clusters[i].append(tokens.values()[k])
        k+=1
        for i in range(10):
            print i
            for j in order_centroids[i, :10]:
                  print(terms[j] )
        for key,valueList in clusters.items():
            with open(path+'\\'+OUTPUTFILE+'_'+str(key)+'.csv','w') as f:
                 for value in valueList:
                    f.write(value+'\n')

`

Clustering (mostly) does not apply to huge data.

Since you cannot automate result evaluation and validation (forget the 'internal' measures, they are useless) nor use the automatic results in any meaningful way (partially because clustering is just way too unreliable for automatic use).

Clustering requires you to study the result, learn, and then decide what to do next. You can't automate this.

It is explorative. A tool for you to study your data; not a tool for machines to automatically predict labels. Some clusters will be good, some will be bad. This is fine, you got some good clusters! Just don't rely on everything being clustered. There will always be data that does not belong to any cluster. And often, there will be clusters that just don't make any sense to you, or that are too trivial to be of use or interest.
Results just do not improve with data size.
Methods don't scale well.

Most clustering methods (including k-means) have worst case runtimes of O(n^2) or O(n^3). That is really problematic. But since they are analyzing an NP-hard problem, that is quite decent.

So I guess, you want to work on a sample.
RAM is big these days, and vectors are compact.

You can usually squeeze a billion vectors into RAM. So anything that - like k-means - works with numerical vectors (and not, say, videos and images) just fits into main memory. No need to go larger than that.


line search in gradient descent applied to a convex function I have been working on implementing a line search method for gradient descent where I made an assumption that at any given point on my surface of the loss function I can reach the minima by the single correct value of the learning rate $\eta$ which I should choose. I have been trying to find this learning rate using binary search but after the entire implementation, I came to realize that my assumption I made is wrong which means I cant directly reach my minima from any given point on the surface of the loss function for any given learning rate in a single step. Can I get a more intuitive explanation of why my initial assumption is wrong?

Edit:  my loss function is convex and has a large number of parameters I am trying to learn ( multidimensional)

This isn't a perfect example but hopefully the idea is clear.
If you start at a point just left of 0, small learning rates will never get it out of the local minimum at the left. But large enough learning rates to leave that local minimum will never let it settle at the global minimum at the right.

A decaying learning rate could work for such a point.


What does RMSE points about performance of a model in machine learning? I am working on Decision Tree algorithm and at the end I calculate RMSE value based on actual labels and predicted values (for regression). Now what I am having difficulty in is in understanding the significance of the RMSE value that I get. I know that the lower the RMSE better is the performance of the model but what RMSE value is considered reasonable low or high? Suppose my RMSE value comes as 20 so does that mean the performance of my model is 80% (means 80% of predictions are correct)?

There are multiple factors to consider, but the first thing to realize is that in regression, you don't want to think about whether an example is "correct" or "incorrect" but rather how close it was to the true target value.  Therefore you can ignore your original intuition about "80% of predictions are 'correct'."

Second remember that RMSE is in the same space as your target values.  So it is relative to the variance in your target values.  The benchmark of random guessing should get you an RMSE = standard_deviation.  So lower than this, your model is demonstrating some ability to learn; above that number, you haven't even learned to guess the mean correctly.

There isn't a cutoff for "my model is doing well" in RMSE space, just like with other metrics.  Everything is relative to a naive solution/benchmark or the state-of-the-art.


I found this question when trying to find a way to add my own precalculated, custom error bars (standard deviations) to a Seaborn barplot with grouped values. I finally managed to find a workaround... The idea is to duplicate the observations drawing from a normal distribution with the desired mean and standard deviation. Then, the built-in "ci" option in sns.barplot does the rest. Not the cleanest solution... but it does the trick (at least for small datasets). An example:

import numpy as np
import seaborn as sns
import pandas as pd
import matplotlib.pyplot as plt

dfBarPlot = pd.read_csv('my_precalculated_values.csv')
#duplicate observations to get good std bars
dfCopy = dfBarPlot.copy()
duplicates = 30 # increase this number to increase precision
for index, row in dfBarPlot.iterrows():
    for times in range(duplicates):
        new_row = row.copy()
        new_row['Y'] = np.random.normal(row['Y'],row['precomputed_std'])
        dfCopy = dfCopy.append(new_row, ignore_index=True)

# Now Seaborn does the rest
sns.set_style("whitegrid")
fig = sns.barplot(x='X',
                  y='Y',
                  hue='Cases',
                  ci='sd',
                  data=dfCopy)

plt.legend(loc='upper right')
sns.set(rc={'figure.figsize':(8,5)})
plt.show()


Categorical features preprocessing for clustering Can anyone tell suggest the best practice for clustering data with mixtured features (both with categorical and continuous). I am struggling with a problem; I realized that for all metrics algorithms it is necessary to scale continuous data before clustering, so I used sklearn.preprocessing.StandardScaler(). With my categorical features, I used onehotencoder transformation, but it's not clear if it's necessary also scale these onehotencoded vector's components regarding to the whole data set or just leave them as they are?

Using Gower distance for the clustering. You can find different article about this measure in this and this article.

  Gower Distance is a distance measure that can be used to calculate distance between two entity whose attribute has a mixed of categorical and numerical values.

  Gower distance is computed as the average of partial dissimilarities across individuals. Each partial dissimilarity (and thus Gower distance) ranges in [0 1].
  $$d(i,j) = \frac{1}{p}\sum_{i=1}^{p}d_{ij}^{(f)}$$

  Partial dissimilarities ($d_{ij}^{(f)}$) computation depend on the type of variable being evaluated. This implies that a particular standardization will be applied to each feature, and the distance between two individuals is the average of all feature-specific distances.

  For a numerical feature $f$, partial dissimilarity is the ratio between 1) absolute difference of observations $x_i$ and $x_j$ and 2) maximum range observed from all individuals: $d_{ij}^{(f)} = \frac{|x_i - x_j|}{|(\max_N(x) - \min_N(x))|}$ , $N$ being the number of individuals in the dataset.
  Partial dissimilarity computation for numerical features ($R_f$ = maximal range observed)
  For a qualitative feature $f$ partial dissimilarity equals 1 only if observations $y_i$ and $y_j$ have different value. Zero otherwise.


Timeseries of odds in race - how to pick a model Being new to AI/ML I'd like some pointers to where to begin.
I got data from horse races. Specifically, I got the odds
for each runner during the race - ca 5 times per second.

t1  r1   r2   r3 ...
1   5.25 2.04 3.25
2   5.10 2.50 2.75
...

I also know if the runner won/placed/lost
My goal is to be able to say that runnerx X will win/place
after ca 50-75% of the expected racetime with say 80% accuracy.

My problem is that I don't know how to model this situation.
I've seem tournament strategies - ie who out of two runner will win -
but here's more data - both in time and in participants

What model should I pay attention to?

/Björn

Odds are directly connected to prediction accuracy. 4 means that you loose your money in 4 cases and win in one case. This is 20% probability $\left(\frac1{4+1}\right)$. If the bet goes down to 0.25 then the probability is 80% $\left(\frac1{0.25+1}\right)$. This logic is true if the spread between betting for the horse and against the horse is low. Betting is a financial market where people with the best betting algorithms bet the most and actually decide what the bet should be. If there is a situation on the market where someone could earn money it will be closed in milliseconds by trading robots. This is called an Efficient Market Hypothesis. In your data you can actually check if the hypothesis is true for horse betting.

If you would like to use ML algorithms to try to predict the outcomes better:

There are several ways to interpret the data.

You predict the probability that each horse is a winner.
You predict a rank of each horse.
You predict time difference between horses and the winner
You predict time

Some algorithms could be better at predicting one problem or the other.

Possible algorithms:

Linear regression. I always start with this because it is easy to calculate and easy to interpret. You build separate models for each horse number. Can be used for all problem definition.
Logistic regression. Similar to regression, but can only be used for the first problem definition.
LGBoost and XGBoost. Most likely the best models for such data, because the data is structured (not an image, sound). Can be used for all problem definitions.
Neural networks. For example MLP (multilayer perceptron).

You can also calculate some additional features. For example, multiply some odds, or calculate inverse normal distribution from the odds.

I would also recommend to collect more data. For example weather data. Then it is good to know the horse. Some horses could be more popular than others and receive better odds. Also time of the year because the horses could have a training plan according to the time of the year.


How to determine whether a bad performance is caused by data quality? I'm using a set of features, says $X_1, X_2, ..., X_m $, to predict a target value $Y$, which is a continuous value from zero to one.

At first, I try to use a linear regression model to do the prediction, but it does not perform well. The root-mean-squared error is about 0.35, which is quite high for prediction of a value from 0 to 1.

Then, I have tried different models, e.g., decision-tree-based regression, random-forest-based regression, gradient boosting tree regression and etc. However, all of these models also do not perform well. (RMSE $\approx $0.35, there is not significant difference with linear regression)

I understand there are many possible reasons for this problem, such as: feature selection or choice of model, but maybe more fundamentally, the quality of data set is not good.

My question is: how can I examine whether it is caused by bad data quality?

BTW, for the size of data set, there are more than 10K data points, each of which associated with 105 features.

I have also tried to investigate importance of each feature by using decision-tree-based regression, it turns out that, only one feature (which should not be the most outstanding feature in my knowledge to this problem) have an importance of 0.2, while the rest of them only have an importance less than 0.1.

First, it sounds like your choice of model selection is a problem here.  Your outputs are binary-valued, not continuous. Specifically you may have a classification problem on your hands rather than a traditional regression problem.  My first recommendation would be to try a simple classification approach such as logistic regression or linear discriminant analysis.

Regarding your suspicions of bad data, what would bad data look like in this situation? Do you have reason to suspect that your $X$ values are noisy or that your $y$ values are mislabeled? It is also possible that there is not a strong relationship between any of your features and your targets.  Since your targets are binary, you should look at histograms of each of your features to get a rough sense of the class conditional distributions, i.e. $p(X_1|y=1)$ vs $p(X_1|y=0)$. In general though, you will need to be more specific about what "bad data" means to you.


How to determine sample rate of a time series dataset? I have a dataset of magnetometer sensor readings which looks like:
TimeStamp       X        Y         Z
1.59408E+12 -22.619999  28.8    -22.14
1.59408E+12 -22.5   29.039999   -22.08
1.59408E+12 -22.32  29.039999   -21.779999
1.59408E+12 -22.38  29.16   -21.6
.
.
.
And so on

The timestamp is in milliseconds where 1.59408E+12, 1.59408E+12, 1.59408E+12 is 1594076006983, 1594076006994, 1594076007004 and so on. So what will be the sampling rate/ frequency of the data?

General Explanation
Generally speaking, the frequency of the data is the difference between consecutive time-stamps.
If all is well, that difference will be constant across your time-series; in this case, this difference is the frequency of your data.
In other cases, it might be a bit more complicated - for example, you can have missing data, or data where some timestamps are shifted a bit forward or backward.
In which case, you might want to 'smooth out' those issues, for example by taking the median of such differences (if some samples come a bit too early and some come a bit too late), or the mode (if some samples are missing, meaning that most will come at the right time and some will be too late by an exact multiple of the real frequency).
Python (pandas) Example
Here's an example in Python, using the pandas library:
let's make up a time series with missing values. Its frequency is 5 minutes, but about 5% of the samples are missing (its data is random integers between 1 and 20, and the 1's have been dropped)
import pandas as pd
import random

index = pd.DatetimeIndex(start='2020-01-01 00:00:00', end='2020-04-01 00:00:00', freq='5T')
s = pd.Series(index=index, data=None)
s = s.apply(lambda x: random.randint(1,20))
s=s[s.ne(1)]

Now let's look at the time-difference between existing (not dropped) samples:
s.index.to_series().diff().value_counts()

yields
00:05:00    23629
00:10:00     1191
00:15:00       63
00:20:00        2

i.e. the vast majority of samples are really 5 minutes apart, but not all of them.
We'll take the actual frequency by taking the median of those differences:
s.index.to_series().diff().median()

yields
Timedelta('0 days 00:05:00')


Should highly correlated features be omitted before applying Lasso? I would greatly appreciate if you could let me know whether I should omit highly correlated features before using Lasso logistic regression (L1) to do feature selection.

In fact, I want to use logistic regression with L1 to do prediction as well as feature selection. However, some of my features are highly correlated e.g., -1 or 0.9. Should I omit them before applying Lasso or let the Lasso decide it?

Really, I read in Mr. Raschka’s book (Python Machine Learning) that

  regularization is a very useful method to handle collinearity (high
  correlation among features).

However, this kernel  (by referring to Wikipedia) states that keeping correlated features in the model would adversely affect the feature selection but it doesn't impair the predictions.

Use scikit-learn package.
In your case you need find sklearn.linear_model.LogisticRegression

and User guide

It's clear enough for understanding. You needn't special actions to win collinearity. But instead of linear method you can use non parametric algorithm like random forest sklearn.ensemble.RandomForestClassifier.

Сompare the results of the logistic regression & random forest on the test data


Continuously training one model with different dimensions each time I'm trying to solve a problem where I need to train one model with N dimensions and again train on top of that model with M dimensions. How can I achieve it?

To give you guys some context, I have 1 bluetooth beacon talking to 5 bluetooth readers that give out some value. Based on the value given out by the reader, I'm predicting the location of the beacon. My training data has 5 dimensions because there are values from 5 readers.

Now I have another scenario where I have 10 readers instead of 5. How do I train on top of my first model with 10 readers this time? Is it even possible?

The number of readers varies depending on the size of the room. So there's no way I can tell that there will always be a fixed number of readers (that is, fixed number of dimensions).

How do I go about solving this problem? Any help is appreciated!

You could try to summarise the data before handling it to the model:

If you have 5 readers, you could gather the information of "mean", "max", "min", "standard deviation" of them, this will work on 10, 15, any N of readers. Unless there is something which stops you from calculating mean, max, etc.


What is word embedding and character embedding ? Why words are represented in vector with huge size? In NLP word embedding represent word as number but after reading many blog i found that word are represent as vectors ? so what is word embedding exactly and  Why words are represented in vector and the vector size is huge. what values dose that vector represent ?

what is the difference between word and character embedding ? please explain in simple term with simple example.

Most problems in NLP require the system to understand the semantic meaning of the text and not just the arrangement of specific words.

  Semantic understanding enables a system to say that,  "I am happy" and "It's joyful", have the same meaning.

To incorporate this feature to a system, we present words of a particular language in form of vectors. Often called as embeddings, they help in establishing similarities between words and phrases.

  For instance, a vector representing the word "happy" will lie in the vicinity of the vectors representing the words "joy", "pleasure", "sad" etc. These vectors are high dimensional but using PCA or other dimensionality reduction techniques they are brought down to 3 dimensions where they could be visualized.

That's why we encode words in the form of vectors. We often use cosine similarity to determine the closest vector to a given vector in analysing sematic similarity.

  For an intuition, the 3D space which contains vectors for all possible English words could be thought of our knowledge base. We tend to keep similar words together in our mind. If we are talking about fast food, for instance, our brain would capture the region of the knowledge base to retreive words related with fast food like "burgers", "chicken" etc.


What Regresson Algorithm Would Fit This Data? I am using regression on lottery numbers, below is the head of my database. I was wondering if anyone had a suggestions on what the best algorithm would be to use for the regression model.

Data Head:

      Date  b1  b2  b3
1984-09-01   9   4   8
1984-09-03   1   5   3
1984-09-04   8   6   6
1984-09-05   5   6   9
1984-09-06   2   1   9

This answer is based on additional information provided in a comment:

  what is x, y?
  Balls 1,2,3 = Y, Date = X

Lotteries and other games (such as Casino games) are designed so that one is (or should be) unable to predict what happens. For instance in roulette, you could win a lot of money by applying the St Petersburg paradox. However, casinos restrict the sequence of playing so that you have no chance to do this.

WRT lotteries, the data generating process is supposed to be 100% random. This means that your X (the date) does not carry structural information about the outcome (y). Consequently, there is no way to make a reasonable prediction of what the numbers of next week would be.


Feature Importance - How to choose the number of best features? What is the standard or what method do you use to select part of features.

For ex: Using random forest, I got the following feature importances:

a : 25.4884726
b : 17.2736393
c : 12.3493490
d : 8.9383737
e : 8.1083837
f : 6.8272717
g : 4.1999999
...
...
...

For final prediction, you need to select only few features from the above. How do you decide the following:

  No. of features that you choose?
  Does the score matter when you choose?

This is an important problem, since many feature selection methods return feature scores/importances rather than a finite feature set. I currently know three approaches:

choose the k best attributes (fixed number defined)
choose the best k * 100% of attributes (relative number of features defined)
make a cutoff at the biggest difference in feature scores: all features are ordered according to their score and a split is made at the largest difference between one score and the next lower (biggest loss in importance)

You find a nice implementation doing this in R with the FSelector package.

I think, there are some research works tackling exactly this problem and may suggest better approaches, but I hadn't had the time to go deeper into this.


Encoding continuous values I've been reading about one-hot encoding for categorical values. Could similar mechanism make sense for continuous values? I'm looking at a problem, where I try to predict the volume of product X, with a variables being the price of product Y.

The reason is to increase the strength of the signal from the price of Y, which otherwise could be understated because overall prices fluctuate in relatively low range. The idea would be to e.g. remove attribute of "Price Y" and replace it with "Promotion Y" based on a set threshold.

E.g.

Price X | Price Y | Vol X | Vol Y  | Promotion X  | Promotion Y
100     | 90      | 40    | 50     | 0            |  0
75      | 75      | 60    | 60     | 1            |  1
100     | 75      | 30    | 70     | 0            |  0

I could just test it out on a dataset, but this could be very specific to this very set and I just want some more theoretical suggestion if and where such approaches would make sense?

No. There is no similar mechanism for continuous variable.

If it worries you, that overall prices fluctuate in relatively low range,
you can

1) demean the price, that is subtract mean price from all price values. Then negative values will clearly show below-average and positive above-average prices.

2) after demeaning you can divide values by the standard deviation of the price. (1) and (2) together is called "standardization".

Alternatively you can

3) rescale your price to the range of values you want. Usual choice is (0,1) range.

If you do this for one feature (price in your case), then it makes sense to do the same for other features.

Whether or not this will help to get better prediction results depends on the model. Some models, a typical example would be SVM, do require such transformation.


What is average unpooling? I'm working on an implementation of a paper and I haven't been able to find a description of average unpooling. I tried using max unpooling instead and my results were not great.

Thanks!

When you go backward for max-pooling you keep track of the position of the maximum:

$$ X = \begin{bmatrix}
1 && 2 \\
3 && 4
\end{bmatrix} \quad \rightarrow \quad dZ *\begin{bmatrix}
0 && 0 \\
0 && 1
\end{bmatrix}$$

This gives you the position of the input value that ultimately influenced the cost/output. The gradient will then be "propagated" back to this value.

For average pooling, all values influence the cost equally and therefore it should look like this:

$$ dZ = z \quad \rightarrow  \quad \begin{bmatrix}
z/n && z/n \\
z/n && z/n
\end{bmatrix}$$

where $n$ depends on the filter size, here $n=4$.

The gradient gets distributed evenly back to all input values.


Automatic legal document reviewing advice I am lawyer from SK. I also like programming and Maths. Which technology should i use for this task? Automatic legal document reviewing. I have a big data set of specific contracts in slovak. Now I want to build my model. Model will tell good/bad contract + bold probable errors based on data set. I don´t know much about NLP, so I´am just asking for advice. Which tech should I study for this task? Word2Vec, keywords extraction algos etc. Thank you guys for your time.

I am not a lawyer but I assume a contract is good if it is comprehensive and covers certain aspects extensively. An "easy" solution would be the following:

You could probably train a model which could classify a contract to which area it belongs (aka topic classification). If the probability of belonging with the rest of its the peer contracts is high then it could be a good contract.

Then again I am not sure how effective this would be and what pre-processing steps should be taken and would certainly need a big amount of labelled contracts. I would encourage you to dive into NLP and check the various NLP tasks that may leverage the process with your domain expertise.

Lastly, you could ask for help from a company like Lawgeex to make the NLP (or LLP I think they call it for Law Language abbreviation) stuff for you. But then again, not sure if companies like this could fit the Slovakian market. It would be cool for them to have Language Model that could fine tune in other languages!


What could explain a much higher F1 score in comparision to accuracy score? I am building a binary classifier, which classifies numerical data, using Keras.

I have 6992 datapoints in my dataset. Test set is 30% of the data. And validation set is 30% of the training set.

When evaluating the model, I get these values:

recall:  0.8914240755310779
precision:  0.7006802721088435
f1_score:  0.7846260387811634
accuracy_score:  0.7035271816800843

How come is the accuracy_score so about 10% lower than the F1-score?

Here is the code I'm using to evaluate the model:

print('recall: ', recall_score(Y_test, y_pred))
print('precision: ', precision_score(Y_test, y_pred))
print('f1_score: ', f1_score(Y_test, y_pred))
print('accuracy_score: ', model.score(X_test, Y_test, verbose=0))

And here is my model:

def create_model(neurons=23):

    model = Sequential()
    model.add(Dense(neurons, input_dim=37, kernel_initializer='normal', activation='relu'))
    model.add(Dense(1, kernel_initializer='normal', activation='sigmoid'))

    # Compile model
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy', precision, recall])

    return model

model = KerasClassifier(build_fn=create_model, epochs=500, batch_size=5, verbose=1)

X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.3, random_state=seed)

transformer = Normalizer().fit(X_train)
X_train = transformer.transform(X_train)
transformer = Normalizer().fit(X_test)
X_test = transformer.transform(X_test)

tensorboard = TensorBoard(log_dir="logs/{}".format(time.time()))
time_callback = TimeHistory()
es = EarlyStopping(monitor='val_acc', min_delta=0, patience=20, verbose=0, mode='auto',restore_best_weights=True)

# Fit the model
history = model.fit(X_train, Y_train, validation_split=0.3, epochs=200, batch_size=5, verbose=1, callbacks=[tensorboard, time_callback])

You have imbalanced classes. Notice that your accuracy is very close to your precision, and quite dissimilar to your recall. This means that your precision (accuracy of positive predictions) is dominating the overall accuracy measure - nearly all of the cases in your data are classified as positive, so the accuracy among predicted positives is almost equivalent to the accuracy among all cases.

The F1 score is the harmonic mean of precision and recall, so it's a class-balanced accuracy measure. You have better performance on the minority class than the majority class, which is evidenced by the nearly equivalent accuracy and precision, and much higher recall.


Python package for machine-learning aided data labelling In a lot of cases unlabelled data needs to be transformed to labelled data. The best solution is to use (multiple) human classifiers. However, going to all the data by hand (i.e. in text-mining or image-processing) is often a daunting task. Is there software that can combine human classifiers and machine-learning techniques in real time? I am especially interested in python packages.

To illustrate, classifying images from video streams is very repetitive. After 100 images (from different streams) a machine-learning algorithm could be used to predict the labels given by the human classifier. The machine classifier might be very confident about some (un)seen samples and very uncertain about others. The human classifier can then focus on the uncertain samples helping the machine classifier to learn better what is does not yet know.

It sounds like you are looking for active learning.  In active learning, the classifier learns which samples would be most useful to have labelled by a human.

There are many techniques for active learning, and many ways to adapt an existing (standard) learning algorithm to the active learning setting.  The particular approach you mentioned is called "uncertainty sampling", and can be applied to any standard classifier that outputs confidence/certainty scores.  There are other selection methods as well, which may perform better in some settings.

You can also apply unsupervised methods to cluster the samples, then label one or a few samples from each cluster.


Basic classification question I am wondering how I can manage a test data after using PCA or normalization and another thing like that in the classification because our model works on the representation given by its input vectors. For example, suppose you have used PCA in your training dataset to gain better accuracy or you have normalized (min-max) data. Now, you have developed a model and want to install it and label the new coming samples. You need to somehow apply PCA to each coming record and normalize that record. Applying PCA to one record will not yield the same effect of the PCA  used in training phase and I think even it doesn't make sense to apply PCA to just one sample. So how can we manage these preprocessing techniques in the training phase in the test data, too?

Thanks in advance.

PCA is a matrix transformation from your original dataset to a set of orthogonal features. The transformation matrix which is applied to the training set is maintained and used in the future with your testing data such that the original testing set features will be mapped to same space as the training set transformed by PCA.

If the training set has $n$ instances and their are $m$ features, the training matrix is of size $n \times m$. The PCA transformation matrix is of dimensions $m \times k$, where $k$ is the number of retained PCA features, the top eigenvalues. Thus we can transform a single instance $1 \times m$, by the $m \times k$ transformation matrix. This results in a $1 \times k$ vector.

I have some text files that I vectorized using bag-of-words. The training set is shown below on the left side and the testing set is on the right side. Each row is a text file and the columns is the word count.

If we plot the first 2 features of this dataset we get

Now we will fit our PCA transformation matrix and we will apply this transformation to both the training and testing set.

from sklearn.decomposition import PCA
pca = PCA(n_components=2, copy=True)
pca.fit(X_train)
train_PCA = pca.transform(X_train)
test_PCA = pca.transform(X_test)

This gives the following plot. The purple and yellow points are the 2 different classes from the training set. Then the light blue points are from the testing set. You can see that the points in the testing set after being transformed by PCA will line up alongside the training set.


There is not enough information to compute forward the number of parameters of each layer, as we don't know the number of input channels to conv2d_7 or its kernel width. Nevertheless, we can try to guess the missing pieces of information with reasonable assumptions from the number of parameters in each layer, which maybe helps you understand the number of parameter computations:

conv2d_7 has an output shape with 64 channels. This means that the filter tensor is $(c \times w \times w + 1) \times 64$ (the "+ 1" is for the bias), where $c$ is the number of input channels and $w$ the kernel size. The number of parameters in this layer is 640, which means that $w \times w \times c + 1= 10$. I would guess that $c=1$ and $w = 3$.
max_pooling2d_7, flatten_7 and dropout_2 don't have trainable parameters (= 0)
dense_14 has 1384576 parameters and, from the output shape, we know that the projection matrix is $? \times 128$. The previous layer has 10816 as output dimensionality. From this, we know that dense_14 has a projection matrix of dimensionality $10816 \times 128$ and a bias vector of 128 dimensions (totaling 10816 * 128 + 128 = 1384576)
dense_15 has 1290 parameters and, from the output shape, we know that the projection matrix is $? \times 10$. The previous layer has 128 as output dimensionality. From this, we know that dense_15 has a projection matrix of dimensionality $128 \times 10$ and a bias vector of 10 dimensions (totaling 128 * 10 + 10 = 1290).

Adding up the three computed number of parameters gives the total: $1384576 + 640 + 1290=1386506$.


How can we use mean imputation without violating feature correlation? Mean imputation is generally bad practice because it doesn’t take into account feature correlation. Imagine we have a table showing age and fitness score and imagine that an eighty-year-old has a missing fitness score. If we took the average fitness score from an age range of 15 to 80, then the eighty-year-old will appear to have a much higher fitness score that he actually should. Thus, I wonder if we have any way to use mean imputation without violating feature correlation.
For me, a straightforward solution is to replace mean of the whole population in data by mean of a group of similar subjects. For example, in the above example, we can fill missing score of a person by mean score of other persons of the same age, or for more flexibility, in the same range of age. This approach still works if we have more than 2 features and still want to take into account correlation among all features. We just need to define a similarity metrics between subjects based on all features, e.g. cosine similarity. Of course, similarity computation is expensive, that is a limitation of this approach. So I wonder if there is any better alternative.

If you want to keep the feature correlation, use an imputer that solves for the missing values by using the feature correlation, such as KNN, regression, etc. These methods work well if there is no underlying reason that the data is missing - Missing at Random.


Deep Q-Learning for physical quantity: q-values distribution not as expected Setting

I am trying to learn a specific physical quantity (radiance) inside a 3D scene with Deep Q-Learning. Just to give a quick overview, my agent shoots rays inside the scene: the reward is the irradiance of the points hit. This means that the reward is given only when a light source is hit - only 1% of the times. This lead to a very sparse reward function.

My state is a tuple of the 3D spatial coordinates inside the scene, my actions are the possible discrete directions used by the agent to scatter rays. The q-values represent this physical quantity based on that specific action(/direction).

Problem

I expect the q-values to be higher in the first 10 actions, and then slightly decrease. This would reflect the physics of my system. When the training starts, this is actually the case:

After some episodes, the q-value of one action starts to spike, as seen in the figure below. This does not reflect the physics of the environment, for which the incoming radiance should be distributed over all the actions.

EDIT:
The trend lines for the q-values can be seen below. Each line is the q-value for a specific action. During the last iterations the q-values for two specific actions explode.

I know about target networks and experience replay (uniform and PER), but this did not solve my issue.

Are there any methods that I can try to deal with this problem? Is this a problem of vanishing/exploding gradient? Thanks

One guess is is that network isn't lucky enough to encounter (sufficient number of times) an example that leads to the reward.  The issue is mentioned here pointing out that RL-agent might favor higher though incorrect estimates too early during learning. Link hypotheses that perhaps "step-annealing"  could be better, instead of gradualy annealing the epsilon-greedy parameter.

In other words, every time you reduce the epsilon-greedy parameter, you reduce the network's chance to take a random action and actually encounter your (already rare by default) reward. Consider reducing the epsion-greedy parameter further in time, or maybe use a step function to anneal it.

There is also Hindsight experience replay. It tells you to concatenate to your input-vector the state in which you wish to ultimately end up (best-case scenario), and feed it into the network, for forward prop.

After it fails to encounter your reward, you take the state where the network ended up, and backpropagate as if that state was what you actually aimed for originally.

According to the above link you can also give it part of the state vector, if the exact destination state is unknown.

HER teaches your network to achieve smaller goals instead of "fixating" on the best-possible goal. Unlike the traditional approach "don't care how, just give me reward", HER describes how the final state looks like. Now, the network is able to learn how states relate to each other, and ultimately how they relate to the destination state the network was meant to go. (instead of mere "I failed" or "I succeeded")

Traditionally, your network won't get a chance to form an impression of the  environment if encountered rewards are zero. With HER, you get such a chance even if the end goal is not visible on the horizon and rewards would be zero or uniform.

Here is a great blog about HER

Video of Hindsight experience replay


Adjusting Probabilities When Using Alternative Cutoffs For Classification I am reading Applied Predictive Modeling by Max Khun. I chapter 16 he discusses using alternate cutoffs as a remedy for class imbalance.

Suppose our model predicts the most likely outcome of 2 events, e1 and e2. We have e1 occurring with a predicted probability 0.52 and e2 with a predicted probability 0.48. Using the standard 0.5 for e1 cutoff we would predict e1, but using an alternative cutoff of 0.56 for e1 we would predict e2 because we only predict e1 when p(e1) > 0.56.

My question is, does it make sense to also readjust the probabilities when using alternate cutoffs. For example, in my previous example using 0.56 cutoff of e1.

p(e1) = 0.52;
p(e2) = 0.48

Then we apply an adjustment of 0.56 - 0.5 = 0.06.

So

p_adj(e1) =  0.52 - 0.06 = 0.46;
p_adj(e2) = 0.48 + 0.06 = 0.54

Basically we shift the probabilities so that they predict e1 when p_adj(e1) > 0.5.

I apologize if there is something obviously flawed with my logic but it feels intuitively wrong to me to predict e2 when p(e1) > p(e2). Which probabilities would be more in line with the real-world probabilities?

First of all, you cannot always consider what a machine learning algorithm outputs as a "probability". Logistic regression outputs a sigmoid activation on a (0, 1) scale, but that doesn't magically make it so!

We simply often scale things to a (0, 1) scale in ML as a measure of confidence.

Also in your example, if the events are mutually exclusive (like classification), just think of them as "event 1" and "NOT event 1". Something like p(e1) + p(~e1) = 1.

So when your book tells you to lower the threshold, it is simply saying that you require a smaller level of confidence to choose e1 over e2. This doesn't mean you are choosing one with smaller likelihood, you are simply making a conscious choice to adjust your precision-recall curve.

There are other ways to combat class imbalance, but changing the threshold to be more sensitive to any indication of confidence of one class over another is certainly a way to do that.


How to reverse ReLU activation in deconvolution I recently came across Matt Zeiler's deconvolution (reversing convolution) paper . How is deconvolution able to reverse the rectified scalar output? From what I understand it sounds analogous to reversing the output of a linear / logistic regression model. How is it possible to go from a scalar to a vector?

It's not reversing the ReLU, but applying normal ReLU activation again after max pooling and max unpooling.

It is used again

  "to obtain valid feature reconstructions at each layer (which also should be positive)"


LSTM: many to one and many to many in time-series prediction I am trying to predict the trajectory of an object over time using LSTM. I have three different configurations of training and predicting values in my mind and I would like to know what the best solution to this problem might be (I would also appreciate insights regarding these approaches).

1) Many to one (loss is the MSE of a single value)

The $input$ is a sequence of $n$ values, the output is the prediction of the single value at position $n+1$.
The loss function is the MSE of the predicted value and its real value (so, corresponding to the value in position $n+1$).
During the online test, a sequence of $n$ values predict one value ($n+1$), and this value is concatenated to the previous sequence in order to predict the next value ($n+2$) etc.. This way, a whole trajectory of $n + t$ values is calculated.

2) Many to one (loss is MSE of multiple values)

The $input$ is a sequence of $n$ values, the output is the prediction of the single value at position $n+1$.
To compute the loss function, the same strategy used before for online test is applied. LSTM predicts one value, this value is concatenated and used to predict the successive value $t$ times. The loss is the MSE of all the predicted values in the trajectory and their real values. Backpropagation is only done when the whole trajectory has been predicted.
Online testing is equal to the previous situation.

3) Many to many

The $input$ is a sequence of $n$ values, the output is the prediction of $m$ consecutive values.
The loss function is the MSE of the $m$ predictions and their corresponding ground truth.

The choice is mostly about your specific task: what do you need/want to do?

Many-to-one (single values) models have lower error, on average, since the quality of outputs decreases the more further in time you're trying to predict. Many-to-one (multiple values) sometimes is required by the task though.

An alternative could be to employ a Many-to-one (single values) as a (multiple values) version: you train a model as (single), then you use it iteratively to predict multiple steps. I personally experimented with all these architectures, and I have to say this doesn't always improves performance.

Based on my experience, Many-to-many models have better performances. For example, I had to implement a very large time series forecasting model (with 2 steps ahead prediction). The best model was returning the same input sequence, but shifted forward in time of two steps. It appeared that the model was better at keeping the predicted values more coherent with previous input values. It was a seq2seq RNN with LSTM layers.

EDIT:

The Loss doesn't strictly depend on the version, each of the Losses discussed could be applied to any of the architectures mentioned. A problem for multiple outputs would be that your model assigns the same importance to all the steps in prediction. This is something you can fix with a custom MSE Loss, in which predictions far away in the future get discounted by some factor in the 0-1 range. In that way your model would attribute greater importance to short-range accuracy. Let me know if that's helpful.


what does this doc2vec based ML predict? I'm trying to understand what does this ML program - which based on doc2vec - predict:

import logging, gensim
from gensim.models.doc2vec import TaggedDocument
from gensim.models import Doc2Vec
import re
import os
import random
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
import numpy as np

model = Doc2Vec.load('reviews_model.d2v') # Already trained

sent = []
answer = []
docvec = []

for fname in ['yelp', 'amazon_cells','imdb' ]:
    with open ('sentiment labelled sentences/%s_labelled.txt'% fname, encoding = ('UTF-8')) as f:
        for i , line in enumerate(f):
            line_split = line.strip().split('\t')
            sent.append(line_split[0])
            words = extract_word(line_split[0])
            answer.append(int(line_split[1]))
            docvec.append(model.infer_vector(words, steps=10))
            print (str(docvec) +  'time')

combined = list(zip(sent, docvec, answer))
random.shuffle(combined)
sent , docvec, answer= zip(*combined)

 clf = KNeighborsClassifier(n_neighbors=9)
 score = cross_val_score(clf, docvec, answer, cv =5)

 print (str(np.mean(score)) + str(np.std(score)) )

The output be something like:

0.7903333333333334

So what does it actually mean it's 79% correct? correct of predicting what exactly?

P.S: the documents learned are positive and negative reviews.

The code is trying to predict sentiment of documents (Amazon reviews or yelp reviews).

Kneighbors classifier works with vectors for input. A pre-trained doc2vec model is used to convert each review to a vector.
The classifier is trained to predict sentiment label from vector representing each review.

Accuracy is mean accuracy on the dataset: percentage of reviews which were correctly classified (as positive or negative).


How can I calculate Kernel matrix K for clustering based Kernel Principal Component Analysis? In practice, a large data set leads to a large K, and storing K may become a problem. One way to deal with this is to perform clustering on the dataset, and populate the kernel with the means of those clusters for speeding up and reducing storage.
How can I calculate the kernel using clustering?

Foremost, we must understand what is clustering.

It is an unsupervised algorithm, applicable to scenario's where the target class is unknown. So essentially it's a data preprocessing algorithm. Continuing further, cluster analysis, helps in detecting patterns in data. Now, to detect patterns in data, one must ensure the sample consist of statistically significant variables. To find such significant variables, we need to perform various data preprocessing tasks like outlier, missing value('s), correlation and dimensionality reduction detection and treatment. Post, performing these steps, the noise from the data would be removed and one would be able to reveal the signal. Otherwise, one can perform the principal component analysis (pca) to detect the signal. This signal('s), can then aid in determining the true patterns or clusters in the data.

Now, coming to the second part of the question. You can use the kernlab package in R to calculate the kernel. Perhaps these posts can help you further, 1, 2, 3, 4.


Why are policy gradient methods preferred over value function approximation in continuous action domains? In value-function approximation, in particular, in deep Q-learning, I understand that we first predict the Q values for each action. However, when there are many actions, this task is not easy.

But in policy iteration we also have to output a softmax vector related to each action. So I don't understand how this can be used to work with continuous action space.

Why are policy gradient methods preferred over value function approximation in continuous action domains?

But in policy iteration also we are have to output a softmax vector related to each actions

This is not strictly true. A softmax vector is one possible way to represent a policy, and works for discrete action spaces. The difference between policy gradient and value function approaches here is in how you use the output. For a value function you would find the maximum output, and choose that (perhaps $\epsilon$-greedily), and it should be an estimate of the value of taking that action. For a policy function, you would use the output as probability to choose each action, and you do not know the value of taking that action.

  So I don't understand how this can use to work with continuous action space ?

With policy gradient methods, the policy can be any function of your parameters $\theta$ which:

Outputs a probability distribution
Can be differentiated with respect to $\theta$

So for instance your policy function can be

$$\pi_{\theta}(s) = \mathcal{N}(\mu(s,\theta), \sigma(s,\theta))$$

where $\mu$ and $\sigma$ can be functions you implement with e.g. a neural network. The output of the network is a description of the Normal distribution for the action value $a$ given a state value $s$. The policy requires you to sample from the normal distribution defined by those values (the NN doesn't do that sampling, you typically have to add that in code).

  Why are policy gradient methods preferred over value function approximation in continuous action domains?

Whilst it is still possible to estimate the value of a state/action pair in a continuous action space, this does not help you choose an action. Consider how you might implement an $\epsilon$-greedy policy using action value approximation: It would require performing an optimisation over the action space for each and every action choice, in order to find the estimated optimal action. This is possible, but likely to be very slow/inefficient (also there is a risk of finding local maximum).

Working directly with policies that emit probability distributions can avoid this problem, provided those distributions are easy to sample from. Hence you will often see things like policies that control parameters of the Normal distribution or similar, because it is known how to easily sample from those distributions.


Standard correlation coefficient of various datasets I understood the correlation coefficient of the first line. But the correlation coefficient of second and third line differ with the first line in figure 1. Why is it so ?. Even the shapes differ for second and third line ?. Can someone help me to gain an understanding of below picture

  The second question is related to the correlation between attributes. The intention behind asking this question it to gain more understanding from StackOverflow data science users. The very one thing I learnt from the below picture is

if you plot attribute against the same attribute, you will end up with lines, which is of no use. For instance, if you notice median_house_value in y-axis against median_house_value in the x-axis, they are basically histograms.

Is there any valuable feedback that you can comment on the below picture ?. If so, please let me know.

First part:
The question you have to ask yourself is "Given I know the x-value, what knowledge do I gain about the y-value". And a small side note: you may only consider linear relationships.
Consider the first row. In the first image, if you know the x-value, then the y-value is y = x, i.e. you know the y-value exactly. In the following two images, if the x-value is high, then the y-value is high too, but there is a random component too. Thus, the correlation coefficient is positive (i.e. knowing x helps in estimating y), but not 1 (i.e. there is no exact equation y = a*x. In the middle image, there is no relationship between the x- and y-coordinate - they are purely random, so no correlation. In the right three images, it is the same story as in the left three images, but the sign is flipped.
Now consider the second row: in the first three cases, given you know the x-value, you can always infer the y-value, e.g. y = x, y = 0.5 x, and y = 0.1 x. There is no random component in y, so if you know x, you also know the exact value of y, thus the correlation coefficient is 1. It doesn't matter if the formula is y = 1 x or y = 0.1 x (i.e. the slope of the line doesn't matter!) all that matters for the correlation coefficient is that there is such a linear coefficient which leads to an exact match.
In the third row, knowing x does give some knowledge on y. For example in the middle plot, you have a relationship of y = x^2 + random, so there is indeed some relationship between x and y, but it is non-linear. Thus the correlation coefficient is zero. There is no way for you to say "a high x-value also leads to a high y-value" or "a high x-value leads to a low y-value".

Second part:
I assume this plot is generated e.g. with Seaborn's pairplot() function, though of course different functions exist for creating this kind of plot. It is important to note that the plots on the diagonal are fundamentally different plots than the other elements.
The off-diagonal plots are two-dimensional scatter plots, i.e. you draw a point for each data sample.
The diagonal plots are histograms of each feature, and not scatter plots. This is because a scatter plot of twice the same variable would always give you a straight line y = x.
The histograms are actually quite useful as you can get a feeling for the data and make a guess about the underlying distribution.
For example, your total_rooms variable is heavily skewed, most houses have very few rooms, but there are a few outliers with huge numbers of rooms. I thus wouldn't trust the mean value of total_rooms much and rather use the median - of course this is highly dependent on what kind of analyse you are doing.


time series prediction using LSTM Hi I have fairly very short time series data. The data set has number of systems $s_{1}, s_{2}, s_{3},..s_{n}$. For each $s_{i}$ we have recorded number of failures on each day. As of now, we have recorded 30 days. I would like to know can I use LSTM for predicting number of system failures for next day.
How about using Vector Auto Regression?. Any starting pointers and code references will be useful. Thank you.

Yes, LSTMs are fairly used in time series prediction. They can even handle missing data which is quite common in timeseries: Learning to diagnose with LSTM recurrent neural network. Still, I don't recommend starting directly with LSTM as the training takes time and you have to try many parameters to find the best that fits your data. You should start with some basic regressors like RandomForest or XGBoost.


Skip Connections in Residual Modules I am a beginner in CNN theory and would like to understand the usage of residual modules better.
As far as I understand residual modules can be skipped, only the activation function must be computed with the given input.

My question is: How does the network know if it should skip the residual module or not?

Residual blocks contain weights as you can see in this overview of some different variants:

Source: this blog post

Since weights are learned parameters the neural net can learn to use or not use non-skip/non-identity paths, i.e. by optimizing with gradient descent the network can learn to skip these blocks (or not).

To phrase it differently: the networks "knows" by following the opposite direction of the gradient.


How to measure the contribution of each input channel to a network's accuracy? As the title says, let's assume we have some neural network. More specifically, a regression network in case that changes anything. It takes multiple 2D input channels. How could one measure how much each input channel is contributing to the accuracy of the network on a validation set, to know which channels to keep and which are not really necessary?

Neural networks are black boxes, and figuring out its internal workings can be really hard. When you decide to use ML for your model, you are sacrificing accuracy for interpretability.
However, that does not mean it is totally impossible to interpret networks.  The most tedious way to determine feature importance would be to remove features and see how it impacts the model. If you're interested in research into the subject, I've encountered some research papers to use decision trees to understand networks.
https://openaccess.thecvf.com/content_CVPR_2019/papers/Zhang_Interpreting_CNNs_via_Decision_Trees_CVPR_2019_paper.pdf
https://arxiv.org/pdf/1802.02195.pdf


Identify social users using graph embeddings I have a social network and I want to identify the most social people in the graph. In a typical graph experiments this could have done using different network features.

However, I recently came across graph embeddings and interested in knowing if my problem can be solved using graph embeddings.

If so, please letme know how I can do that?

This question can have many ways to answer but here is what I think:

Lets assume the example of very common, facebook.

Very naive sense

Number of friends is equal to the 'social-ness' of the person. In graph context, a node with more degree will considered to be strongly connected in the graph. But ofcourse we know this is not at all a good way of defining the social-ness.

Every human point of view

Ask yourself a question about when do you like or comment in someone's post or photo ? The answer most probably will be when you and that person have good terms or friendship or relationship. So, with this kind of approach you can say that the person with more number of likes/comments in their posts is more social than the one with less likes/comments.

Now, coming to your question, it depends on how you define 'social' as I can also say a person with similar interest as the community with largest number of members with similar interest is most social. So, lets say we have a facebook graph and you want to measure social-ness of vertices you need to exploit the edge information also which is nothing but likes/comments/messages shared between two people. You can convert this edge data into some kind of number to denote the strength of that edge. Also, in the facebook scenario there is a directed edge between any two nodes (people) as it can happen that one person likes/comment/message the another person but vice-versa is not true. So, there are many more kinds of data/information that you can use in facebook graph.

Then as answered in this post graph embedding will give you the representation of a graph/subgraph as a whole. So, you need vertex embeddings in this case so you can represent  every vertex separately and compare against one another.

One approach is to use deepwalk and get vector embeddings based on topology (you need to modify it a bit as the original method is only for undirected graphs). Then you can form another vector like [1050, 300, 500, 2, ...] where each element will be some kind of number denoting number of total likes, number of total comments, number of friends, frequency of messages, frequency of likes etc. You can then train a neural network which basically outputs a number between 1-5, 1 being least social and 5 being most social. So, in this way you can take advantage of vector embeddings.

This is one approach that I can think of. If you have any other thoughts please share them.


How to approach different image resolutions in deep learning for regression problem? I have an image dataset of various resolutions and using regression DNN model with fixed n*n input resolution. As model learns certain positions in the image, I've been using zero padding to fit images resolutions to maintain 1:1 aspect ratio.
Is there a better way to preprocess images?
Without zero padding, I get worse results, and I guess maintaining aspect ratio is necessary to avoid objects' shape distortions in DNN input.

In a fully convolutional neural network, i.e., a neural network containing only convolutional and pooling layers, you can get away with input images of different sizes. but that would only work if you didn't need the output of your network to be of a fixed size. for example for segmentation tasks, because the output is the same size as the input, you can actually design a fully convolutional network where inputs are images of different resolution. This is basically one of the ideas behind FCN.
Aside from that in classification or regression, because your output is a certain fixed size, you have to fix each layers size and this also means that the input must have a certain size. Plus in classification or regression tasks, fully connected layers are used as the last layers of the network, thus because of this reason too, you have to have fixed size inputs.
Long story short, in your case, there is nothing you can do architecture-wise to get away with this. So you have to handle it in your sample data.
What is usually done, is that you transform your dataset and zero-pad all the images to a fixed resolution and then use it.
Don't scale your images to a certain resolution as convolutional layers are not scale invariant and this will mess up your test phase.


Attribute error when loading pickle file in python jupyter notebook I have created a config file with the following code for an object detection task and saved in the local disk.

# Create the config
C = Config()
C.use_horizontal_flips = horizontal_flips
C.use_vertical_flips = vertical_flips
C.rot_90 = rot_90
C.record_path = record_path
C.model_path = output_weight_path
C.num_rois = num_rois
C.base_net_weights = base_weight_path

with open(config_output_filename, 'wb') as config_f:
pickle.dump(C,config_f)

I am trying to load this pickle file in another jupyter notebook.

with open(config_output_filename, "rb") as f:
C = pickle.load(f)
# turn off any data augmentation at test time
C.use_horizontal_flips = False
C.use_vertical_flips = False
C.rot_90 = False
print(C.num_rois)

But it gives me the following error.

------------------------------------------------------------------------

AttributeError             Traceback (most recent call
last)
 <ipython-input-20-c5528d5ef98b> in <module>
  1 with open(config_output_filename, "rb") as f:
   ----> 2     C = pickle.load(f)

  AttributeError: Can't get attribute 'Config' on <module '__main__'>

But I was able to load the pickle file from the disk without errors in my previous attempts and applied the configurations on the test set.

I guess the Config() is a class that you created. It will be needed again when you load a pickle file storing the config. Therefore, you need to import it in the jupyter notebook you use to load your pickle file.

Hope this could be helpful.


Categorizing Customer Emails I am working on a project for a company which needs to categorize customer e-mails regarding loans and insurance. The e-mails are labeled uniquely from set of 13 category labels. The number of records in the dataset are 3000. All e-mails are not in proper English or format, there are a lot of spelling mistakes and unwanted features as Indian Names, addresses etc.. The dataset is very imbalanced, the frequency of top 3 categories combines upto 2000 and there are 2 categories with frequencies ranging from 40-80.
I managed to pre-process the data and remove most of the unwanted stuff and spelling mistakes.

I have created a supervised learning model using LinearSVC which gave around 78% on validation dataset, after which I used XGBoost classifier which increased the accuracy to 80%. Both of these models use Tfidf Vectors with unigrams and bigrams of the pre-processed(stop words removed and stemmed text) as input.
I have tried SMOTE, TruncatedSVD and SelectKBest with chi2 but they decreased the accuracy.

In terms of Machine Learning Models I have used Multinomial Naive Bayes, Random Forests, Decision Tree Classifiers. I have even tried Feed Forward Neural Networks(MLP) and Recurrent Neural Networks for the classification but still XGBoost gives the best accuracy. I have also used Topical Modeling using LDA. I generated top words for each category and tried finding cosine similarities of each test set record with all the generated top words belonging to each category, still no improvement in accuracy.

So, my question is what kind of features should I look for? How do I extract more sophisticated features? My main objective is to increase the accuracy as the product is going to be an in-house software for the company. I am looking forward to context aware feature extraction or somehow engineer the features(add more or enhance them) to increase accuracy

PS: In the emails the customer ask for multiple actionable items, but still in the training set they are pre-classified as only one label corresponding to one of the actionable item asked in the e-mail. Plus, the data is also manually  wrongly classified by the previous workers. Please suggest and approach to capture the actionable items. I Have tried using class probabilities but they don't seem like giving the right answer, plus if the person has asked for only one actionable item then also the class probabilities will indicate other. I want to capture the actionable items as per asked by User. I have tried POS-tagging for this but due to bad quality of data, it is not giving good results.

PPS: The Email data which I have is in HTML format and has threads. I tried removing previous conversation threads before training but it resulted in bad accuracy. It is very strange that the more I try to refine the data by removing unwanted contents like disclaimers, auto-response emails, replies from customer-care representatives the more the accuracy falls.

Looking forward to get some insights on how to change my approach and specifically the features.

Thank You

(1) Data quality. The single best way to improve your accuracy. Garbage in garbage out. You already said your data is suspect. Some data was mis-classified; data only has single label, when multiple labels are possible. This is the biggie - this will improve your accuracy more than any other technique: improve the quality of your training data. One way to go about this. Recruit expert labellers (these could be workers in your own company who do this task, or they could be external people, e.g., mturkers who have been trained to do this labelling task). In general, the more labellers, the better. At a minimum I would have 3 labellers, label all n emails in your train/test set, using either majority label wins, or require all-three consensus depending on your confidence needs. You will want to measure your inter-rater reliability to ensure the integrity of your data using Cohen's kappa or similar. Importantly, this data will become your gold standard data. That is to say the average agreement upon labels provided by your original labellers is the ground truth to which your algorithm attempts to approach.

Obviously, this approach to data quality is costly in terms of resources (time to label all the emails; cost of hiring mturkers, etc.). But depending upon your specific needs it needn't be... basically it depends upon the needed level of "expertise" to do the task. Sometimes, it is safe to consider you (the researcher/developer) as the "expert" (e.g., you yourself said that some emails were mis-classified... how do you know that? You must have some sort of expert knowledge.). However, using a single labeller for the task makes your data less reliable due to individual processing differences introducing a bias into the data.

(2) Including stop words. Yes, you read that correct including stop words. Grammatical function words (e.g., the, and, of, etc.) can often be highly indicative of specific text types or genres. It is premature to eliminate stop words without first ensuring that they are an uninformative feature. Similarly, try including punctuation, try including capitalization features. Time and time again, I see people blindly pre-processing text without empirically determining whether these features are significant predictors of class.

(3) Capture spelling variation. As the OP stated, the original texts have many "spelling mistakes." You can improve accuracy by reducing this variance - basically map variants to their canonical form before extracting your word features (e.g., "colour" -> "color"). I often rely on external corpora to rapidly create these spelling maps.

(4) Balance your training data - The OP mentions that his classes are severely imbalanced. Try over- and/or under-sampling to balance your classes.

(5) Describe your problem. It is also important to understand what the underlying purpose of your classifier. If your classifier is intended to run on historical data, perhaps to flag incorrectly human-classified emails, then you would probably want to keep auto-response and correspondence thread info. On the other hand, if the goal is to classify new customer emails so that they get properly routed, for example, then you probably want to remove these correspondence features from your training and test data.


Python Matplotlib Radar not correctly centered I'm trying to create radar plots for players's profiling. I got this solution from this previous question, but the graphic is not correctly centered and in fact there is a label that ends outside the plot.

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns  # improves plot aesthetics

def _invert(x, limits):
    """inverts a value x on a scale from
    limits[0] to limits[1]"""
    return limits[1] - (x - limits[0])

def _scale_data(data, ranges):
    """scales data[1:] to ranges[0],
    inverts if the scale is reversed"""
    for d, (y1, y2) in zip(data[1:], ranges[1:]):
        assert (y1 <= d <= y2) or (y2 <= d <= y1)
    x1, x2 = ranges[0]
    d = data[0]
    if x1 > x2:
        d = _invert(d, (x1, x2))
        x1, x2 = x2, x1
    sdata = [d]
    for d, (y1, y2) in zip(data[1:], ranges[1:]):
        if y1 > y2:
            d = _invert(d, (y1, y2))
            y1, y2 = y2, y1
        sdata.append((d-y1) / (y2-y1) * (x2 - x1) + x1)
    return sdata

class ComplexRadar():
    def __init__(self, fig, variables, ranges,
                 n_ordinate_levels=6):
        import ipdb; ipdb.set_trace()
        angles = np.arange(0, 360, 360./len(variables))
        axes = [fig.add_axes([0.1, 0.1, 0.9, 0.9], polar=True,
                label="axes{}".format(i))
                for i in range(len(variables))]
        l, text = axes[0].set_thetagrids(angles,
                                         labels=variables)
        [txt.set_rotation(angle-90) for txt, angle
         in zip(text, angles)]
        for ax in axes[1:]:
            ax.patch.set_visible(False)
            ax.grid("off")
            ax.xaxis.set_visible(False)
        for i, ax in enumerate(axes):
            grid = np.linspace(*ranges[i], num=n_ordinate_levels)
            gridlabel = ["{}".format(round(x, 2))
                         for x in grid]
            if ranges[i][0] > ranges[i][1]:
                grid = grid[::-1]  # hack to invert grid. gridlabels aren't reversed
            gridlabel[0] = ""  # clean up origin
            ax.set_rgrids(grid, labels=gridlabel,
                          angle=angles[i])
            # ax.spines["polar"].set_visible(False)
            ax.set_ylim(*ranges[i])
        # variables for plotting
        self.angle = np.deg2rad(np.r_[angles, angles[0]])
        self.ranges = ranges
        self.ax = axes[0]

    def plot(self, data, *args, **kw):
        sdata = _scale_data(data, self.ranges)
        self.ax.plot(self.angle, np.r_[sdata, sdata[0]], *args, **kw)

    def fill(self, data, *args, **kw):
        sdata = _scale_data(data, self.ranges)
        self.ax.fill(self.angle, np.r_[sdata, sdata[0]], *args, **kw)

def data2():
    metrics = ('Shots', 'Goal Conversion %', 'Assists', 'Passing %', 'Int+Tackles', 'Disposessed')
    firmino = (3.4, 15.7, 0.3, 82.3, 2.6, 1.7)
    ranges = [(1, 5), (4.5, 22.5), (0.1, 0.4), (65, 85), (1, 4.5), (1, 3)]
    fig = plt.figure(figsize=(12, 9))
    radar = ComplexRadar(fig, metrics, ranges)
    radar.plot(firmino)
    radar.fill(firmino, alpha=0.2)
    plt.savefig('firmino.png')

if __name__ == '__main__':
    data2()

Seems to me that fig.add_axes() is miscalculating the dimensions here, essentially doubling the "left" and "bottom" values, not sure why. So you should use 0.8 for "right" and "top" values, as in:

axes = [fig.add_axes([0.1, 0.1, .8, .8], polar=True,
            label="axes{}".format(i))
            for i in range(len(variables))]


How to run AgglomerativeClustering on a big data in python? I run AgglomerativeClustering on a sample of data and fit a model. then I decide to predict this fit for all of my data but I got MemoryError.

How can I run AgglomerativeClustering on a big dataset?
should I create a classification based on clusters label?!

You can't.

By definition, the algorithm needs O(n²) memory and O(n³) runtime.

This does not scale to big data.

Use a different algorithm. Or subsample your data.

Results don't necessarily get better just because you use more data. In many cases it really does not matter. The quality of estimating the mean grows with sqrt(n), so it quite quickly does not pay off to use more data, as this will only affect small digits of the result.


Training NLP with multiple text input features Question:

How can I train a NLP model with discrete labels that is based on multiple text input features?

Background:

I'm trying to predict the difficulty of a 4-option multiple choice exam question (probability of a test-taker selecting the correct response) based on the text of the question along with its possible responses. I'm hoping to be able to take into account how some incorrect yet convincing responses, the exact subject of which is relative to the content of the question, can skew the difficulty of the exam question, as well as how the wording of the question can make the question misleading.

Intuition, Attempts and Options:

My intuition is that the content of the question and its responses are both significant in the prediction of its difficulty. However, when using a library like Spacy, NLTK or Textacy, training seems to be done on only one text column at a time. I'm looking at potentially five text columns at a time, or two if I concatenate the question responses together.

I haven't been able to find much on the topic, but here is an attempt I found. I thought this attempt was flawed because they were just doing a single-column train multiple times, and for example training the City against a salary value and concatenating that to your train of Job Description against salary value is not going to give a meaningful improvement to your first model, since the Job Description did not depend on the City when training.

My options that I've found are:

Following the above attempt after all (which I think is flawed)
Concatenating my text features (which I can't understand why that would make sense in this case but seems to be the norm)
Eliminating some of my features entirely, such as narrowing the question down by subject matter and disposing of the question content and just training on the response options concatenated together (which also removes some very important information in the question content that can lead to prediction values)

Thoughts and advice? Is there a library that can make this easier? Thanks!

Concatenating the whole question and its answers in a RNN could be an option to try, but then always use a reserved special token (or various) to mark where the questions start. E.g. you could concatenate like:

Question text <1> answer 1 <2> answer 2 <3> answer 3 <4> answer 4

where <1>, <2>... are the special tokens so that the model, with enough examples, may be able to understand its meaning.

The "enough examples" is worth stressing, specially as this type of model may require a large complexity (in terms of number of parameters) to make it work, and hence you will need a large enough data set, possibly of the order of the 10k or larger. You can also test some data augmentation by mixing the order of the answers in the input, and of course changing the correct label.


Memory error when using more layers in CNN model On my dell core i7 - 16GB RAM - 4gb 960m GPU laptop, I am working on a project to the classify lung CT images using 3d CNN. I'm using the CPU version of tensorflow. The images are prepared as numpy array size (25,50,50).

My CNN model had 2 conv layers, two maxpool layer, one FC layer and output layer. With this architecture I could train the model with approximately (5000 to 6000) samples. After adding more layers my model now has 6 conv layers, 3 max-pool layers, FC and output layer. My problem is after changing the architecture with just more than 1000 samples my memory gets filled and I get memory error. I tried to make smaller batches, but every time getting same error. I have two questions:

Why by adding more layers the model needs more memory?
Is there any way to deal with this type of problem?

More layers mean more parameters for your network, which in turn means more required space in memory to store those parameters.
The only solution (besides increasing the memory of your computer) is reducing the size of your network. A few pointers on this: Firstly, 3-D CNNs require much more space than 2-D ones. One idea could be to shift to a 2-D one. Other than that, the layers with the most parameters are the FC ones. These are sometimes redundant. I'd suggest reducing the size of those first.


python pandas optimization: filtering on text index values I have to filter a pandas data frame by matching a complex regular expression on a text index.

The data frame is multi level indexed, and contains more than 2 million records.

The way I'm doing is:

identifiers = self._data.index.get_level_values('identifier')
filt = ... # an_array_of_np.bool_with_the_same_length_as_my_data

pattern = ... # a complex regular expression, as a string
filt = filt & np.array(identifiers.str.contains(pat=pattern, case=False, regex=True), dtype=np.bool)

... # other filterings

Unfortunately, the line beginning by filt = filt & is very slow.

I'm wondering if you have some ideas to make it faster. I guess that's because of the identifiers.str.contains

Thanks a lot!

EDIT:

Thanks @Emre

I'm not allowed to share those data, but the code below demonstrates the problem:

Step 0: 0:00:00.013527
Step 1: 0:00:00.010127
Step 2: 0:00:04.468114
Step 3: 0:00:02.109594
Step 4: 0:00:00.027437

In fact, my feeling is that we apply the regular expression on all the values of the identifiers, while I would expect that the filter applies on the possible values of the index (lost of values are reused many times).

import pandas as pd
import numpy as np
import datetime

N = 2000000
N_DVC = 10000

def getData():
    identifiers = np.random.choice(np.array([
        "need", "need: foo", "need: bar", "need: foo: bar", "foo: need", "bar: need",
        "not: need", "not: need: foo", "not: need: bar", "not: need: foo: bar", "foo: need: not", "bar: need: not",
        "need ign", "need: foo ign", "need: bar ign", "need: foo: bar ign", "foo: need ign", "bar: need ign",
        "ign need", "need: ign foo", "need: ign bar", "need: foo: ign bar",
    ]), N)
    devices = np.random.choice(np.arange(N_DVC))
    timestamps = np.random.choice(pd.date_range('1/1/2016 00:00:00', periods=60*60*24, freq='s'), N)
    x = np.random.rand(N)
    y = np.random.rand(N)
    data = pd.DataFrame({'identifier': identifiers, 'device': devices, 'timestamp': timestamps, 'x': x, 'y': y})
    data.set_index(['device', 'identifier', 'timestamp'], drop=True, inplace=True)
    return data

def filterData(data):
    # I know those regular expressions are not perfect for the example,
    # but it mimics the real expressions I have
    rexpPlus = '^(?:[^\s]+:\s)*need(?:(?::\s[^\s]+)*:\s[^\s]+)?$'
    rexpMinus = '(?::\s)(?:(?:not)|(?:ign))(?::\s)'

    tic = datetime.datetime.now()
    identifiers = data.index.get_level_values('identifier')
    print("- Step 0: %s" % str(datetime.datetime.now() - tic))

    tic = datetime.datetime.now()
    filt = np.repeat(np.False_, data.shape[0])
    print("- Step 1: %s" % str(datetime.datetime.now() - tic))

    tic = datetime.datetime.now()
    filt = filt | np.array(identifiers.str.contains(pat=rexpPlus, case=False, regex=True), dtype=np.bool)
    print("- Step 2: %s" % str(datetime.datetime.now() - tic))

    tic = datetime.datetime.now()
    filt = filt & (~np.array(identifiers.str.contains(pat=rexpMinus, case=False, regex=True), dtype=np.bool))
    print("- Step 3: %s" % str(datetime.datetime.now() - tic))

    tic = datetime.datetime.now()
    data = data.loc[filt, :]
    print("- Step 4: %s" % str(datetime.datetime.now() - tic))

    return data

if __name__ == "__main__":
    filterData(getData())

I've found a solution: instead of filtering through all the values, I compute a hash table (in fact a DataFrame) of the possible values and filter on those values:

Step 0: 0:00:00.014251
Step 1: 0:00:00.010097
Init fast filter: 0:00:00.353645
Step 2: 0:00:00.027462
Step 3: 0:00:00.002550
Step 4: 0:00:00.027452
Globally: 0:00:00.435839

It runs faster because there are lots of duplicated values.

Here's a class which is doing that stuff. It works for indexes and normal columns:

import pandas as pd
import numpy as np

class DataFrameFastFilterText(object):
    """This class allows fast filtering of text column or index of a data frame
    Instead of filtering directy the values, it computes a hash of the values
    and applies the filtering to it.
    Then, it filters the values based on the hash mapping.
    This is faster than direct filtering if there are lots of duplicates values.
    Please note that missing values will be replaced by "" and every value will
    be converted to string.
    Input data won't be changed (working on a copy).
    """

    def __init__(self, data, column, isIndex):
        """Constructor
        - data: pandas Data Frame
        - column: name of the column - the column can be an index
        - isIndex: True if the column is an index
        """
        self._data = data.copy()  # working with a copy because we will make some changes
        self._column = column
        self._isIndex = isIndex
        self._hashedValues = None
        self._initHashAndData()

    def _initHashAndData(self):
        """Initialize the hash and transforms the data
        1. data: a. adds an order column
                 b. creates the column from the index if needed,
                 c. rename the column as 'value' and
                 d. drops all the other columns
                 e. fill na with ""
                 f. convert to strings
        2. hash: data frame of 'hashId', 'hashValue'
        3. data: a. sort by initial order
                 b. delete order
        """
        self._data['order'] = np.arange(self._data.shape[0])
        if self._isIndex:
            self._data[self._column] = self._data.index.get_level_values(self._column)
        self._data = self._data.loc[:, [self._column, 'order']]
        self._data.rename(columns={self._column: 'value'}, inplace=True)
        self._data['value'] = self._data['value'].fillna("", inplace=False)
        self._data['value'] = self._data['value'].astype(str)
        self._hash = pd.DataFrame({'value': pd.Series(self._data['value'].unique())})
        self._hash['hashId'] = self._hash.index
        self._data = pd.merge(
            left=self._data,
            right=self._hash,
            on='value',
            how='inner',
        )
        self._data.sort_values(by='order', inplace=True)
        del self._data['order']

    def getFilter(self, *args, **kwargs):
        """Returns the filter as a boolean array. It doesn't applies it.
        - args: positional arguments to pass to pd.Series.str.contains
        - kwargs: named arguments to pass to pd.Series.str.contains
        """
        assert "na" not kwargs.keys()  # useless in this context because all the NaN values are already replaced by empty strings
        h = self._hash.copy()
        h = h.loc[h['value'].str.contains(*args, **kwargs), :].copy()
        d = self._data.copy()
        d['keep'] = d['hashId'].isin(h['hashId'])
        return np.array(
            d['keep'],
            dtype=np.bool
        ).copy()

And the example code is adapted as follows:

...
tic = datetime.datetime.now()
fastFilter = DataFrameFastFilterText(data, 'identifier')
print("- Init fast filter: %s" % str(datetime.datetime.now() - tic))

tic = datetime.datetime.now()
filt = filt | fastFilter.getFilter(rexpPlus)
print("- Step 2: %s" % str(datetime.datetime.now() - tic))

tic = datetime.datetime.now()
filt = filt & (~fastFilter.getFilter(rexpMinus))
print("- Step 3: %s" % str(datetime.datetime.now() - tic))
...


Clustering on categorical attributes I have a dataset with only 2 categorical attributes out of 9. How can I get a clustering analysis on it?
I am using R. Do you have any advices about instructions, how to do it, topics, ...?
here's my dataset

thanks

Here is nice implementation of mixed type data in R-

https://dpmartin42.github.io/posts/r/cluster-mixed-types

This question right here-
K-Means clustering for mixed numeric and categorical data

and a Discussion Thread of Kaggle-

https://www.kaggle.com/general/19741

There are ways, to either map your categorical data to numeric type and then you can go about the business as usual, or choose similarity measures which works for categorical data type, in this case you have options to choose from counting frequencies etc  .


Tokenization of data in dataframe in python I am performing tokenization to each row in my dataframe but the tokenization is being done for only the first row. Can someone please help me. thank you.
Below are my codes:

import pandas as pd
import json
import nltk

nltk.download('punkt')
nltk.download('wordnet')
from nltk import sent_tokenize, word_tokenize

with open(r"C:\Users\User\Desktop\Coding\results.json" , encoding="utf8") as f:
     data = json.load(f)
df=pd.DataFrame(data['part'][0]['comment'])
split_data = df["comment"].str.split(" ")
data = split_data

print(data)

def tokenization_s(data): # same can be achieved for words tokens
    s_new = []
    for sent in (data[:][0]): #For NumpY = sentences[:]
        s_token = sent_tokenize(sent)
        if s_token != '':
            s_new.append(s_token)
    return s_new

print(tokenization_s(data))

My output is:

[nltk_data] Downloading package punkt to
[nltk_data]     C:\Users\User\AppData\Roaming\nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package wordnet to
[nltk_data]     C:\Users\User\AppData\Roaming\nltk_data...
[nltk_data]   Package wordnet is already up-to-date!
0                             [enjoy, a, lovely, moment]
1      [I, was, there, for, my, honeymoon., The, hote...
2      [Had, an, amazing, stay, for, 2, nights.\nThe,...
3                 [Had, a, good, time., Food, is, good.]
4      [A, highly, recommendable, hotel., Value, for,...
                             ...
131    [Wonderful, experience,, a, quite, different, ...
132                            [Was, a, paradise, stay.]
133    [It, was, really, a, place, to, be, for, relax...
134    [It, was, just, perfect, with, an, excellent, ...
135                               [It's, was, excellent]
Name: comment, Length: 136, dtype: object
[['enjoy'], ['a'], ['lovely'], ['moment']]

Process finished with exit code 0

What should I do for the system to tokenise each row in the dataframe?

You can try with this:

import pandas as pd
import nltk

df = pd.DataFrame({'frases': ['Do not let the day end without having grown a little,', 'without having been happy, without having increased your dreams', 'Do not let yourself be overcomed by discouragement.','We are passion-full beings.']})

df['tokenized'] = df.apply(lambda row: nltk.word_tokenize(row['frases']), axis=1)


Export Python Code from Orange Since Orange is just running python code under the hood... is it possible to set up and do my analysis in Orange but then export or output the underlying code for documentation or replication?

Since I was also looking for it just yesterday being a newbie,sharing what I found , As per Orange FAQ it says

and it directs to the link @K3---rnc mentioned.

But i found similar request as well not sure how well it works-

Orange Data Mining load saved models

https://stackoverflow.com/questions/45598627/orange-save-model-in-python-script


Complete IPv4 Address List Is it possible to pull a complete list of all IPv4 addresses and put them into a text file? Since there are 4,294,967,296 IPv4 addresses and each one takes about 4 bytes I would guess the file would be about 17.179869184 GB. Is it possible to download all of these addresses into one .txt file?

with easy if you know a little Python.
According to
https://gist.githubusercontent.com/pootzko/ac34906e11d9715d9514c75507c24dc0/raw/1c28b679d1ca23da56a930950f21a82dba379cd0/ip_generator.py

def ipRange(start_ip, end_ip):
   start = list(map(int, start_ip.split(".")))
   end = list(map(int, end_ip.split(".")))
   temp = start
   ip_range = []

   ip_range.append(start_ip)
   while temp != end:
      start[3] += 1
      for i in (3, 2, 1):
         if temp[i] == 256:
            temp[i] = 0
            temp[i-1] += 1
      ip_range.append(".".join(map(str, temp)))

   return ip_range

output = []
# sample usage
ip_range = ipRange("192.168.1.0", "192.171.3.25")
for ip in ip_range:
    output.append(ip)
fo = open("output.txt", "rw+")
line = fo.writelines(output)
fo.close()


Structure of Convolutional Neural Network to analyze a sequence of frames I want to apply a CNN to a series of image sequences to classify that sequences of frames/images in two groups/categories. We are in a binary classification problem. My dataset is composed by a lot of 'batches' of frames. For example, each batch of frames could be compose of 20 frames of 64x64 pixels. One important thing is that the order of that 20 frames is important. If you shuffle the order of that 20 frames the output could change.

For all this, I want to create a CNN to solve this binary classification problem. I'm usig Keras and TensorFlow.

What is my question? Well, I'm not sure if I have to use a TimeDistributed layer or not. The input shape of the neural network is the following one: (20, 64, 64, 1), whose meaning is: 20 frames with a 64x64 size (1 channel - grayscale). Should I use a TimeDistributed layer?

I'm not sure if I have to use a TimeDistributed layer or not

You definitely don't have to use TimeDistributed. You have other choices, that may be equally valid, depending on your data:

Flatten your example data into 81920 features per example and use a simple Dense layer.
Use a Conv3D layer.
Use some form of RNN, such as LSTM.

From your data description, I would expect either a TimeDistributed or 3D CNN based approach to be a good first bet. Intuition suggests that the CNN would work better if there was minor change between frames (because it has capability to directly find subtle frame differences), the TimeDistributed approach would work better processing larger changes (because it will ignore frame differences until the fully-connected layer).

  Should I use a TimeDistributed layer?

Only you can answer this, by trying it and measuring performance of your classifier. However, it should function correctly, and intuition suggests it would be a good choice if your frames are in sequence but visually disjoint.


cannot use sklearn.naive_bayes MultinomialNB to predict from one attribute I tried to create a simple example for Naive Bayes by learning from one attribute.  It seems like I cannot use sklearn.naive_bayes.MultinomialNB to predict from one attribute.  This is because the predict_proba is the same for every input.

Here is the code:

import numpy as np
X = np.array([1, 1, 1, 0, 0, 1, 1])
X = X.reshape(-1,1)
y = np.array([0, 0, 0, 0, 1, 1, 1])

from sklearn.naive_bayes import MultinomialNB
clf = MultinomialNB()
clf.fit(X, y)

print(clf.predict(X))
print(clf.predict_proba(X))

and its result.

[0 0 0 0 0 0 0]
[[0.57142857 0.42857143]
 [0.57142857 0.42857143]
 [0.57142857 0.42857143]
 [0.57142857 0.42857143]
 [0.57142857 0.42857143]
 [0.57142857 0.42857143]
 [0.57142857 0.42857143]]

If your problem is shaped as you stated, you might need to reconsider your Naive Bayes model. MultinomialNB does not seem to be appropriate for your case. You can rather use the Naive Bayes classifier that uses the gaussian distribution.

from sklearn.naive_bayes import GaussianNB
gnb = GaussianNB()
gnb.fit(X, y)
print(gnb.predict(X))
print(gnb.predict_proba(X))


Constant validation loss & accuracy, training accuracy fluctuates I am training a Squeeze-net model for binary classification of images. I have 79968 images for training (50:50 for and against) and 8892 images in the validation set. After 35000 iterations my training accuracy fluctuates between 1 and 0.96875. The validation accuracy is more or less constant between [0.985, 0.986]. The base learning rate is 0.01 and decreased upto 0.00001. As far as I can tell visually training loss doesn't exactly fluctuate between two numbers as such but is in the range [0.02, 0.09] for most part except a few occasional spike.

My question is what can i infer from this?
(1) Overfit?
(2) Model has converged? If not should I reduce the learning rate?
(3) Is the model stuck at a local minima? I am using softmax-with-loss as my loss layer.

Did the loss decrease up to a certain point and then start fluctuating between .02 and .09? If so you may have just reached your convergence and now it's oscillating due to the size of the learning rate being too large in relation to the size of the minima you have hit. Reduce your learning rate and see where it ends up, my guess is that you will get close to .02 this time around, unless you have truly hit a local minima, in which case you may go lower.