Predicting and clustering at the same time? I want to build a segmentation to substitute the existing RFM segmentation which is a basic segmentation based on the Recency, Frequency and Monetary values.

The new segmentation will be used for two purposes:

Rank customers per value
Have homogeneous groups of customers, so there should be an interpretation and meaning for each group.

I think this shouldn't be an unsupervised learning, but a mix of both supervised and unsupervised learning.

So What I did is that I defined a target variable that takes 1 if the customer is active in the next period and 0 if not. then I used a decision tree to predict this value, so that the leaves represent the clusters. the percentage of active customers in each group is used to rank these clusters.

Do you think that's a good solution and do you have other ideas ?

Thanks in advance

Your solution seems interesting. However, it implies that the customers’ value is assessed through whether they are active or not in next period. It might be interesting to look at the purchase by customer for, say future 8 weeks (or future 6 weeks or 12 weeks for that matter – the time frame depends a lot on the particularity of the business). This measurement reflects whether customers are active or not in the next period and also quantifies their 'activity’.

For a somewhat similar project I implemented the following solution:

Start by randomly selecting one breakpoint for each of the three dimensions (RFM); by breakpoint I mean a random value among the unique values of Recency, another randomly selected value for Frequency, and another one for Monetary.
Draw some initial segments based on these three breakpoints by selecting groups of customers that are relatively similar in terms of RFM according to the selected breakpoints. As there are 3 breakpoints (one per variable), you will get 2^3 segments because each variable will be broken down in 2 parts leading to 8 segments in total. In other words, you will have:

Segm 1: Recency < breakpoint_Recency & Frequency < breakpoint_Frequency & Monetary < breakpoint_Monetary

Segm 2: Recency < breakpoint_Recency & Frequency >= breakpoint_Frequency & Monetary < breakpoint_Monetary

Segm 3: Recency < breakpoint_Recency & Frequency < breakpoint_Frequency & Monetary >= breakpoint_Monetary

Segm 4: Recency < breakpoint_Recency & Frequency >= breakpoint_Frequency & Monetary >= breakpoint_Monetary

Segm 5: Recency >= breakpoint_Recency & Frequency < breakpoint_Frequency & Monetary < breakpoint_Monetary

Segm 6: Recency >= breakpoint_Recency & Frequency >= breakpoint_Frequency & Monetary < breakpoint_Monetary

Segm 7: Recency >= breakpoint_Recency & Frequency < breakpoint_Frequency & Monetary >= breakpoint_Monetary

Segm 8: Recency >= breakpoint_Recency & Frequency >= breakpoint_Frequency & Monetary >= breakpoint_Monetary

Compute the mean of future 8 weeks purchases for each of the resulted segments
Then, of course, this segmentation was random and is not the optimal one. To get the optimal segmentation you then implement a Genetic Algorithm to re-do the previous procedure and explore the space for better breakpoints. By this you will likely get the optimal segmentation of homogeneous customers in terms of their purchasing behavior (as characterized by RFM) – optimal with respect to their value (as you decide to define their value).

By the way, the fitness function for the GA aims to maximize the mean of future 8 weeks purchases per segment. Or you can also set your own objective on how to measure the value of customers and define the fitness function accordingly.

Hope this helps. Good luck!

Update: This solution allows you to find 'the homogenous groups of customers' and 'rank segments per value'. I am not sure if this is what you meant by 'rank customers per value'. In my view supervised clustering defeats the idea of individual ranking (but maybe I am wrong).
Moreover, customers change behaviors from one period to the other and it is perhaps better to cluster them in terms of purchasing behavior, not to rank individuals at a point in time. In this way, when a customer changes behavior (in terms of RFM), we know where s/he went (i.e. in which cluster), and thus we can assess whether s/he 'improved' or not. In the latter case, marketing people can take action to incentivize the concerned customer to move back into the better off segment.


adding summary to tensorflow model error I have a CNN model for classifying lung CT images, the code is written in tensorflow, I added some tensorflow summaries to my code to show graph, scalar, histogram, ... of my tensorflow model in tensorboard, in last step when I want to add_summary to the file writer it gives me an error, here is my code:

def train_CNN(input):
    train_predict = CNN_Model(x_img)
    with tf.name_scope("cross_entropy"):
        cost = tf.nn.softmax_cross_entropy_with_logits_v2(logits=train_predict, labels=y_label, name='cross_entropy')
        cost = tf.reduce_mean(cost, name='reduce_mean')
        tf.summary.scalar("cost", cost)
    with tf.name_scope("optimization"):
        optimizer = tf.train.AdamOptimizer(learning_rate, name='AdamOptimizer').minimize(cost)

    with tf.name_scope("accuracy"):
        correct_predict = tf.equal(tf.argmax(train_predict, 1), tf.argmax(y_label, 1))
        accuracy = tf.reduce_mean(tf.cast(correct_predict, tf.float32))
        tf.summary.scalar("accuracy", accuracy)

    #tf.summary.image("input", x_img, 5)

    sess.run(tf.global_variables_initializer())
    log_train_path = 'C:/temp/tensorflow_logs' + '/train_{}'.format(datetime.now().strftime("%Y-%m-%d-%H%M%S"))
    summary_writer = tf.summary.FileWriter(log_train_path)
    summary_writer.add_graph(sess.graph)
    merged_summary = tf.summary.merge_all()

    all_time = 0

    for epoch in range(num_epochs):
        start_time = time.time()
        ep_loss = 0
        for data in train_data:
            X = data[0]
            Y = data[1]
            summary, _, c = sess.run([merged_summary, optimizer, cost], feed_dict={x_img: X, y_label: Y})
            ep_loss += c
            summary_writer.add_summary(summary, epoch)
        end_time = time.time()
        all_time += int(end_time-start_time)
        print('Epoch', epoch+1, 'completed out of',num_epochs,'loss:',ep_loss, 'time usage: '+str(int(end_time-start_time))+' seconds')

        print('Accuracy of this epoch:',accuracy.eval({x_img:[i[0] for i in val_data], y_label:[i[1] for i in val_data]}))

    print('Finall Accuracy:',accuracy.eval({x_img:[i[0] for i in val_data], y_label:[i[1] for i in val_data]}), 'time usage: '+str(all_time)+' seconds')

after running the model it give me error, can anyone tell me how to solve it?

here is the error:

InvalidArgumentError: Expected dimension in the range [-1, 1), but got 1
     [[Node: accuracy/ArgMax_1 = ArgMax[T=DT_FLOAT, Tidx=DT_INT32, output_type=DT_INT64, _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_y_label_0_1, accuracy/ArgMax_1/dimension)]]

Check out dimension

 X = data[0]
 Y = data[1]

Its likely that have dimensions like (1, ) what means that's 1 dim vector.

That should be work

import numpy as np
X = np.array(X).reshape(-1,1)
Y = np.array(Y).reshape(-1,1)


What does the relative sum of squares error signify in neural network analysis in SPSS? What range of values of the relative sum of sqaures error is acceptable for a good neural network? I am getting around 0.9 of the relative error for 1 model and around 0.4 for another. Are both the networks significant?

Check this:

  A value closer to 0 indicates that the model has a smaller random error component, and that the fit will be more useful for prediction.


NLP: How to group sub-field into fields? Suppose I have a list of strings that captures a sub-field of academic research and would like to group them as higher-level fields.
For example,

'Quantum Mechanics'  => 'Physics'
'Abstract Algebra'   => 'Mathematics'
....

My understanding is that standard NLP techniques may not fit here, because the relationship between sub-fields and fields are linked by its meanings but not word-frequency or word-embedding etc.

I wonder if there is anything done that could be useful to tackle this problem (papers or packages)?

You need an ontology or any form of third-party data which describes the relationships between fields and sub-fields. You could use resources such as Wikipedia categories or standard library classifications for instance. There are probably other options for scientific fields.


While using reindex method on any dataframe why do original values go missing? This is the original Dataframe:

What I wanted : I wanted to convert this above data-frame into this multi-indexed column data-frame :

I managed to do it by this piece of code :

# tols : original dataframe
cols = pd.MultiIndex.from_product([['A','B'],['Y','X']
['P','Q']])
tols.set_axis(cols, axis = 1, inplace = False)

What I tried : I tried to do this with the reindex method like this :

cols = pd.MultiIndex.from_product([['A','B'],['Y','X'],
['P','Q']])
tols.reindex(cols, axis = 'columns')

it resulted in an output like this :

My problem :
As you could see in the output above all my original numerical values go missing on employing the reindex method. In the documentation page it was clearly mentioned :
Conform DataFrame to new index with optional filling logic, placing NA/NaN in locations having no value in the previous index. A new object is produced unless the new index is equivalent to the current one. So i don't understand:

Where did i particularly err in employing the reindex method to lose my original values
How should i have employed the reindex method correctly to get my desired output

Regarding your code the tols must be cols?

cols = pd.MultiIndex.from_product([['A','B'],['Y','X']
['P','Q']])
tols.set_axis(cols, axis = 1, inplace = False)

This must be:

cols = pd.MultiIndex.from_product([['A','B'],['Y','X']
['P','Q']])
cols.set_axis(cols, axis = 1, inplace = False)

Personally I would have removed nan with a default value before performing the multi-indexing.

df_noNAN = from_product.fillna(value=-1)

In your case it looks like the inplace is producing a bug and better reported to the pandas team. On Stack exchange they are extremely responsive.


Data Science in C (or C++) I'm an R language programmer. I'm also in the group of people who are considered Data Scientists but who come from academic disciplines other than CS.

This works out well in my role as a Data Scientist, however, by starting my career in R and only having basic knowledge of other scripting/web languages, I've felt somewhat inadequate in 2 key areas:

Lack of a solid knowledge of programming theory.
Lack of a competitive level of skill in faster and more widely used languages like C, C++ and Java, which could be utilized to increase the speed of the pipeline and Big Data computations as well as to create DS/data products which can be more readily developed into fast back-end scripts or standalone applications.

The solution is simple of course -- go learn about programming, which is what I've been doing by enrolling in some classes (currently C programming).

However, now that I'm starting to address problems #1 and #2 above, I'm left asking myself "Just how viable are languages like C and C++ for Data Science?".

For instance, I can move data around very quickly and interact with users just fine, but what about advanced regression, Machine Learning, text mining and other more advanced statistical operations?

So. can C do the job -- what tools are available for advanced statistics, ML, AI, and other areas of Data Science? Or must I lose most of the efficiency gained by programming in C by calling on R scripts or other languages?

The best resource I've found thus far in C is a library called Shark, which gives C/C++ the ability to use Support Vector Machines, linear regression (not non-linear and other advanced regression like multinomial probit, etc) and a shortlist of other (great but) statistical functions.

Or must I loose most of the efficiency gained by programming in C by calling on R scripts or other languages?

Do the opposite: learn C/C++ to write R extensions. Use C/C++ only for the performance critical sections of your new algorithms, use R to build your analysis, import data, make plots etc.

If you want to go beyond R, I'd recommend learning python. There are many libraries available such as scikit-learn for machine learning algorithms or PyBrain for building Neural Networks etc. (and use pylab/matplotlib for plotting and iPython notebooks to develop your analyses). Again, C/C++ is useful to implement time critical algorithms as python extensions.


Dataset: language audio clips and country labels I'm looking for a dataset containing audio clips of different languages from around the world, including a label indicating which country the clip comes from.

Does such a dataset exist? Or is there an easy way to hack one together?

Topcoder hosted competition for identifying spoken languages. They released a database containing 176 languages. The datasets and the related resources can be found here.


One hot encoding vs Word embedding I am very confused between one hot encoding and word embedding in terms of structure of the network and how it reduces the dimensionality.

I am currently using encog with c# which has some documentation but not many differing examples.

If I take an example of text classification by neural networks and have:
1000 different texts
10,000 unique words

What I was previously doing was taking each text and then working out the unique words and putting them in a list. I would then work out the number of times each unique word appeared in a text and I would end up with a matrix of:

1000 x 10,000 i.e. for each text a count of the number of times each unique word appears (text as rows and unique words (features) as columns).

I would then normalise this and use equilateral encoding (similar to one hot).

My network would then consist of 10,000 input neurons (one for each word (feature)) and then a hidden layer and then an output layer trying to predict a classification of an output column with 5 different values.

This appears to be totally wrong but it did train and pretty accurately. I'm trying to work out with one hot encoding what exactly that would mean in terms of matrix sizes, what those matrices consist of and the layout of the network.

This is also related to using word embedding as when I use this I get the following size: 1000 (number of texts), 872 (text size), 300 (vector size). I therefore have 3 dimensions so that made me think in one hot encoding I should probably have: 1000, 872, 10000 (unique words).

If someone could explain where I am going wrong here and the difference between how I would approach one hot encoded text classification vs an embedded layer using:

1000 texts, 10000 unique words, 872 text length and 300 vector size

Work with a Embedding Matrix. The Embedding Matrix Dimension = all of your unique Tokens x Vector Dimension.

Your Embedding Layer should have the Dimension WordVektor x length of your Text.

Watch this: watch this video Part 16 and Part 17. Easy explanation and short videos.


How would you improve and optimize a manual encoding like this? df.features is like a list of features in each row.
First I set all those encoded to 0, and then did this:

for index, row in df.features.iteritems():
if type(row) == str:
    if 'Academia' in row:
        df.academia.iloc[index] = 1
    if 'Churrasqueira' in row:
        df.churrasqueira.iloc[index] = 1
    if 'Cinema' in row:
        df.cinema.iloc[index] = 1
    if 'Espaço gourmet' in row:
        df.espaco_gourmet.iloc[index] = 1
    if 'Espaço verde / Parque' in row:
        df.parque.iloc[index] = 1
    if 'Gramado' in row:
        df.gramado.iloc[index] = 1
    if 'Jardim' in row:
        df.jardim.iloc[index] = 1
    if 'Piscina' in row:
        df.piscina.iloc[index] = 1
    if 'Playground' in row:
        df.playground.iloc[index] = 1
    if 'Quadra de tênis' in row:
        df.quadra_tenis.iloc[index] = 1
    if 'Quadra poliesportiva' in row:
        df.quadra_poli.iloc[index] = 1
    if 'Quintal' in row:
        df.quintal.iloc[index] = 1
    if 'Salão de festas' in row:
        df.salao_festas.iloc[index] = 1
    if 'Salão de jogos' in row:
        df.salao_jogos.iloc[index] = 1
    if 'Aquecimento' in row:
        df.aquecimento.iloc[index] = 1
    if 'Ar condicionado' in row:
        df.ar_condicionado.iloc[index] = 1
    if 'Conexão à internet' in row:
        df.internet.iloc[index] = 1
    if 'Depósito' in row:
        df.deposito.iloc[index] = 1
    if 'Elevador' in row:
        df.elevador.iloc[index] = 1
    if 'Garagem' in row:
        df.garagem.iloc[index] = 1
    if 'Lareira' in row:
        df.lareira.iloc[index] = 1
    if 'Lavanderia' in row:
        df.lavanderia.iloc[index] = 1
    if 'Massagem' in row:
        df.massagem.iloc[index] = 1
    if 'Mobiliado' in row:
        df.mobiliado.iloc[index] = 1
    if 'Recepção' in row:
        df.recepcao.iloc[index] = 1
    if 'Sauna' in row:
        df.sauna.iloc[index] = 1
    if 'Spa' in row:
        df.spa.iloc[index] = 1
    if 'TV a cabo' in row:
        df.tv_cabo.iloc[index] = 1
    if 'Circuito de segurança' in row:
        df.circuito_seguranca.iloc[index] = 1
    if 'Condomínio fechado' in row:
        df.condominio_fechado.iloc[index] = 1
    if 'Interfone' in row:
        df.interfone.iloc[index] = 1
    if 'Segurança 24h' in row:
        df.seguranca_24h.iloc[index] = 1
    if 'Sistema de alarme' in row:
        df.alarme.iloc[index] = 1
    if 'Vigia' in row:
        df.vigia.iloc[index] = 1
    if 'Área de serviço' in row:
        df.area_servico.iloc[index] = 1
    if 'Cozinha' in row:
        df.cozinha.iloc[index] = 1
    if 'Escritório' in row:
        df.escritorio.iloc[index] = 1
    if 'Sacada' in row:
        df.sacada.iloc[index] = 1
    if 'Varanda gourmet' in row:
        df.varanda_gourmet.iloc[index] = 1

The easiest and most efficient way to do it, is to use the One Hot Encoding ScikitLearn

This is an example:

>>> from sklearn.preprocessing import OneHotEncoder
>>> enc = OneHotEncoder()
>>> enc.fit([[0, 0, 3], [1, 1, 0], [0, 2, 1], [1, 0, 2]])
OneHotEncoder(categorical_features='all', dtype=<... 'numpy.float64'>,
       handle_unknown='error', n_values='auto', sparse=True)
>>> enc.n_values_
array([2, 3, 4])
>>> enc.feature_indices_
array([0, 2, 5, 9])
>>> enc.transform([[0, 1, 1]]).toarray()
array([[ 1.,  0.,  0.,  1.,  0.,  0.,  1.,  0.,  0.]])


TensorFlow Sigmoid activation function as output layer - value interpretation My TensorFlow model has the following structure. It aims to solve a binary classification problem where the labels are either 0 or 1. The output layer uses a sigmoid activation function with 1 output.

model = keras.Sequential([
    layers.Dense(10, activation='relu', input_shape=[len(train_dataset.keys())]),
    layers.Dense(1, activation='sigmoid')
  ])
  optimizer = 'adam'

  model.compile(loss='binary_crossentropy',
                optimizer=optimizer,
                metrics=[tf.keras.metrics.Precision(), tf.keras.metrics.Recall(), tf.keras.metrics.Accuracy()])

The output given is an array of dtype=float32 numbers that lie between 0 and 1.

array([[9.5879245e-01],
       [3.6847022e-01],
       [3.4174323e-04],
       ...,
       [2.6283860e-03],
       [3.2045375e-04],
       [1.0798702e-03]], dtype=float32)

The Tensorflow tutorials state that
"Using the sigmoid activation function, this value is a float between 0 and 1 and represents a probability, or confidence level".
https://www.tensorflow.org/tutorials/keras/text_classification

My question is:

Do I interpret the float values from output as:

"How likely it belongs to the first class label - in this case class 0 is my first class label?" e.g. model.predict() yields 0.99998 and therefore has a 99% chance of belonging to my first class label (class 0) and 1% belonging to the other class (class 1).

or

"The closer the output from model.predict() is to 0.0 the more likely it is class 0 and the closer the output is to 1.0 the more likely it is class 1"

In short, value of model.predict() function is interpreted as mentioned in option 2.

In order to clarify, let's assume we are talking about spam detection application. Label 0 represents that text/email is not spam and label 1 represents that text/email is spam.

Suppose, after running the function model.predict(), we get value 0.9899. Then we can interpret it as:

chance of given text/email being spam is 98.99%
chance of given text/email not being spam is 1.01%

Usually, when we talk about a binary classification problem, we will often use labels 0 & 1 to represent the classes. So the predicted value being closer to 0 means it is more likely to fall under the class 0 (in our example class 0 represents text/email not being spam.). Likewise, if the predicted value being closer to 1 means it is more likely to fall under the class 1 (in our example class 1 represents text/email being spam.)


How to calculate NDCG in recommendation system This is a question about NDCG, which is a recommendation evaluation metric.

The following are being used as evaluation indicators for recommendations.

$$DCG = r_1 + \sum\limits_{i=2}^{N}\frac{r_i}{log_2i}$$
$$nDCG = \frac{DCG}{DCG_{perfect}}$$

The code is as follows:

def dcg_score (y_true, y_score, k = 20, gains = "exponential"):
    """Discounted cumulative gain (DCG) at rank k
    Parameters
    ----------
    y_true: array-like, shape = [n_samples]
        Ground truth (true relevance labels).
    y_score: array-like, shape = [n_samples]
        Predicted scores.
    k: int
        Rank.
    gains: str
        Whether gains should be "exponential" (default) or "linear".
    Returns
    -------
    DCG @k: float
    """
    order = np.argsort (y_score) [::-1]
    y_true = np.take (y_true, order [: k])

    if gains == "exponential":
        gains = 2 ** y_true-1
    elif gains == "linear":
        gains = y_true
    else:
        raise ValueError ("Invalid gains option.")

    # highest rank is 1 so +2 instead of +1
    discounts = np.log2 (np.arange (len (y_true)) + 2)
    return np.sum (gains / discounts)

def ndcg_score (y_true, y_score, k = 20, gains = "exponential"):
    """Normalized discounted cumulative gain (NDCG) at rank k
    Parameters
    ----------
    y_true: array-like, shape = [n_samples]
        Ground truth (true relevance labels).
    y_score: array-like, shape = [n_samples]
        Predicted scores.
    k: int
        Rank.
    gains: str
        Whether gains should be "exponential" (default) or "linear".
    Returns
    -------
    NDCG @k: float
    """
    best = dcg_score (y_true, y_true, k, gains)
    actual = dcg_score (y_true, y_score, k, gains)
    return actual / best

Assumes k = 5.

At this time, how should NDCG calculate for items that could not be recommended within kth?

For example,

y_true = [5,4,3,2,1]

y_score = [0,0,0,0,0] # 0 means we could not recommend within the top 5

At this time,

>>> np.argsort ([0,0,0,0]) [::-1]
array ([3, 2, 1, 0])

So, following the above code,

NDCG @ 5 = 1.0

This looks strange.

In such a case, should the score be 0 and not be included in the NDCG score calculation?

If you have any references, just showing them is fine with me.

Thank you.

IMHO,
The fundamental definition of DCG is that it is a measure of ranking quality. This assumes that you have computed the utilities of each document/item and ranked them in a certain order.
With this definition in mind, if you have n-items with same utility (which is 0 in your case), computing NDCG to measure the ranking quality within this subset of items (since you are only looking at items 5, 4, 3, 2 and 1, all of which are not recommended), will yield you a NDCG score of 1 - since your ranking is perfect if you are only looking at these items.
NDCG is merely a way to quantify the quality of ordering, i.e., current order Vs perfect order (items sorted w.r.to their utilities). This is meaningless if you are looking ONLY at items with same utility score.
I hope this answers your question.


Mixed types of data for clustering I have the following types of data for clustering - Numeric, Categorical and Latitude Longitude data for a location in one dataframe in python. I would like to know how can I go about doing clustering when the data is mixed to this degree.

Can I DBSCAN or hierarchical clustering and what do I need to do to convert categorical data to numeric.  Same with geo location data.

You don't need to convert the attributes to numeric data for DBSCAN nor for HAC (hierarchical clustering).

What you need is a distance function.

While there are some such as Gower's, these are just heuristics.

If you want really good results, you need to carefully design a proper distance function that is able to quantify how similar records are for your purpose. This is very much usage dependent, and hence you cannot find the answer already solved for you.


How to restrict the columns to be passed to final classifier in PMML Pipeline I am working on building XGBoost PMML using SKLearn and SKLearn2PMML.
I am having some numerical,somecategorical and datetime columns from which i am creating new feature inside the pipeline. When i am trying to train the model, it gets failed as the original categorical features also gets passed to the final classfier by default. Is there any way to restrict the features by specifying the feature names ?

After digging down too much and some help from sklearn2pmml creator, I
managed to filter the final columns to be passed to the classifier.
Note : Here recorder is DataFrameMapper object.

1.Getting categorical column indexes.
cat_cols = [recorder.transformed_names_.index(c) for c in categoricalCols if c in recorder.transformed_names_]

2.Adding ColumnTransformer to filter those column with the help of their indexes.
pipeline = PMMLPipeline([
    ("mapper", recorder),
    ("select", ColumnTransformer([("drop", "drop", cat_cols)], remainder='passthrough')),
    ("classifier", xgb.XGBClassifier())
])

3.Fitting the Data to the pipeline.
pipeline.fit(X_train,y_train)

4.Creating PMML file out of Pipeline.
out_file = "XGBoost.pmml"
sklearn2pmml(pipeline, out_file)


Question on reservoir sampling I have a general question on reservoir sampling. When I use this method to sample a very large dataset for training machine learning classification algorithms, I am curious as to how to make my pipeline robust to fluctuations in  class distribution across samples.

For example, suppose I am working on a binary classification problem. I want to sample a relatively small subset of data on which to evaluate my algorithms, so I used reservoir sampling. However, it seems to me (thought I could be wrong) that by chance I may draw a sample wherein the class distribution is significantly different from that seen in the larger population of data. If this is a correct inference, can anybody tell me how to remedy the situation? If reservoir sampling is not the answer, what other procedure are out there that I can leverage in this case (apart from using distributed environment)?

I am learning, so some explanation, hint, and direction would be greatly appreciated.

Consider the class to be the variable that you are sampling.

Your "reservoir sample" should still be as good as uniformly drawn from your data.

Yes, there may be fluctuations, in particular if you have small samples. If you sample a single observation, the class distribution in that sample will be 100% of one class, there is no way around that. ;-)


How to apply a trained Random Forest model to a new data set in R? So I have a data set that is essentially football players statistics in 2017 and 2018. I have trained my model to use the 2017 data to predict the 2018 number of touchdowns. My code is below:

set.seed(1)
data.rf <- randomForest(2018_td ~ ., data = data, proximity = TRUE)

In my  data set, I had the actual # of touchdowns in 2018, and trained a random forest algorithm to predict that value. Now, I want to apply the trained random forest on the same 2018 data set, but to predict 2019 # of TD, that I don't have.

I'm not sure if I'm missing something or if I have a fundamentally wrong understanding of how RF works. How would I go about predicting those 2019 values from my data.rf model?

You need to setup another data frame that has the unlabeled 2019 observations. Assuming you have multiple predictors, your new data would have the exact same columns as your 2018 data but with no 2019_td column (since you presumably don't know anything about the 2019 season). That is, the players in 2019 that you wish to predict on, along with their predictors, but no target column.

Once you have the data frame, you can use predict(trained_model, newdata) to score your new predictions.

Edit: As an aside, you have data that is influenced by time. What happens if a player gets injured next season? What about trends in the players performance?


In a random forest, are all decision trees given same priority? In random forest algorithm, m (say) number of decision trees are generated using p ( $log_2 n$ +1, where n= number of features) randomly selected features. The label of any sample from test data is chosen by the number of votes by the result of these decision trees. But these decision trees result in different accuracy ( quite obvious). My question is, why should we give same priority to all the decision trees? If a decision tree has higher accuracy in predicting the labels of the test samples, shouldn't we give it more priority than those with lower prediction accuracy ?

As you mentioned, each decision tree is trained on p (sometimes sqrt(p) random features). This ensures that each tree is “grown” (trained) differently so that the model 1. Does not overfit the training data (reduces variance) and 2. Generalizes better to new data (reduces bias). Therefore we don’t weigh the trees differently, as this would be similar to having all tress trained on the same features.

You can change the voting threshold however from the standard 50% to anything you want (e.g 40%, 70%, 90%) which will change the precision and recall accuracies of the model.

EDIT: changing the voting threshold means changing the number of trees needed to make a classification. For example, in binary classification most standard random forests require 50% or more of the trees to vote for a class for that class to “win” (be predicted to be that class). But if you change this threshold, to say 70%, then 70% or more of the trees need to vote for the same class for that class to win.


xgboost speed difference per API How can it be that a xgboost.cv cross-validation operation where n-folds are evaluated is quicker than a single XGBoostClassifier.fit(X,y) of the xgboost.sklearn API?

I believe this is the answer: https://github.com/dmlc/xgboost/issues/651
the sklearn api uses n_estimators= 100 as default whereas xgb.train is using n_boost_rounds=10

As both refer to the same parameter this could explain the huge difference.


Is it normal for F1 scores to be lower on a binary classification task as compared to a 3-class classification task? I am trying to understand if the F1 scores are higher for a binary classification problem than for a multiclass classification problem.

In general the lower the number of classes the easier it is for a classifier to assign the right category. However this completely depends on the characteristics of the data, in particular how well the features match the classes.

For example if one tries to somehow classify pictures of dogs, cats and rabbits into two classes, it's possible that the performance will be poorer than into 3 classes. This is an obvious example but sometimes the data might contain patterns which fit more easily into 3 groups than 2.


How to increase model's prediction accuracy I am using the InceptionV3 model for training. Here is the link for the code (https://github.com/maxmelnick/tensorflow/blob/no_random/tensorflow/examples/image_retraining/retrain.py) Initially I have a small size dataset. So, I used the augmentation technique to increase the size of the dataset.
While training phase dataset was divided into training, validation, and testing. During the training phase, it shows 96% accuracy for 11 classes. But When I predict any new input image(Unseen data) it gave 56% accuracy. What's the problem lies with the model?
I have already used Dropout, Cross-validation, OverSampling techniques but not achieved good results over the new input image.
Parameters used while training.
Training Samples - 800 images in each class

Training Samples - 70%
Validation Samples - 20%
Testing Samples - 10%

Testing Samples (Unseen data other than Training Samples) - 51 images in each class
Epochs - 10,000

It sounds like you’re overfitting the training data, and so you should research some approaches to regularisation of neural networks. I see you’ve used dropout, which is one of these, but may not be sufficient.
You could also consider dividing your training data into a training set and a validation set and halting training when performance on the validation set stops improving. This is called early stopping


Pandas DataFrame Rollup Error I am trying to compute Percentage Contributions on a pandas dataframe. I have a data frame which has three columns (Buyer, Product, CountofProductBought).

I create a new dataframe using groupby at Buyer level and using sum on the Count of Products Bought. I use the as_index=False.

Finally, I merge this data frame with the original data frame to compute the % [count of Product Bought/ SUM of count of Product Bought].

I get the following error when I call pd.merge:

  The column label 'Buyer' is not unique. For a multi-index, the label
  must be a tuple with elements corresponding to each level.

Please help as there is no concrete help on this.

From my understanding you have dataframe containing a list of Buyer ID's, the product they bought and how many of it they bought. You want to find out what percentage of the total of each product bought each customer was responsible.

For instance if buyer 1 bought all of A then they would 100% and if buyer 2 bought half of all B then they would be 50%.

Each buyer only buys one type of product at a time.

I generated a dataset to replicate this as so:

options = 'ABCDEF'

product = []
count = []
buyer = []

for i in range(0,100):
    product.append(test[np.random.randint(0,5)])
    count.append(np.random.randint(0,10))
    buyer.append(i)

df = pd.DataFrame(data = [buyer, product, count]).transpose()

df.columns = ['buyer','product','count']
print(df.head())
>>> buyer product count
0     0       B     4
1     1       B     6
2     2       A     2
3     3       D     2
4     4       D     5

In order to calculate each customers percentage you will need the total count of each product bought

totals = {'A' : 0,
         'B' : 0,
         'C' : 0,
         'D' : 0,
         'E' : 0,
         'F' : 0}

for i in range(0, len(df)):
    currentProduct = df.iloc[i,1]
    totals[currentProduct] = totals[currentProduct] + 1

Now all you need to do is calculate count/total for each product and save to a new column in the dataframe.

for i in range(0, len(df)): # Iterate over every row in
    # new column | count bought | total bought by all
    df.iloc[i,3] = df['count'][i]/totals[df['product'][i]]
print(df.head())
>>> buyer product count  contribution
0     0       B     4      0.210526
1     1       B     6      0.315789
2     2       A     2      0.080000
3     3       D     2      0.111111
4     4       D     5      0.277778
print(totals)
>>> {'A': 25, 'B': 19, 'C': 15, 'D': 18, 'E': 23, 'F': 0}

I know it's not the same method that you where using but it should still work, if you want please post a sample of your code so that we can look bugs, alternatively just tick this answer and change the title to represent the question that was answered.

If i misunderstood anything just say so


Gradient of NN output with respect to inputs I've trained a neural network (NN) on a problem where multiple inputs can be mapped to the same output. I'd like to use this NN to go from an output to an input i.e. given an output vector $y$, I want to find an input $x$ such that the NN returns some $z$ close to the given output $y$ when fed an input of $x$.

I was thinking of using gradient descent to do this. Do any of the common deep learning APIs let you take gradients of NNs with respect to their inputs?

I've looked around and haven't found anything, but figured I'd check here before moving forward.

Yes there is. This is called "back-propagation to the input". I would like to invite you to read this awesome blog which relies on lucid. You will see some code on their notebooks.


R - Bayesian network for satisfaction survey data I'm trying to build a bayesian network for satisfcation survey data.
My data is made of 13 questions about services, products etc... each customer can answer from 1 (Very unsatisfied) to 4 (very satisfied) with no "neutral feeling"). There is an other question about the overall satisfaction where the customer can answer 1 (very unsatisfied), 2 (unsatisfied), 3 (satisfied), 4(very satisfied) and 5 (totally satisfied).
I plan to use bnlearn to build the bayesian network. The goal is to identify important features who lead customers to give 5 on overall satisfaction item.
In your opinion, what is the best way to recode the overall satisfaction : 0 for 1, 2, 3, 4 and 1 for 5 (totally satisfied)? I tried but results let me skeptical.

Thanks for your help.

Best regards,

I see the task is to understand the pattern rather than developing a prediction model.
I would use regression trees rather than Bayesian network.


How to get common values between two multi sets? I am working in R. I have two vectors; A and B of lengths 5913 and 3733 respectively (with entries are repeated). I want to extract those values (with repetitions) that are present in both A and B.
I have done (A %in% B) (lets call it C) and (B %in% A) (lets call it D).
The length of C is 3906 and that of D is 3607 (so 2007 elements in A are not in B and 126 elements in B are not in A).

But How do I find the common values ? I don't think I can use intersect() method as it expects vectors with no duplicate values. I have many many repetitions.

Note: Due to the length of these vectors, I couldn't mention them here.

Any help would be greatly appreciated.

Thanks

Since you have multiple repetitions in A and B, and these two vectors are of different lengths (5913 and 3733), it is expected that vector C and D are of different size. However, C and D might contain the same unique elements.

Let's take the example proposed by Biswajit Banerjee:

A = c(1,1,2,2,3,3,4,4,5,5,6,6,9,11,15,17,19,19)
B = c(1,2,2,2,3,3,3,4,5,6,7,8,8,8)

C= A[(A %in% B)]
D = B[(B %in% A)]

all(unique(C) == unique(D))
TRUE

You could verify that your vectors C and D of length 2007 and 126 contains the same unique values. Does it solve your problem ?


Classify ciphertext vs. plaintext I'm attempting a rather simple exercise in machine learning and trying to classify samples of text as either plaintext or ciphertext (encrypted).

Here are two samples:

Plaintext: This is a sentence in plaintext which any human person can read
Ciphertext: 5oXbLiEZbMUgOOdYy+q4+rsDaqUngBrrUbpVeuu2ggvP6hHObC4GgTLhq

The specific encryption used doesn't have any special attributes I can use to classify (for example, the ciphertext isn't guaranteed to be significantly longer than plaintext), so the task is all about figuring out what text is indistinguishable from random characters and what text is readable plaintext.

My current heuristic involves counting whitespaces and assuming anything with a whitespace ratio above a certain threshold is plaintext, but I'm trying to find a better robust algorithm.

Simply counting the frequency of characters ought to easily distinguish between English language and ciphertext, because they're so obviously different.

You can just count the frequency of characters in a big corpus of English, and a big corpus of ciphertext, and apply a chi-squared test to each to figure out which one matches the counts in a new chunk of text.

Or if you can assume ciphertext has a roughly uniform distribution over characters, that alone lets you construct a good test for whether new text is unlikely to be ciphertext.

I did a short blog post on something similar. https://blog.cloudera.com/blog/2016/09/solving-real-life-mysteries-with-big-data-and-apache-spark/


Intuitive explanation of Noise Contrastive Estimation (NCE) loss? I read about NCE (a form of candidate sampling) from these two sources:

Tensorflow writeup

Original Paper

Can someone help me with the following:

A simple explanation of how NCE works (I found the above difficult to parse and get an understanding of, so something intuitive that leads to the math presented there would be great)
After point 1 above, a naturally intuitive description of how this is different from Negative Sampling. I can see that there's a slight change in the formula but could not understand the math. I do have an intuitive understanding of negative sampling in the context of word2vec - we randomly choose some samples from the vocabulary V and update only those because |V| is large and this offers a speedup. Please correct if wrong.
When to use which one and how is that decided? It would be great if you could include examples(possibly easy to understand applications)
Is NCE better than Negative Sampling? Better in what manner?

Thank you.

Taken from this post:https://stats.stackexchange.com/a/245452/154812

The issue

There are some issues with learning the word vectors using an "standard" neural network. In this way, the word vectors are learned while the network learns to predict the next word given a window of words (the input of the network).

Predicting the next word is like predicting the class. That is, such a network is just a "standard" multinomial (multi-class) classifier. And this network must have as many output neurons as classes there are. When classes are actual words, the number of neurons is, well, huge.

A "standard" neural network is usually trained with a cross-entropy cost function which requires the values of the output neurons to represent probabilities - which means that the output "scores" computed by the network for each class have to be normalized, converted into actual probabilities for each class. This normalization step is achieved by means of the softmax function. Softmax is very costly when applied to a huge output layer.

The (a) solution

In order to deal with this issue, that is, the expensive computation of the softmax, Word2Vec uses a technique called noise-contrastive estimation. This technique was introduced by [A] (reformulated by [B]) then used in [C], [D], [E] to learn word embeddings from unlabelled natural language text.

The basic idea is to convert a multinomial classification problem (as it is the problem of predicting the next word) to a binary classification problem. That is, instead of using softmax to estimate a true probability distribution of the output word, a binary logistic regression (binary classification) is used instead.

For each training sample, the enhanced (optimized) classifier is fed a true pair (a center word and another word that appears in its context) and a number of kk randomly corrupted pairs (consisting of the center word and a randomly chosen word from the vocabulary). By learning to distinguish the true pairs from corrupted ones, the classifier will ultimately learn the word vectors.

This is important: instead of predicting the next word (the "standard" training technique), the optimized classifier simply predicts whether a pair of words is good or bad.

Word2Vec slightly customizes the process and calls it negative sampling. In Word2Vec, the words for the negative samples (used for the corrupted pairs) are drawn from a specially designed distribution, which favours less frequent words to be drawn more often.

References

[A] (2005) - Contrastive estimation: Training log-linear models on unlabeled data

[B] (2010) - Noise-contrastive estimation: A new estimation principle for unnormalized statistical models

[C] (2008) - A unified architecture for natural language processing: Deep neural networks with multitask learning

[D] (2012) - A fast and simple algorithm for training neural probabilistic language models.

[E] (2013) - Learning word embeddings efficiently with noise-contrastive estimation.


Which parameters are hyper parameters in a linear regression? Can the number of features used in a linear regression be regarded as a hyperparameter? Perhaps the choice of features?

I like the way Wikipedia generally defines it:

  In machine learning, a hyperparameter is a parameter whose value is
  set before the learning process begins. By contrast, the values of
  other parameters are derived via training.

On top of what Wikipedia says I would add:

Hyperparameter is a parameter that concerns the numerical optimization problem at hand. The hyperparameter won't appear in the machine learning model you build at the end. Simply put it is to control the process of defining your model. For example like in many machine learning algorithms we have learning rate in gradient descent (that need to be set before the learning process begins as Wikipedia defines it), that is a value that concerns how fast we want the gradient descent to take the next step during the optimization.

Similarly as in Linear Regression, hyperparameter is for instance the learning rate. If it is a regularized Regression like LASSO or Ridge, the regularization term is the hyperparameter as well.

Number of features: I would not regard "Number of features" as hyperparameter. You may ask yourself whether it is a parameter you can simply define during the model optimization? How you set the Number of features beforehand? To me "Number of features" is part of feature selection i.e. feature engineering that goes before you run your optimization! Think of image preprocessing before building a deep neural network. Whatever image preprocessing is done is never considered hyperparameter, it is rather a feature engineering step before feeding it to your model.


Should we first remove the unwanted independent variable or split the data into test train data? I wanted to know if we are dropping any independent variable as it has too many missing values(~75% or more) from the data then should we do it before splitting the data or after splitting the data

Whether it's removed before/after splitting doesn't matter: the variable will not appear in any part of the data so it cannot cause any data leakage anyway.

However what might matter (at least in theory) is from investigating which part of the data was the decision made. Any preprocessing and design decision should be made based on the training set only in order to avoid any risk of data leakage (see example). In practice it's not really a problem in the case you mention.


ML: Classification Model Comparison Given is a dataset that I need to use for a classification and I want to compare the performance of different classification models.
Let's assume, I want to look at logistic regression (with different cut-off-points) and KNN.
Is there anything problematic if I proceed as follows:

Split data in training and validation data (and a test set for the performance evaluation of the winning model).
Train a logistic regression model and a KNN classification model on the training set.
I consider for each cut-off point t between 0 and 1 the logistic regression model as a classification model - so the regression model leads to many classification models.
I now compare for a certain range of t (lets say 0.01 to 0.99) the classification performance of all my classification models (logistic regression for those t and KNN) on the validation data.
The one with the best performance (based on a certain metric) I'll choose.

I was discussing this with somebody else who argued that t needs to be considered as hyperparameter and this parameter needs to be tuned separately.
If this is true - why? And what's wrong with my arguments above?

In principle you can use your approach.

However, you should not optimize on your test set (step 3). Instead you should select the best t using your validation set.
Then you compare it against KNN, also on the validation set. Finally, the best model should be evaluated on the test set.


CIFAR-100: What is the difference between vehicles 1 and vehicles 2? The superclasses in the CIFAR-100 dataset are mutually exclusive and all but the vehicle ones are quite well defined by its label.
Example: It is very clear why bees belong to the superclass insects and none of the other superclasses.
This appears not to be the case for the two superclasses vehicles 1 and vehicles 2: The two appear not to be clearly separable if we ignore their subclass labels.
Example: It is not clear why pickup trucks belong to the category vehicles 1 and not vehicles 2.
Clearly, vehicles 1 vs. vehicles 2 are not distinguished by size, number of wheels, number of passengers, ….
Hence, my question is: What distinguishes classes vehicles 1 and vehicles 2?
As a reminder, these are the two vehicle-classes:

vehicles 1:   bicycle, bus, motorcycle, pickup truck, train
vehicles 2:   lawn-mower, rocket, streetcar, tank, tractor

The difference between vehicles 1 and vehicles 2 is that vehicles 1 are more common and vehicles 2 are less common.

CIFAR-100 dataset super/coarse labels are not very rigorous. The CIFAR-100 official documentation points out flaws in their taxonomy:

  fruit and vegetables: apples, mushrooms, oranges, pears, sweet peppers
  large carnivores: bear, leopard, lion, tiger, wolf
  …
  Yes, I know mushrooms aren't really fruit or vegetables and bears aren't really carnivores.


Sequence models word2vec I am working a data-set with more than 100,000 records. This is how the data looks like:

email_id    cust_id campaign_name
123         4567     World of Zoro
123         4567     Boho XYz
123         4567     Guess ABC
234         5678     Anniversary X
234         5678     World of Zoro
234         5678     Fathers day
234         5678     Mothers day
345         7890     Clearance event
345         7890     Fathers day
345         7890     Mothers day
345         7890     Boho XYZ
345         7890     Guess ABC
345         7890     Sale

I am trying to understand the campaign sequence and looking for the next possible campaign for the customers.

Assume I have processed my data and stored it in 'camp'.

With Word2Vec-

from gensim.models import Word2Vec

model = Word2Vec(sentences=camp, size=100, window=4, min_count=5, workers=4, sg=0)

The problem with this model is that it accepts tokens and spits out text-tokens with probabilities in return when looking for similarities.

Word2Vec accepts this form of input-

['World','of','Zoro','Boho','XYZ','Guess','ABC','Anniversary','X'...]

And gives this form of output -

model.wv.most_similar('Zoro')
[Guess,0.98],[XYZ,0.97]

Since I want to predict campaign sequence which occurs more frequently in combination with target word, I was wondering if there is anyway I can give below input to the model and get the campaign name in the output

My input to be as -

[['World of Zoro','Boho XYZ','Guess ABC'],['Anniversary X','World of
Zoro','Fathers day','Mothers day'],['Clearance event','Fathers day','Mothers
day','Boho XYZ','Guess ABC','Sale']]

Output -

model.wv.most_similar('World of Zoro')
[Sale,0.98],[Mothers day,0.97]

I am also not sure if there is any functionality within the Word2Vec or any similar algorithms which can help finding the next possible campaign for individual users.

Word2Vec operates on words and you want to compare 'texts' (series of words of varied length). For that, doc2vec might more appropriate.
You have very short 'texts' (names of campaigns) so generating embedding from them (only from them) want give greate effects. You could start with some pretrained vectors but anyway probably you want achieve much more. But the questions is:
It's not sure from your examples and explanation what will be nature of texts you want to find the most similar campaign name with. On one side you are writing about finding the one most frequent with a given word - then you could just create statistics of words and campaigns. On the other side, in your example, you pass whole text - then above mentioned apply, finding most similar text using text vectors.


How to transform features order to make their distribution normal? After sns.pairplot(df) command, I got this picture:

The question is what transformation to use in order to make the distributions normal?

Just looking at a pairplot isn't enough to suggest changes, in my opinion. Moreover, what are you trying to test? What is your null hypothesis, what is the alternative hypothesis? What kind of data is it, how neat was your collection process? Are there any co-variables?

With the info provided, all I can suggest is to perform parametric and non-parametric tests. May try Box-Cox transformation!


Continuous values in a dataset In a data analysis continuous data needs to be discretised, however in a computer it's impossible to have values $\in \mathbb{R}$, so what does this mean?

First of all, it is not a general rule that continuous values should be discretized. Especially in the context of regression, it is actually very much wrong.

Regarding discretization, I believe you may have a wrong definition. Discretizing means transforming a continuous interval, say $[0,10]$ into sub intervals, $[0,1), [1,2),...,[9,10]$, and then attributing the same value to points in the same interval, for instance, $\forall x \in [0,1) \mapsto 0.5$.

I'm not sure what is your point have having value $\in \mathbb{R}$. If it's because you don't know what the interval you have is, then you can simply find the minimum and maximum of your data points, and assume these to be the total interval. Then you can discretize that interval in however many pieces you want.


parsing dictionary to JSON I used the following code to pass a json string as a parameter in requests.post method,
import json
data = {'Key': 'Value',"title": "hello world"}
# get string with all double quotes
json_string = json.dumps(data)
json_string

output is
'{"Key": "Value", "title": "hello world"}'

How to remove the single quotes around the dictionary without using
print(json_string). So that I can pass this in requests.post like this
results=requests.post(url=url,data=json_string)

If I give the json_string without removing single quotes, it gives me internal server error.

You don't pass a string to requests.post's data argument, you pass a dictionary. So basically, instead of converting your dictionary to a string, you just do:
requests.post(url, data=data)

[EDIT]: From my experience, it is better to leave requests take care of everything, instead of using requests.post(url, data=data), you can use requests.post(url, json=data), which will format the header correctly.
See here for the doc.


Mismatch between optimum features and grid scores using RFECV? I have a dataset with 5K columns focused on binary classification. I have more than 60 columns. I am trying to find the best features through RFECV approach. Though it produces 30 optimum features, when I plot in graph, I only see 12 features. Please see my code and plot below

model  = RandomForestClassifier(n_estimators=100, random_state=0)
model_b = LinearSVC(class_weight='balanced',max_iter=1000)
# create the RFE model and select 15 attributes
rfe = RFECV(model,step=5,cv=5)
rfe = rfe.fit(X_train_std, y_train)
# summarize the selection of the attributes
feat = rfe.support_
fret = rfe.ranking_
features = X.columns
print(rfe.n_features_) # this returns 30 as output

print(rfe.grid_scores_) this produces the below output

I was expecting to see the grid scores for 30 features and in the plot, I was expecting to see the x-axis to have 30 features as well. But it shows only for 12 features. Similarly if I had only 19 features in my dataset, RFECV returns all 19 as optimal feature which is fine. But again in grid score it only shows 4

q1) Does this mean beyond 12 features, there is no increase in model accuracy?

q2) I assume grid_scores are nothing but weightage/rank which indicates the influence that a feature has on outcome. But how do I get the name of this 12 features?

q3) Wwhy does it show optimum no of features as 30 but grid scores is shown only for 12.

Can you help me with these question please?

RFE stands for Recursive Feature Elimination, meaning that the search starts off with a full set of features and with each step some features (in your case, 5) are dropped in an attempt to improve the predictive power of the model. If we take a closer look at sklearn's RFECV reference, it states:

  grid_scores_ : array of shape [n_subsets_of_features]

  The cross-validation scores such that grid_scores_[i] corresponds to the CV score of the i-th subset of features.

The i-th subset of features being referred to is the group of features being evaluated at that particular step of the search. Judging by your output (and the plot), only 12 steps were needed to figure out which 30 features are optimal and that no further improvements could be found using the chosen approach - that's why you see an increase in CV score followed by a plateau. Your assumption in q1 is almost correct, but you are looking at algorithm steps, not features. This is simply an issue of misinterpretation, so is q3.

As for q2, you need to look at support_ attribute of the RFECV object. It returns a boolean mask that corresponds to X.columns (if True, then that feature is selected, if False, then not). You can also look at ranking_, but it provides information on the "goodness" of discarded features only (all selected features are ranked 1). The importances of selected features are model-specific. For RandomForestClassifier, you can look at feature_importances_. For LinearSVC, look at coef_.

Hopefully this answers your question. Good luck with the selection!


Naive Bayes for Multi label text classification How to use Naive Bayes for multi-label text classification in R.

I tried using naiveBayes() from e1071 library but it seems that while training, it doesn't accept multi-label class variable.

I created TermDocumentMatrix using the text document corpus and tried to create the model using this matrix and class variable(list of topics a particular document belongs to). Below is the code that I used.

trainvector <- as.vector(traindata$bodyText)

trainsource <- VectorSource(trainvector)

traincorpus <- Corpus(trainsource)

trainmatrix <- t(TermDocumentMatrix(traincorpus))

model <- naiveBayes(as.matrix(trainmatrix), as.factor(traindata$topics))

The last line gives below error:

Error in sort.list(y) : 'x' must be atomic for 'sort.list'

Have you called 'sort' on a list?

I tried using

model <- naiveBayes(as.matrix(trainmatrix), as.factor(unlist(traindata$topics)))

but got error:

Error in tapply(var, y, mean, na.rm = TRUE) :
  arguments must have same length

Since generally, Naive bayes alogorithms implemented by most libraries don't support multilabel classification. Although You can devise your own algorithm taking inspirations from naive bayes approach.

For a particular class c, and document d (with 3 words w1,w2,w3)

'/' stands for 'given' p(a/b) = probability(a given b)

As per bayes theorem.

p(c/d) = p(c,d) / p(d)

p(c/d) = (p(c)*p(d/c)) / p(d)

where:

p(d/c) = p(w1/c)*p(w2/c)*p(w3/c)

Since words are assumed to be independent of each other.

And p(w1/c) can be obtained using your code, calculating count of w1  inside class c documents divided by overall count of w1 in all documents or you can use your own logic

But If want to avoid writing detailed code

you can restructure your input data to achieve multilabel classification. such that a given document d with n labels/classes (eg d labeled with c1,c2,c3) get expanded into data of n samples (3 here) of same document d with different label each time ((d,c1),(d,c2),(d,c3)). Now you can pass this data into any R/python library supporting multinomial naive bayes. Same needs to be done in your dataset as well. Currently, you are passing traindata$topics which is y variable (training labels) as it is, without modification.

Even after you train your data using this approach. You need to use a probability threshold e.g 0.2, so that class labels with probability above 0.2 will be assigned to that test document.

More better Approach which requires restructuring your input

If you have 'n' class labels then you can train 'n' different binary naive bayes classifier for each of the class. For example, for training a classifier for class 'c1', you train a naive bayes classifier with dependent  variable Y denoting the presence of class c1 on that document as '1' and absence as '0'.

After you train 'n' binary naive bayes classifier using this approach. You will now use the output of these n classifiers. e.g if out of these, n classifiers, if a particular classifier, which corresponds to class 'c1' has output probability above 0.5, then class label 'c1' will be assigned to this test document.


Should I take random elements for mini-batch gradient descent? When implementing mini-batch gradient descent for neural networks, is it important to take random elements in each mini-batch? Or is it enough to shuffle the elements at the beginning of the training once?

(I'm also interested in sources which definitely say what they do.)

It should be enough to shuffle the elements at the beginning of the training and then to read them sequentially. This really achieves the same objective as taking random elements every time, which is to break any sort of predefined structure that may exist in your original dataset (e.g. all positives in the beginning, sequential images, etc).
While it would work to fetch random elements every time, this operation is typically not optimal performance-wise. Datasets are usually large and are not saved in your memory with fast random access, but rather in your slow HDD. This means sequential reads are pretty much the only option you have for good performance.
Caffe for example uses LevelDB, which does not support efficient random seeking. See this, which confirms that the dataset is trained with images always in the same order.


Multitask multivariate regression? I'm trying to solve a multivariate regression problem similar to PLS regression.

The problem can be described as a connectivity analysis problem where we have two regions with unknown unidirectional connections(many-to-many) and given a set of input region patterns and output region patterns, we want to infer the underlying connections.

Mathematically, the problem can be formulated as below

$Y = BX \qquad$  where $Y \in \mathbb{R}_+^{M\times N}$, $X \in \mathbb{R}_+^{L\times N}$, and $B \in \mathbb{R}_+^{M\times L}$ with $L > M >> N$

The column of $X$ and $Y$, will be a vectorized version of 2D image.

Although this would result in highly underdetermined system, I do have some prior knowledge about the pattern in input/output regions that I can incorporate in the model.

Is there a model/idea that I can use in situation like this?

You may want to model your problem using bayesian regression, which may allow you to introduce your prior knowledge in the form of priors (a priori distributions) of the model parameters. They would also allow you to model latent variables that govern the dynamics of the interactions (and impose priors on them as well).

The specific approach may be based on sampling (e.g. Markov Chain Monte Carlo) or optimization (e.g. Variational Bayes).

One of the most popular bayesian frameworks is Stan, which has bindings to R (rstan) and python (pystan). In R there are other alternatives such a BUGS and JAGS. In the python realm other options are PyMC (which is also pretty popular) or Edward.


Using Machine Learning techniques for text-analysis I am analysing a bunch of tweets and I want to understand which political party the authors support.

I am using Mathematica but I am thinking to re-write my code in Python. If you have any suggestion, do not esitate.

Let say, for example that I have five parties labelled by a number: Party1, Party2,...,Party5 and I provide a list of tweets which are for sure associated to the parties. This list is roughly trainingList = {tweet1-> "Party1",tweet2->"Party2",...,tweet5->"Party5",tweetK->"Neutral"}. Moreover, I added a class "Neutral" because some tweets are not classifiable. Actually, I provide more than one tweet par party. In fact, the trainingList is much longer.

Notice that I remove stop-words from the tweets and I apply a stemming-words algorithm to simplify the expressions. Then, every tweets is a list of potentially-significant words.

I create the classifier function, providing a prior (the explicit for of the prior is not important)

c = Classify[trainingList, ClassPriors-> prior]

Then, I want to do some checks. I take a bunch of tweets belonging to other party-supporters and try

c[{word1,word2,word3,word4,word5,word7},"Probabilities"]

which gives me a list of probabilities $p_i$. For example,

<| 'Party1' -> p1, 'Party2' -> p2, 'Party3' -> p3, 'Party4' -> p4, 'Party5' -> p5, 'Neutral' -> p6 |>

Question1 Is there a way to understand how the algorithm associate these probabilites? More specifically, how each word is associated to a party

Question2 Is there a way to see which are the most frequent n-grams of words associated to a given party (class)?

Take a few words you know are linked with e.g republicans and with democrats.
Extract their word embedding. That is the vector representation of the word which will be high dimensional.
This will assume you have trained word embedding, either pretrained, or even better trained them yourself on your data. The word embeddings could have been trained/extracted through e.g LSTM training or matrix factorisations.

On the word embeddings, do dimension reduction(PCA as an example) to 2d. Plot! Those who are close together will be "similar" for the objective they are trained for.

Pretrained word embeddings exist to download. However, they might not be the optimal embeddings for your objective, but better than nothing still.


Confusion about the MSE ERROR I created a Random Forest regressor model and calculated my own error. I also want to calculate MAE, MSE and RMSE to compare my results to similar use cases. I am confused by the results as the values are far lesser.
The random forest regressor function I used is the default one. So I am just curious if there is anyone who can interpret my MAE, MSE, RMSE? Or did I do something wrong?
# Random Forest Regressor trainieren
random_state = 42

rf.fit(X_train, y_train)

These are the results of my own error:
Testset size: 19688
mean error(AVG): 0.10146151388130142
median error (Median): 0.061079373768676316

These are the results that look confusing to me:
# predict the testset
y_pred = rf.predict(X_test)

# Random Forest Basline Model evaluation
print('Mean Absolute Error:', metrics.mean_absolute_error(y_test, y_pred))
print('Mean Squared Error:', metrics.mean_squared_error(y_test, y_pred))
print('Root Mean Squared Error:', np.sqrt(metrics.mean_squared_error(y_test, y_pred)))

#Mean Absolute Error: 0.027392195979941525
#Mean Squared Error: 0.00142917555312868
#Root Mean Squared Error: 0.03780443827288907

After giving much consideration, let's review the mechanism of Random Forest Regression (RFR):

So this idea or process of averaging models is a technique called Ensembling. Additionally, Random forest is a Supervised Learning algorithm which uses an ensemble learning method for classification and regression. Random forest is a bagging technique and not a boosting technique. It uses a multitude of decision trees at training time and outputting the class that is the mode of the classes (classification) or mean prediction (regression) of the individual trees.

If you check sklearn documentation, the metrics for evaluation are:

MSE (default) ---> Root of MSE ---> RMSE (optional)
MAE           ---> Percentage form of MAE ---> MAPE (optional)

Additionally, MSE is recommended to use on continuous data via this post
You can use the method of score(X, y):
Return the coefficient of determination R^2 of the prediction.
The coefficient R^2 is defined as (1 - u/v), where:

u is the residual sum of squares ((y_true - y_pred) ** 2).sum()
v is the total sum of squares ((y_true - y_true.mean()) ** 2).sum()

The best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse).
A constant model that always predicts the expected value of y, disregarding the input features, would get a R^2 score of 0.0.
The problem with this question is OP didn't share info about the evaluation of used metrics on the train and test datasets except for his own defined metric formula and some SKlearn metrics as follows:
#OP's defined metric #MSE
average_mean_error = (SalesPrice - predPrice) / Salesprice

result_rfbl.insert(2, "predResidualValue", rf.predict(X_test), True)
result_rfbl.insert(2, "predPrice",  round(result_rfbl["predResidualValue"] * result_rfbl["NEUPREIS"], 2), True)
result_rfbl.insert(2, "SalesPrice", round(result_rfbl["RESTWERT"] * result_rfbl["NEUPREIS"], 2), True)

# absolute prozentualer Fehler #MAPE
result_rfbl.insert(2, "ERROR", abs(result_rfbl["SalesPrice"] - result_rfbl["predPrice"])/result_rfbl["SalesPrice"], True)

I assume that his model seems to be overfitting If validation MSE higher than the training MSE and get a very decent result with the training MSE nevertheless I don't want to judge by the cover of the book. If it is the case, RFR-model needs to apply feature engineering or RFR-analysis and Regularization like Ridge (L2 regularization) & Lasso (L1 regularization).
To understand better the reason behind the difference between manual defined MSE and Sklearn MSE, I draw your attention to minimum weighted MSE calculation for the first Split of a Decision Tree on SalesPrice from this source.


R: comparison (6) is possible only for atomic and list types I am getting the error in the subject line for the following piece of code in R.

r<- matrix(c(2,4, 1,5,5,3, 4,5,1,2,3,4), nrow=3)
for (u in 1:nrow(r))
  for (x in 1:ncol(r))
    r_hat<- function(u,x) {

      if (r_hat >r[u,x]) {
        print(r[u,x]-r_hat)
      }
    }

How can I get rid of the error?

The code works and leads to no error. The result is a definition of one variable rand one function r_hat.

You get the same with this snippet (I removed the loop as not relevant, what you do is creating the same function repeatedly in a loop)

> r<- matrix(c(2,4, 1,5,5,3, 4,5,1,2,3,4), nrow=3)
>
> r_hat<- function(u,x) {
+       if (r_hat >r[u,x]) {
+         print(r[u,x]-r_hat)
+       }
+ }

The problem is in the function, after calling it you get the mentioned error.

> r_hat(0,0)
Error in r_hat > r[u, x] :
  comparison (6) is possible only for atomic and list types
>

You reference r_hat within the function, are you trying to define a recursive function? Than the recursive call should have the parameters and the function some return value...

My guess is you tries to compare the function definition with an element of a matrix, which leads to this error.

If this is not enough for you and you explains what the code should deliver, I'll try to complete my answer.


Loss function for sparse tagging I am writing a musical transcription system with a RNN (LSTM).

Input: 1 vector of features per timestep (about 40 timesteps in a second)
Output: 1 binary vector of notes per timestep (dimension=36) (1 is on, 0 if off).
Model : LSTM(512) + LSTM(256) + Dense(36, activation='sigmoid')

I am currently training on monophonic data (i.e. my output has at most one 1 per timestep).

The problem is that because of the sparsity the best strategy to use is to always return 0.
I tried the loss functions 'mean_squared_error', 'mean_absolute_error', 'binary_crossentropy', 'cosine_proximity' and one custom I wrote with keras :

K.sum(K.abs(y_pred - y_true), axis=-1) * K.mean(y_pred, axis=-1) / K.maximum(K.mean(y_pred * y_true, axis=-1), K.epsilon())

All those function led with sufficient training to the always zero output.

I can either change my loss function or my encoding, but the problem is that I need to support polyphonic data, i.e. when there is more than one class to select.

Or can I train as many LSTM as there are notes (88 for a piano keyboard), and each one detects if its note is played but I think this is pretty much equivalent, isn't it ?

I think my data is pretty much equivalent to what keras calls categorical data (binary matrix).

Currently I have only 36 classes, and 24 are always zero.

In the case of 1 note at a time maximum you could use categorical_crossentropy as loss and add a class for when no note is played. This is a log loss on a last softmax layer. This would turn it into a 37 one-hot-encoding representation.

In the case of having the possibility of multiple notes I would keep the 36 (or 88 for the pian) dimensional representation and have a sigmoid activation on each of the notes at the end. Then sum the binary log loss of each of the nodes and use that as the loss. Then you can treshold the nodes individually at prediction time to see if a note is on or off.


How to access maximum volume of tweets using Twitter Streaming API, without firehose access? Twitter is a popular source of data for many applications, especially involving sentiment analysis and the like.  I have some things I'm interested in doing with Twitter data, but here's the issue:  To get all Tweets, you have to get special permission from Twitter (which, as I understand it, is never granted) or pay big bucks to Gnip or the like.

OTOH, Twitter's API documentation says:

Few applications require this level of access. Creative use of a combination of other resources and various access levels can satisfy nearly every application use case.

Using the filter api with keyword tracking seems like something that would be a big part of this, but you obviously can't enumerate every keyword.  Using a User stream on many User accounts that follow a lot of people might be an option as well, and I'm not sure if it makes sense to think about using the search API in addition.

So here's the question "What combination of other resources and access levels is the best way to get the maximum amount of data from Twitter"?

Topsy - The site provides much more rate limit, and has archived all the tweets from 2006.
Its API provides a free trial.

A solution that is paid will be Gnip's Full history search API


Automatic data summarization with text I would like to automate periodic report writing based on data. Given one/some data tables, the machine should output texts like Stock A rose by 10% this year and hit 5 year high on 2019-12-01, or we made a large profit in sector B.

I can find the subject automatic text summarization, but seems it's about reading text and shorten it to its key sentences. Not exactly data summarization.

Could someone recommend some book/paper/video/MOOC on text generation based on data?

Then check out this tutorial generative/abstractive text summarization.


Topics to cover for software developer interested in data analytics First of all I don't know if this is the appropriate place to post this question. If it's not, I apologize in advance. It seems like the most relevant Stack Exchange sub.

A little about myself:

I'm a software developer and have been working in the field for about 5 years. I have a two year degree from a community college that didn't involve much math above pre-calc high school level stuff. I've never taken a statistics course.

Most of the software I write is business software. It's typically back end server or database stuff, never user facing. I do allot of data aggregation and normalization.

Lately I've been thinking about expanding my skillset. Data and statistics have always interested and I'd like to learn ways to measure and interpret data. Ideally this would lead to thinking of useful metrics and uses for data that would help the company I work at.

The problem is I'm not sure what I should be looking into. There are so many data science topics and I don't know what would be useful for me to learn. I prefer structured learning environments such as courses or video series.

I know this isn't a great question... I'm just not sure where to ask and I'm not sure how to ask for what I'm looking for... or even sure what I am looking for. Any guidance or suggestions would be very much appreciated. Even if the suggestion is a better place to ask the question or a better question for me to ask. Thanks much.

Welcome to the community!! There can be a lot of answers to this question but I would suggest you the approach I took when I shifted from software development to the data science field.

1) Refresh your statistics and probability concepts. You should not go into too much details but you must understand basic things like Gaussian Distribution, Mean, Variance, Probability,etc.

2) Go through the basics of Machine Learning concepts. I prefer Andrew Ng's machine learning course on Coursera. That will help you build a strong foundation and will give you a great start in the field

3) Choose a particular language Python/R for building models. Though it's totally upto you but I prefer python as it has great libraries for machine learning as well as Deep learning.

4) Take part in competitions. We learn by doing not by taking only lectures. I suggest you should join Kaggle and the slack community out there namely, 'KaggleNoobs'. It's a great community. I learn everyday a new thing from there.

P.S: Data Science is a vast field. It demands from you various skill set like Data Analysis, Data Visualisation, Machine Learning,etc. So sometimes it can become frustrating too. But once you start enjoying it, you will become a master eventually.


Is it possible to decode which neuron represent which feature and why does it represent it? In a neural network, Each neuron in the network represents some part of non-linear feature of the input. Ex: Like in mnist data, Consider the stem of number 9 is cut into multiple pieces and different part is represented by different neurons in the first hidden layer(Just an example from 3B1B neural networks video). My questions are:

What determines which neuron get to represent which part of the stem??

Is it possible that if we pass in the same input multiple times, each neuron can represent different part of the stem?? OR Is it that its all the magic of chain rule (i.e At the beginning, all neuron represent some trash feature and as updation of weights occur and then particular features have become synonymous to a particular neuron.) If so, How does this happen?

Thanks in advance

Short answer: NO.

Think about it, you could have 5 or 5000 neurons, and each neuron would represent a different thing in different models with different inputs.

One of the problems of neural networks is that it is quite difficult to understand what is going inside them, they are a bit of a black box. Other techniques such as decision trees are much more easy to understand or explain to non-technical people.

The closest thing you could do is visualize the hidden layers and that might give you some sort of insight.


How to estimate real distance between two detected objects in an image? You may think this is a duplicate, but my situation is different than previously asked questions. The only information I have is the width and height of the bounding boxes of detected people. The dataset I'm working on has images captured in different environments (street, garden, mall, ...). In other words, there is no fixed object in all images I can use as scale. The angle at which each image is captured varies drastically from almost parallel to the ground to prependicular to the ground. This means that bounding boxes don't always mimic the real size of a person.
My objective is to measure the real distance between people. Any hints on how I can do this ? I am starting to think it is not feasible
Thank you!

Distance in an image is based on pixels. You will need to convert this distance to the actual real world metric like cm or inch. To make this happen, you will also need a reference object in your image whose real world dimensions are already known to you. Or you will need the distance between the object and camera. If you manage to get this distance using your camera lens, then you won't need any reference object. And then, it is very complex to get distances between objects placed in 3D space.
This is a great article that explains the aforementioned approaches in 2D space. It might be a good start for you.


Relation between amount of training samples and model depth? When I add more hidden layers to my CNN (e.g. Dense Layers) it seems that the model needs more training samples to produce good results for classes with few training samples. In the single layer case the model provided better results, even for classes with few samples. (multi-class text classification with >10 classes)

Is there any evidence that my assumption is correct?

Yes, this is common knowledge. Every time you add a parameter to a model, you will need to give it more data in order for it to learn as well as the simpler model. Every individual weight in a neural network is a parameter. The more weights, the more parameters, and the more data is needed.

One of the fundamental tasks in model-building is finding a good tradeoff between having enough parameters to learn fine details and having enough data to train all those parameters. Too many parameters lead to overfitting in part because if you don't have enough data you end up memorizing individual cases, as opposed to learning about and averaging over a spread across several cases.

This is one reason why class imbalance is a problem. If you don't have enough data on one of the classes, then that class will be poorly understood by the model.


Imbalanced dataset - Positive majority class My dataset consists of 150 patients where 50 are controls/healthy (negative) and 100 are sick (positive). If I want my model to have high sensitivity at hight specificity, in other words to have low false positive rates, should I correct my model by applying weights to it? Because usually the positive class is the minority class and I see why you need to correct for it but should I in my case?

Thanks

There are two things that might help you:

1)
If you use a classifier $f$ that returns a value $f(x)\in[0,1]$ between 0 and 1 instead of a direct class assignment, then you can use a threshold $\theta\in[0,1]$:

$f(x)=\begin{cases}
    1,& \text{if } f(x)\geq\theta\\
    0, & \text{else }
\end{cases}$

The higher the value of $\theta$, the more you want to be sure to classify somebody as positive, which means that there will be fewer false positives, but at the same time you will have a fewer sensitivity (recall) because there will be more false negatives.

2) if your classifier always predicts the majority class, you could try oversample the minority class (using SMOTE or other oversampling techniques) to make the training data set balanced.


Geolocation Based Anomaly Detection in IPs Using Isolation Forest I'm trying to detect anomalies based on geolocation from IP addresses on a server access log file. I have created two features country and geo_velocity, using the IP address and the timestamp of each request. However, since all the requests are from stationary clients and all the clients are from one country in the log file I have, my dataset ends up looking something like this.
| Country     | geo_velocity|
| ----------- | ----------- |
| USA         | 0           |
| USA         | 0           |
| USA         | 0           |

Basically, If I plot the whole dataset in a scatterplot, it would condense onto a single point. Therefore, literally, every other value to these features should be an anomaly for this dataset.
I used Isolation Forest with GridSearchCV to tune hyperparameters. And for the scoring parameter in GridSearchCV, I used a custom scoring function as shown in the code below. The problem is, the model classifies everything as inliers even though they should be clearly outliers.
from sklearn.ensemble import IsolationForest
import numpy as np
import pandas as pd
from sklearn.model_selection import GridSearchCV

# this is similar to my dataset after label encoding the "country"
data = np.zeros((5000,2),dtype=int)
dataset = pd.DataFrame(data, columns=['country','geolocation'])

params = {'n_estimators':[70,80,100], 'max_samples':['auto'],
     'contamination':[0,0.001,0.010,0.1,0.5], 'max_features':[1,2],
     'bootstrap':[True,False],'n_jobs':[-1],
     'random_state':[None,1,], 'warm_start':[True]}

def scorer_f(estimator, X):
      return np.mean(estimator.score_samples(X))

isolation_forest = GridSearchCV(IsolationForest(), params, scoring=scorer_f)
model = isolation_forest.fit(dataset)

best_model = model.best_estimator_

predictions = best_model.predict([[0,0],[100,100]])
print(predictions)

output : [1,1]

The dataset generated in the above code is similar to the dataset I have after encoding. Even though the second point should clearly be an outlier, the model classifies it as an inlier.
What seems to be wrong with this approach? Thanks in advance!

IsolationForest doesn't work on Euclidean distance. Hence [0,0] is almost as good as [100,100]
It builds random Trees on the dataset and expects that the Outlier will singled-out very early in the Tree while the Inliers will go deep. With that logic, it can figure out the Outlier.

The IsolationForest ‘isolates’ observations by randomly selecting a feature and then randomly selecting a split value between the maximum and minimum values of the selected feature.
 Since recursive partitioning can be represented by a tree structure, the number of splittings required to isolate a sample is equivalent to the path length from the root node to the terminating node.
This path length averaged over a forest of such random trees, is a measure of normality and our decision function.
Random partitioning produces noticeably shorter paths for anomalies.
[scikit-learn doc]

Your data is very clean and unique. If we observe here, we can differentiate [0,0] and [100,100] in a single split. So the depth of Node will be same for both cases.
You may try a euclidean distance-based Model i.e. LocalOutlierFactor
from sklearn.neighbors import LocalOutlierFactor

model = LocalOutlierFactor(n_neighbors=10,novelty=True).fit(dataset)

model.predict([[0,0],[100,100]])

Output - array([ 1, -1])
Scikit-Learn guide for Outliers/Novelty - Link


Develop a marketing look-alike profile for prospective customers I work for a business-to-business company that has a large database of existing clients (small businesses) with various columns of data (e.g., industry, credit-worthiness, financial data, etc) describing each client. There are several marketing data vendors I know of that maintain small-business databases with columns of data similar to what my company has.

Question: Using the data my company already has, how I can I identify which customers in the vendors' databases have the highest potential of becoming a customer if marketed to? Should I build a predictive model? Should I do some sort of clustering?

The problem is that you only have positive instances (businesses who have become a customer) but no negative instances (businesses who haven't become a customer).  This prevents use of supervised learning.  You could try unsupervised learning -- e.g., a one-class classifier -- but I don't expect it to be particularly effective.

If you had data on businesses who you tried approaching, but ultimately did not become a customer of yours, then you could try applying supervised learning.


Basic sympy problem in anaconda I have enabled sympy on Anaconda to use it to solve basic linear equations, but whenever I try to type something up it gives me an error when defining the variables:

from sympy import *
x,y=symbols(’x,y’);
solution=solve((4*x-3*y-17,7*x+5*y-11),x,y);
P=(solution[x],solution[y]);
print(P);

This code gives me this error message:

  x,y=symbols(’x,y’);
                 ^
SyntaxError: invalid character in identifier

Does anyone know how I could fix this?

Just change the ’ to " it works just fine, see:

from sympy import *
x,y=symbols("x,y");
solution=solve((4*x-3*y-17,7*x+5*y-11),x,y);
P=(solution[x],solution[y]);

print(P)
(118/41, -75/41)


Scalar entities for k means clustering I am trying to understand kmeans clustering and I read a article where kmeans is used for clustering the features generated in network logs. This clustering is followed by a supervised classification.

They mention in article :

"Due to extensive repetition of input and output network ip addresses, portnumbers, and protocol information in the collected data, these were deemed scalar entities for analysis and excluded while clustering. The remaining attributes  were chosen as input to clustering  algorithm"

I fail to understand what scalar entities mean here? Does this mean that these input and output ip address and port numbers are not considered as features during feature extraction stage? If the ip addresses arent considered, how are the features mapped to the input data of network logs?

While doing clustering analysis, you would prefer to have features which discriminate data points and help you discover more homogeneous segments. However, if the value of a particular feature or variable is same for large percentage of the data, then that feature would not add any information for clustering.

Let us take an example, say you are dealing with bank customers data and there's a feature called 'country' in it. If in the given data, most of the customers have country as USA, then you would want to simply drop that feature as it adds no useful information. This is the extreme case in which there is single unique value.

In your network data logs example, port numbers and protocols would be mostly the same. The web server might be listening on port 443 for HTTPS protocol connection. So in that case, the two columns can be excluded from clustering analysis.


MobileNet and MobileNetV2: Bad Inference Results unfortunately I am having subjectively bad results in inference with pre-trained models of both MobileNet v1 and v2:

from keras.applications.mobilenet_v2 import MobileNetV2
ConvNet = MobileNetV2(input_shape = None, include_top = True, weights = 'imagenet', input_tensor = None, pooling = None, classes = 1000)

I have a local copy of these networks for the corresponding image size (224x224), depth multiplier 1.0 and weights trained for ImageNet.

After loading a model of MobileNetV2, I am exectuing a classification on random images from ImageNet or Google Images. Almost always the top-1 classification does not make any sense, for example very often I get the suggestions of a "Shower Curtain" or a "Pillow", although this is obviously not the case.

Testing with other models (VGG16, ResNet50) and changing only the model type (keeping the same parameters), I obtained correct or at least more understandable results that were also consistent among these two different models:

ConvNet = VGG16(input_shape = None, include_top = True, weights = 'imagenet', input_tensor = None, pooling = None, classes = 1000)

Having correct results here with the other models, I assume that my script is working correctly.

My question is: Has anybody ever experienced these issues with inference with MobileNet or MobileNetV2? And/or do you have any idea why this error occurs and how to solve it?

I appreciate any answer, please also consider seemingly trivial solutions since I am still quite a newbie ;)

Thanks a lot,
Tim

I found the solution: The source of the error was the wrong preprocessing. I used

from keras.applications.imagenet_utils import preprocess_input

instead of

from keras.applications.mobilenet_v2 import preprocess_input

This wrong function for preprocessing caused the images being misclassified. The latter is the correct preprocessing function.


Hindsight Experience Replay (HER) results obtained 50 times faster than original paper? I am reproducing the results from Hindsight Experience Replay by Andrychowicz et. al. In the original paper they present the results below, where the agent is trained for 200 epochs.

200 epochs * 800 episodes * 50 time steps = 8,000,000 total time steps.

I try to reproduce the results but instead of using 8 cpu cores, I am using 16 CPU cores.

Fetch Push

I train the FetchPush for 80 epochs, but with only 50 episodes per epoch. Therefore 80 * 50 * 50 = 200,000 iterations. I present the curve below, generated using two random seeds.

After 20 epochs = 50,000 iterations we solve this environment. In the paper above, it took the original authors 100 episodes = 4,000,000 iterations to do so.

How is my algorithm converging 50 times faster?

Pick and Place

I train the FetchPickAndPlace for 80 epochs, but with only 50 episodes per epoch. Therefore 80 * 50 * 50 = 200,000 iterations. I present the curve below, generated using three random seeds:

and logger output for the first two epochs, showing that indeed I have 50 episodes per epoch:

Now, as can be seen from my tensorboard plot, after 40 epochs we get a steady success rate, close to 1. 40 epochs * 50 episodes * 50 time steps = 100,000 iterations. Therefore it took the algorithm approximately 100,000 time steps to learn this environment.

The original paper took approximately 50 * 800 * 50 = 2,000,000 time steps to achieve the same goal.

How is it that in my case the environment was solved nearly 20 times faster? Are there any flaws in my workings above? Surely I am doing something wrong, right?

Results are also faster than another paper which also uses 19 MPI workers:

As stated in this paper: "We train for  50 epochs (one epoch consists of 19  2  50 = 1 900 full episodes), which amounts to a total of 4.75 x10^6 timesteps." It took around 2,000,000 timesteps to reach a median success rate of 0.9.

SUMMARY

Any suggestions on what I may be doing wrong would be appreciated.

Code
Reddit thread

EDIT - Logging

Logging process shows that the rank 0 worker is reporting results. Inside her.py)

   if rank == 0:
        logger.dump_tabular()

The function responsible for writing all diagnostics is dumpkvs() inside logger.py:

def dumpkvs():
    """
    Write all of the diagnostics from the current iteration
    """
    Logger.CURRENT.dumpkvs()

Code can be found here:

https://github.com/openai/baselines/blob/master/baselines/logger.py

I am reading the results incorrectly.

1 epoch = 50 episodes = 2500 timesteps
50 epochs = 2500 episodse = 125,000 timesteps
125,000*16 CPU cores = 2 million time steps.

Will give a more detailed answer later.


Sum of squares for matrix valued data over $\mathbb{R}$ and $\mathbb{C}$ Let us assume we have $k \times k$ matrix valued data and assume this is organized (possibly as time series):
$$ M_1, M_2, \ldots, M_n $$
Now, assume we are interested in writing down an error function that mimics sums of squares. This can naively be written as
$$ \sum_{i=1}^n (M_i - \hat M_i)^2 $$
where $\hat M_i$ is the $i$-th estimation. The question is, what is actually the proper way to write this function explicitly? For vectors, the Euclidean norm is "naturally" picked. What about this case?
One option is to multiply out these matrices and treat each of the resulting matrix's elements on its own. For example the element at position 11 would have its own "error function" that looks like:
$$
\sum_i (a_{11}^2 +a_{12}a_{21})
$$
and similarly for the other three elements. Here $M-\hat M \equiv A = (a)_{ij}$. Does this even make sense?
Furthermore, how to treat the same example having complex valued matrices?

Essentially you want to pick a function that will give you the "size" of a matrix. The most obvious way I can think of is by choosing a matrix norm, which is a map $\lVert \cdot \rVert \colon \mathbb{R}^{k, k} \to [0, \infty)$ (or you could generalise to a complex $k \times k$ matrix if you wished).
Your suggestion seems similar to computing
$$S = \sum_i (M_i - \hat M_i)^2$$
then using the Frobenius norm $\lVert S \rVert_F$ to turn this into a real number. The Frobenius norm essentially means "squash $S$ into a vector of dimension $k \times k$, then compute the Euclidean norm".
There are plenty of other norms you could consider, such as the operator norms. Interestingly enough it can be proved that any two matrix norms are equivalent up to a constant scaling factor, so your choice of norm isn't particularly important, and minimising the "sum of squares" under any norm should be more or less the same.