Ram usage of model keep increasing when training to previously trained model Update
It keep increasing when i load the saved model like this.

return tf.keras.models.load_model("models/min__1576172002.model")

and i am saving model using

   agent.model.save(f'models/min__{int(time.time())}.model')

I declare my model as

model = Sequential()
model.add(Dense(6,activation='relu',input_shape=(4,)))
model.add(Dropout(0.2))
# model.add(Flatten())

model.add(Dense(5,activation='relu'))
model.add(Dropout(0.2))
# model.add(Flatten())

model.add(Dense(5,activation='relu'))
model.add(Dropout(0.2))

model.add(Dense(2,activation='softmax'))

model.compile(loss="mse", optimizer=Adam(learning_rate=0.001), metrics=['accuracy'])

Then its working ok.
But after some time as it was not showing results, I use

model = Sequential()
model.add(Dense(6,activation='relu',input_shape=(4,)))
model.add(Dropout(0.2))

# model.add(Flatten())

model.add(Dense(5,activation='relu'))
model.add(Dropout(0.2))
# model.add(Flatten())

model.add(Dense(5,activation='relu'))
model.add(Dropout(0.2))

model.add(Dense(5,activation='relu'))
model.add(Dropout(0.2))

model.add(Dense(2,activation='softmax'))

model.compile(loss="mse", optimizer=Adam(learning_rate=0.001), metrics=['accuracy'])

I added only one more layer. But now.
I see RAM usage of computer. It is keep increasing and not stopping. If i pause training then RAM will also pause at that point and when i continue training  then RAM will increase and it took 32GB of RAM (and still increasing )and previous model with one less layer took only 6GB of RAM

I found it and answering as it might help somebody.

I saw pattern in this problem. Every time I load the model from my saved ones and then start them training again it will start eating my ram. Slowly it will reach to limit of my laptop.
But same model if i train from scratch will not do same.
I was loading model using

tf.keras.models.load_model("models/min__1576172002.model")

So Instead of training it again. I Build a new one and assign its weights = trained models weight. This way this problem is solved.

        prev_modal = tf.keras.models.load_model("models/min__1576224504.h5")
        model = Sequential()
        model.add(Dense(6,activation='relu',input_shape=(4,)))
        model.add(Dropout(0.2))
        # model.add(Flatten())

        model.add(Dense(5,activation='relu'))
        model.add(Dropout(0.2))
        # model.add(Flatten())

        model.add(Dense(5,activation='relu'))
        model.add(Dropout(0.2))

        model.add(Dense(2,activation='softmax'))

        model.compile(loss="mse", optimizer=Adam(learning_rate=0.001), metrics=['accuracy'])

        model.set_weights(prev_modal.get_weights())


What if multiple final prediction values for multi-class Neural Network are equal If, for example, your final prediction for a multi-class problem, say for ["mouse","cat","dog","lion"], is [0.1,0.3,0.3,0.3], should the neural network predict that this data is "cat","dog" or "lion"? Should this be picked randomly?

I'd say it's not very common to have absolutely equal probabilities with a model that has been trained for sufficient epochs and that does not overfit hard.

That said, if you happen to have equal probabilities, the choice of the class to predict should depend on the domain you're approaching, e.g:

A possible implementation is like in numpy's argmax, by choosing the first index.
You could also create a new class "unknown" specifically for this issue, in your test set.
Is it worse for your business problem if you make a mistake and predict class "2" instead of class "1"? You should then choose the class whose probability of mistake is less costly (Think about the medical domain and predicting several stages of cancer)
Is the class balance the same? You could go for the most probable class in the training set


Control which features are used for every task in multioutput classification? I would like to perform a multiclass-multioutput classification task, on vectorized textual data. I started by using a random forest classifier in a multioutput startegy:

    forest = RandomForestClassifier(random_state=1)
    multi_target_forest = MultiOutputClassifier(forest, n_jobs=-1)
    multi_target_forest.fit(X_train, y_train)
    y_pred_test = multi_target_forest.predict(X_test)

When looking on the feature importance for the individual estimators (multi_target_forest.estimators_ ) I've noticed that some features in my dataset are very relevant and useful for some tasks, but are disrupting for another class.
Example:

Task 1: classify documents for Date (q1, q2, q3, q4)
Task 2: classify document for Version (preliminary, final, amendment)

For task 1, features related to dates, such as 'April', are very useful. However, for the second task, the feature 'April' gets a high importance but is a consequence of overfitting to a small dataset. Knowing this I would like to actively remove such features.

Is there a way to control which features are used for every task?

I could just explicitly train separate classifiers for every task, but is that equivalent to multioutput-multiclass? or is there some joined probability calculation going on, that I'll be missing?

Thank you!

I don't think its possible.

First, let's see what's the difference in explicitly modeling separate trees for different tasks versus modeling them in joint manner.

Lets suppose we have 2 task with each n classes. In the later case, to be able to jointly model the correlations, one must create new classes which is a subset of all the permutations available from nC2 combination of the classes from the 2 tasks. Now, if it is true that one feature (say feature A) is beneficial for task 1 but not for task 2, then how would one decide whether to use the feature A while figuring out the final classes in both the task? Task 1 wants the feature but Task 2 doesn't, so this creates a deadlock which would prevent us from putting a bias in the tree model to not use feature A for the classification!

So, if you are certainly sure that a particular feature is not beneficial for a particular task the then the simpler and more effective way would be to model them separately as you mentioned.


Does it make sense to train a convolutional neural network on lo-res, use on hi-res pictures? this is my first machine learning project and actually also my first question here. I am a novice to machine learning with a background in theoretical physics.

I want to use a CNN to detect scratches from manufacturing defects in hi-res images. The images have to be hi-res; downsampling would destroy the very features we want to detect. Also, since we are specifically looking for scratches in certain parts of the product, we cannot simply divide the picture into many tiles and treat each pile - the network will have to "understand" both the small-scale structure (single scratch) as well as the large-scale structure (entire product) of the picture.

I'm concerned about limited model capacity (no of parameters) as well as training time. Also, since the number of training pictures are few, it does not seem to make sense to train a whole network from scratch just using my product pictures.

So, my question is, do you think the following approach would work:
* use a pre-trained (on general lo-res pictures) off-the-shelf CNN (possibly the CNN layers of VGG19?), followed by a pretty radical max-pooling (for downsizing the picture without introducing any further parameters), and then adding 1 or 2 fully connected layers with further successively lower resolution, and training ONLY those last layers using my pictures?

PS. btw I am planning to use Keras as a platform.

When it comes to questions like this, you should be thinking about this in reverse. By that I mean, you want to actually use this algorithm at some point, don't you? So, the real question is: What do your images look like at prediction time?

Once you establish that, then what is the pre-processing you're going to be doing on these images? My guess is that you're not going to be changing things like the resolution. Sure, you might do some cropping or re-sizing, things like that. But the bottom line is that you should be training your algorithm based on what your prediction items are going to look like.


What are some appropriate models to use for inventory forecast based on consumption history or trends? I am working on an inventory management system where I have daily/monthly/yearly consumption history for a particular item, which may or may not follow a repeating trend. In order to forecast demands within an upcoming time-frame, which model / distributions are recommended?

Data Format: Data is a series of (x, y) pairs where x is a timestamp and y is count of consumption.

As far as I studied, random forest or tree based models do not handle trend patterns effectively, while linear models do not seem to be appropriate either given the fact that data do not seem to be following a linear pattern.

Is this problem solvable using a single model? Should Linear Regression models be used for this problem?

The model you are looking for is called ARIMA, which are time series models where you can obtain trends, cycles, etc.

The ARIMA models lets you model your univariate data by detecting monthly/annual cycles, natural growth (trends) for a variable depending on the history of the same variable.


Resume Parsing - extracting skills from resume using Machine Learning I am trying to extract a skill set of an employee from his/her resume. I have resumes stored as plain text in Database. I do not have predefined skills in this case. How should I approach this problem?

I can think of two ways:

Using unsupervised approach as I do not have predefined skillset with me.
I will extract the skills from the resume using topic modelling but if I'm not wrong Topic Modelling uses BOW approach which may not be useful in this case as those skills will appear hardly one or two times. (I would appreciate if you enlighten me more about topic modelling).
Another approach is manually labeling the skills for resume and making it supervised learning problem. But I have around 500 resumes, manual labeling will be very tedious and very time consuming.

Any suggestions are welcome.

Thanks.

I'm not sure Topic Modelling will help you here, as it tries to extract abstract topics from text. I'm afraid resumes might be too 'dry' for it to work nicely.

Here are a few sources I found that might be helpful:

A resume parser
The reply to this post, that gives you some text mining basics (how to deal with text data, what operations to perform on it, etc, as you said you had no prior experience with that)
This paper on skills extraction, I haven't read it, but it could give you some ideas

For some personal suggestions :

I'm going to make the assumption you don't want to label those resumes by hand, and will try to explain how I would proceed first.
It might not be the best solution, as there's a lot of heuristics in it, but it's a beginning that's mostly aimed at studying your data and gain some insight on it.

You could see if the structure of the document helps: the skills you're looking for might often be appearing in a section with a specific title.
You could also work with gazeteers, which are lists of key words of interest, and combine these with colloquations (groups of words frequently appearing together). If, for example, you have a keyword appearing in your gazeteer, you could use ngrams to see what words appear around it the most, and turn your single word match into a multiple word skill (e.g : machine learning (2grams), natural language processing (3grams), etc)
Once you have an interesting list of keywords, you could move to ML and try word vectors, maybe use word2vec, and build vectors around your word of interests. If new skills come in, their vectors might be similar to the ones you have defined (sorry for being a bit unclear here, I only have a vague knowledge of how word2vecs work).

This is no silver bullet solution, but this should get you started.

Good luck !


Regression yields much smaller standard deviation and the mean is off, what could be wrong? I'm modeling a regression problem. An initial attempt yields the following:

labels.mean(): 0.00018132978443886167
labels.std(): 0.013450786078937208

predictions.mean(): 0.0005549060297198594
predictions.std(): 0.00430255476385355

As you can see, the mean is off, and the standard deviation is totally different. I wonder what does it indicate?

My guess: does it mean that my features are not discriminative enough, so that the model see examples w/ positive and negative labels alike, hence the small variance in the output?

I'm running the regression using XGBRegressor, with early-stopping. I have 1M training examples, 100K validation examples (for early-stopping), and another 100K for testing purpose (for which the mean and the standard deviation are shown above).

I also checked that the label distribution of the three sets are mostly basically the same.

The difference in standard deviation is nothing suspicious.  It is only to be expected, if you have a weak correlation.

Suppose the regression is y ~ x, i.e., y = ax + b.  Suppose that x only explains a small fraction of the variability in y, i.e., y is scattered all over the place and the least regression line only weakly fits the data.  Suppose also that the line is nearly horizontal (i.e., a is small).  Then the standard deviation of the predicted y-values will be small (since the line is nearly horizontal) but the standard deviation of the actual y-values might be large.

But really, the way to figure out what is going on is to visualize the data.  You should always start by visualizing the data.  Plot a scatterplot, and superimpose the least squares fit line on top of it.  I bet you'll immediately have a better sense of what might be going on.


RNN model with 3 hidden layers In a paper, it mentioned: ANN, RNN, and LSTM NN are optimized to contain three hidden layers with 1000 hidden units in each layer.

I would like to model the RNN model in Keras. But my code fails in an error!

My code:

model=Sequential()
model.add(SimpleRNN(1000,input_shape=(320,15),activation='relu'))
model.add(SimpleRNN(1000))
model.add(SimpleRNN(1000))
model.add(Dense(1600))

Error:

    ValueError                                Traceback (most recent call last)
<ipython-input-49-ff01ce62eb30> in <module>()
      1 model=Sequential()
      2 model.add(SimpleRNN(1000,input_shape=(320,15),activation='relu'))
----> 3 model.add(SimpleRNN(1000))
      4 model.add(SimpleRNN(1000))
      5 model.add(Dense(1600))
.....
....
...
..
.

ValueError: Input 0 is incompatible with layer simple_rnn_2: expected ndim=3, found ndim=2

How can I code for the RNN model which is optimized to contain three hidden layers with 1000 hidden units in each layer?

Thank you so much

The shape should be 3d array --> (samples, timesteps, features)

It needs to add return_sequences=True in first two RNN layers.

model=Sequential()
model.add(SimpleRNN(1000,input_shape=(1,320*15),activation='relu', return_sequences=True))
model.add(SimpleRNN(1000, return_sequences=True))
model.add(SimpleRNN(1000))


Value on Decision Tree plot After plotting a sklearn decision tree I check what it says in each box and there is one feature "value"
that I am not sure what it refers.

The first line will be the column and the value where it splits, the gini the "disorder" of the data and sample the number of samples in the node.

But value?

value represents the number of items in each class.

If you look at the top node, you should view it as:

There are:

35100 samples of class 0
16288 samples of class 1
which sums up to 51388 samples total


Are there any activation functions which on inputting integer data will produce the output as integers? Idea is to create the model for ethereum mining which deals with only integer data.

There are quite a lot of activation functions $f : \mathbb{R} \rightarrow \mathbb{R}$ such that their restriction to integers $f|_\mathbb{Z} : \mathbb{Z} \rightarrow \mathbb{Z}$ produce integer values. $f(x) = max\{0,x\}$ is quite famous.


Keras - Implementation of custom loss function with multiple outputs I am trying to replicate (a way smaller version) of the AlphaGo Zero system. However, in the network model, I am having a problem. The loss function I am supposed to implement is the following:

$$l = (z - v)^2 - \pi^T log(p) + c ||\theta||^2$$

Where:

$z$ is the label (a real value between -1 and 1) of one of the two heads of network and $v$ is this value predicted by the network.
$\pi$ is the label of a distribution probability over all actions and $p$ is the distribution probability over all actions predicted by the network.
$c$ is the L2 regularization parameter.

I pass to the network a list of channels (representing the game state) and an array (same size of the $\pi$ and $p$) representing which actions are indeed valid (by putting 1 if valid, 0 otherwise).

As you can see, the loss function uses both the target and the network predictions for the calculation. But after extensive search, when implementing my custom loss function, I can only pass as parameter y_true and y_pred even though I have two "y_true's" and two "y_pred's". I have tried using indexing to get those values but I'm pretty sure it is not working.

The modeling of the network and the custom loss function is in the code below:

def custom_loss(y_true, y_pred):

    # I am pretty sure this does not work

    output_prob_dist = y_pred[0]
    output_value = y_pred[1]
    label_prob_dist = y_true[0]
    label_value = y_pred[1]

    mse_loss = K.mean(K.square(label_value - output_value), axis=-1)
    cross_entropy_loss = K.dot(K.transpose(label_prob_dist), output_prob_dist)

    return mse_loss - cross_entropy_loss

def define_model():
    """Neural Network model implementation using Keras + Tensorflow."""
    state_channels = Input(shape = (5,5,6), name='States_Channels_Input')
    valid_actions_dist = Input(shape = (32,), name='Valid_Actions_Input')

    conv = Conv2D(filters=10, kernel_size=2, kernel_regularizer=regularizers.l2(0.0001), activation='relu', name='Conv_Layer')(state_channels)
    pool = MaxPooling2D(pool_size=(2, 2), name='Pooling_Layer')(conv)
    flat = Flatten(name='Flatten_Layer')(pool)

    # Merge of the flattened channels (after pooling) and the valid action
    # distribution. Used only as input in the probability distribution head.
    merge = concatenate([flat, valid_actions_dist])

    #Probability distribution over actions
    hidden_fc_prob_dist_1 = Dense(100, kernel_regularizer=regularizers.l2(0.0001), activation='relu', name='FC_Prob_1')(merge)
    hidden_fc_prob_dist_2 = Dense(100, kernel_regularizer=regularizers.l2(0.0001), activation='relu', name='FC_Prob_2')(hidden_fc_prob_dist_1)
    output_prob_dist = Dense(32, kernel_regularizer=regularizers.l2(0.0001), activation='softmax', name='Output_Dist')(hidden_fc_prob_dist_2)

    #Value of a state
    hidden_fc_value_1 = Dense(100, kernel_regularizer=regularizers.l2(0.0001), activation='relu', name='FC_Value_1')(flat)
    hidden_fc_value_2 = Dense(100, kernel_regularizer=regularizers.l2(0.0001), activation='relu', name='FC_Value_2')(hidden_fc_value_1)
    output_value = Dense(1, kernel_regularizer=regularizers.l2(0.0001), activation='tanh', name='Output_Value')(hidden_fc_value_2)

    model = Model(inputs=[state_channels, valid_actions_dist], outputs=[output_prob_dist, output_value])

    model.compile(loss=custom_loss, optimizer='adam', metrics=['accuracy'])

    return model

# In the main method
model = define_model()
# ...
# MCTS routine to collect the data for the network input
# ...

x_train = [channels_input, valid_actions_dist_input]
y_train = [dist_probs_label, who_won_label]

model.fit(x_train, y_train, epochs=10)

In short, my question is: how do I correctly implement this custom loss function that uses both the network outputs and label values of the network?

As you are mixing y_true[0] with y_pred[0] and y_true[1] with y_pred[1], you could consider having different losses for each, and using loss={'Output_Dist': custom_loss, 'Output_Value': losses.MSE} when compiling. Internally, it will add the result of each one in a final loss.
def custom_loss(y_true, y_pred):
    return -K.dot(K.transpose(y_true), y_pred)


Multiclass imbalanced classification I have a dataset with the target variable having 3 classes.  Value counts of Target variable are as follows:

Class A :26000
Class B: 5000
Class C: 500
Class C  prediction is most important from a business standpoint.

To date I have mostly done binary class (mostly 1:0.12) distribution of target class,wherein at the end, deciling of prediction probabilities is done, in decreasing order of probabilities starting with D1 as the highest probability and D2 the lowest.

Now with influx of a new class that too with so less population,I am perplexed on three aspects.

What algorithms are best?
What metric should be used?
How to get a deciling of probabilities in this case, if applicable?

Any leads/kernel/blog post would be really helpful.

You can weight the loss of each class with a suitable value which is inversely proportional to the class size. One example would be to use: (Total number of data points)/(number of data points of the class). For loss you can use the standard Binary Cross Entropy (BCE) loss.

As for the metric, I would advise you to use AUROC (Area Under the Receiver Operating Characteristics). You can generate an AUROC for each class to see the performance of the model on these individual class and average them to get an overall sense of your models performance.

Edit: I noticed you have put a tag on f1 score so I would like to point out that AUROC should be preferred as the metric as they more robust to class imbalance as compared to f1 scores.


Select best answer from several existing ones for a question After analyzing questions on a forum, a human support team has created a set of general answers, that can be used to provide basic answers on the forum.

I am trying to build a system that:

Selects best answer from this set of answers for a given question. How to do this?
Estimates acceptability of such an answer. Which metrics to use?

Using document embeddings, such as doc2vec to find similarity between question and answer does not solve the problem, I think.  Other ideas?

Update 1

In my case I don't have labeled data set with good answers to train my model. My problem is unsupervised learning problem.

This problem is multiple choice answering question. I can see you have already tried gensim, doc2vec etc. You can try pytorch based transformer solution. Here is the link: multiple-choice . You can create your data in swag format and remove --do_train in below code for prediction on your dataset.

It has been trained on swag dataset and has given decent accuracy.

If it works fine for you then good, else you would like to finetune it. For finetuning --do_train should be mentioned below:

#training on 4 tesla V100(16GB) GPUS
export SWAG_DIR=/path/to/swag_data_dir
python ./examples/run_multiple_choice.py \
--model_type roberta \
--task_name swag \
--model_name_or_path roberta-base \
--do_train \
--do_eval \
--do_lower_case \
--data_dir $SWAG_DIR \
--learning_rate 5e-5 \
--num_train_epochs 3 \
--max_seq_length 80 \
--output_dir models_bert/swag_base \
--per_gpu_eval_batch_size=16 \
--per_gpu_train_batch_size=16 \
--gradient_accumulation_steps 2 \
--overwrite_output


When should i use fit(x_train) and when should i fit( x_train,y_train)? I am a rookie in data science and I'm confused about some part of ML.
My Problems:
If I use a package of linear-model like LiearRegression() or PolynomialFeatures():

Should i to fit my (x_train,y_tarin) or only is enough when i fit my (x_train)?
Should i to transform my (x_train,y_tarin)?
Should i fit my (x_test) too or it will be only transformed? If only transform why?

from sklearn.linear_model import LinearRegression()
LinReg=LinearRegression()
LinReg.fit(x_train) or LinReg.fit(x_train,y_train) ???? ;-|
LinReg.transforn(x_test)

When you are fitting a supervised learning ML model (such as linear regression) you need to feed it both the features and labels for training. The features are your X_train, and the labels are your y_train. In your case:
from sklearn.linear_model import LinearRegression
LinReg = LinearRegression()
LinReg.fit(X_train, y_train)

If you are performing some unsupervised learning approach, or simply some data transformation (such as PolynomialFeatures) you simply fit on your feature space (X_train) since there are no labels required for such an approach. Like so:
from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(2)
poly.fit(X_train)
X_train_transformed = poly.transform(X_train)

For your second point - depending on your approach you might need to transform your X_train or your y_train. It's entirely dependent on what you're trying to do.
As for your last point - never ever fit on testing data. It defeats the purpose of a train/test split. Usually what is done is your pipeline step is fit either with X_train and y_train or just X_train alone. This fit transformer can then be applied to your testing data (X_test) using the .transform() method but never use this data for .fit()


Example of convex activation function For a particular task, I need a convex activation function with the following properties:

f''(x) > 0
0 <=    f(x) <=  1
f(x) is monotonic
f(x) is not "exploding" i.e. avoiding functions such as f(x) = x²

The only example I have in mind for this is the softplus activation function. Would you have anything else in mind?

There is no such function.

A convex function defined on all of $\mathbb{R}$ with bounded range is constant.


NER and context mapping I want to extract various amounts and tenure of contracts from different contract documents that we have.

For example :

  Mr xyz, this contact is valid for 3 Months and has to be executed within 1 Month.
  you have to pay \$3000  as  contract fee, \$60 as taxes, \$1200 as security deposit and \$1200 as rent

Expected output : Contract tenure : 3 Months, Amount to pay : \$3060

Please note : I tried NER but that is showing 2 tenured and 2 amounts. However I am looking for a technique by which we can associate amount to contract.

Standard NER is going to extract individual entities, which in this case is time (3 months, 1 month) and currency ($3000, etc). You'll also want to think about Relationship Extraction, which identifies how two pieces of text relate to one another. For example, from a Euclidean distance measure, "contract" is related to "valid" and "executed", and "valid" is related to "3 months" while "executed" is related to "1 month". Based on what you said your desired output should be, you'll want to train your model to calculate the shortest distance between "contract" and "3 months", which in this case means teaching it to look for "valid" while ignoring "executed". There are different ways of doing this which you'll want to think about in terms of what works best for your corpus of text.
Here's a link to get you started (also includes links to other resources).


NLP - extract sentence parts related to people Thank you for your help, I appreciate your time.

This is not a standard problem but you should be able to roughly do this using two basic kinds of tools that usually go together anyway:

Use an NER system to identify people (as opposed to organizations) in sentences. Most systems have a default model that flags people.
Use an Open Information Extraction system to get relation triples from sentences like (Sarah, has, brown_hair). See OpenIE for an example. You could also use a dependency parser.

So you take the relationships you get from the tools in part 2 and throw out any where the referrent/noun isn't a person. If you have to deal with relative clauses like in your first example ("John, who was..."), you'll also have to deal with coreference resolution, which is its own complicated problem.


Construct a multivariate neural network I want to build a neural network with a data input of 15-18 variables. I want to use the model for anomaly detection based on the reconstruction error. I've done some tutorials now but the data was always already given so I have to do the very first step on my own now.
Thus, I wonder, how do I handle the multivariate data?

I see three options:

1) Build a single neural network for each variable (univariate)
2) Build a neural network with a single outcome based on all multivariate input variables
3) Build a neural network with a multivariate output based on a multivariate input.

1) and 2) is quite cumbersome I'd say. But is 3) even possible? If so, what do I have to take into account?
Will the outcome variable then just be a data frame, or some other type like a certain array or so?

I do not understand your problem, the first layer, i.e. input layer, of your neural net must have a size equal to the number of variables you have. The last layer, i.e. output layer, must have a size which is equal to the number of values you want to predict. Please be kind and tell me what you want to know more specifically, if my answer doesn't satisfy you.


Low training and validation loss but bad predictions what does it mean if in a neural network, the training and validation losses are low but the predictions (so use model on test set) are bad?

Neural network architecture:

model = tf.keras.Sequential()
model.add(Masking(mask_value=0., input_shape=(timesteps, features)))
model.add(Bidirectional(LSTM(units=100, return_sequences=True, dropout=0.5, recurrent_dropout=0.5), input_shape=(timesteps, features)))
model.add(Dense(1, activation='sigmoid'))

My loss function is:

def get_top_one_probability(vector):
  return (K.exp(vector) / K.sum(K.exp(vector)))

def listnet_loss(real_labels, predicted_labels):
  return -K.sum(get_top_one_probability(real_labels) * tf.math.log(get_top_one_probability(predicted_labels)))

Fitting settings:

model.fit(training_dataset, training_dataset_labels, validation_split=0.2, batch_size=1,
            epochs=number_of_epochs, workers=10, verbose=1,
            callbacks=[SaveModelCallback(), keras.callbacks.EarlyStopping(monitor='val_loss', mode='auto', patience=3)])

Thanks in advance.

Perhabs your network is overfitting. Overfitting is where networks tuned its parameters perfectly to your training data and therefore it has very low loss on training set. Unfortunately, it will perform badly when new samples are provided within test set. Overfitting is broadly descipted almost everywhere: https://en.wikipedia.org/wiki/Overfitting

What you can do with overfitting:

try neural network with simplier structure, it should help your network to preserve ability to generalize knowledge,
use early stopping; try to measure validation loss at every epoch. And when it gets higher for like 3 epochs in a row - stop network training,
dropout: dropout is simple technique that prevents big networks from overfitting by dropping certains connection in each epochs training then averaging results. It is easy to use because it is implemented in many libraries like Keras or PyTorch.
$l_1$ and $l_2$ regularization: they are modification to your loss function making it involve sum of weights. Minimizing sum of net's weights prevents situation when network is oversensitive to particular inputs.

The other cause for this situation could be bas data division into training, validation and test set. Training and validation set's loss is low - perhabs they are pretty similiar or correlated, so loss function decreases for both of them. Then relation you try to find could by badly represented by samples in training set and it is fit badly. I would check that division too.


Query related to Matlab Neural Network toolbox If anyone has used the Neural Network toolbox in matlab, what does the two values in the performance section means? Like considering this image:

The performance section has the value 0.484 and 1.32e-07. what are these two values. Among them which is the mean squared error value. If one is the mean squared error value, then what is the other value?

I don't see it explicitly discussed in the documentation, but it appears that the numbers on the right side of the screen are the stopping criteria.  That run appears to have stopped because the error on the validation set increased six times (explained here).  I'm guessing it would also stop after 1000 epochs, when 0 MSE was achieved, the gradient was below 1e-6, or after 4 resets.

Thus the number to the left of the bar should be the MSE on the epoch with the best validation error*.  I'm guessing the value of the blue bar is the lowest that MSE got during the iterations that were actually increasing the validation error.

From the docs:
"In the default setup, the training stops after six consecutive increases in validation error, and the best performance is taken from the epoch with the lowest validation error."


Which features are important in determining fluency of participants measured by weighted composite score? I'm starting to learn about data science. I have sample data including different features run over several people, for each person n times. these features represent the fluency of the participants in some languages. they are including the number of correct words per sentence, the average number of the correct words in total, the average of speaking duration, the average of pauses in each turn, etc. So for features such as the number of correct words, the higher the score the better, and there is a feature such as average pauses which the lower the score the better.
I want todecide at the end which features are more significant and worth keeping and which features can be ignored.
I read about the weighted composite scores, and as far as I understood the main step is figuring out the coefficients (weights). but I didn't understand how these coefficients should be calculated. some cases are based on the analysis intuition, as I saw in some examples, but it can be so subjective.

This is a tentative answer.
One can try to determine first the relative importance of each feature (eg by Factor Analysis or Principal Component Analysis, ..)
Once the more significant features have been identified (or guessed perhaps), then one can try combinations of scores with these features.

Take simply the average as a composite score.
Do a linear regression and use the coefficients as the weights of the composite score.


Discarding correlation among inputs in a neural network I am working on a problem with 4 inputs and 1 continuous output variable. The sum of all values of the 4 input variables is always 1.

  a1+a2+a3+a4=1

So, they are correlated.

My question is: should I use all 4 variables for neural network training? Or, should I use any 3 of them to get rid of correlation? Is there any problem if I use all 4?

You could try some feature selection techniques if you want to choose 3.

That said, have you tested the data for multicollinearity?  Maybe you have but I don't think that a1+a2+a3+a4=1 implies a high degree of correlation.

To answer your question though, it really doesn't hurt to try and train a neural net with the four variables.  If one of the inputs is unnecessary, the NN training may well set the weight on one to close to zero without you having to choose one.


How to convert nested list into a single list in python? I have a list that contain id number. Some elements of the list is an another list. To convert nested list into a single list, I write a recursive function using collections module.

My codes are following.

from collections import Iterable
def single_list(list):
for item in list:
    if isinstance(item, Iterable):
        yield from single_list(item)
    else:
        yield item

Item_list = [10,20,[30,40],[50,'Null',70],100]
items_single=single_list(Item_list)
for item in items_single:
print(item)

When I run my program, I get the following error message.

  Traceback (most recent call last):
  File "/Research/SoftDev/SEPJ/StackOverflow_qs.py", line 42, in <module>
  for i in items_single:
  File "/Research/SoftDev/SEPJ/StackOverflow_qs.py", line 36, in single_list
  yield from single_list(item)
  File "/Research/SoftDev/SEPJ/StackOverflow_qs.py", line 36, in single_list
  yield from single_list(item)
  File "/Research/SoftDev/SEPJ/StackOverflow_qs.py", line 36, in single_list
  yield from single_list(item)
  [Previous line repeated 986 more times]
  File "/Research/SoftDev/SEPJ/StackOverflow_qs.py", line 35, in single_list
  if isinstance(item, Iterable):
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/abc.py", line 184, in __instancecheck__
if subclass in cls._abc_cache:
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/_weakrefset.py", line 75, in __contains__
return wr in self.data
 RecursionError: maximum recursion depth exceeded in comparison

 Process finished with exit code 1

I do not know how to fix the error.

You are getting the error because of string value contains in the nested list.
You need to handle string value as well.

You can fix the error by following codes:

from collections import Iterable

def single_list(list,ignore_types=(str)):
for item in list:
    if isinstance(item, Iterable) and not isinstance(item, ignore_types):
        yield from single_list(item,ignore_types=(str))
    else:
        yield item

Item_list = [10,20,[30,40],[50,'Null',70],100]
items_single=single_list(Item_list)
for item in items_single:
    print(item)


Neural Network Hidden Neuron Selection Strategy I'm trying to determine what is the best number of hidden neurons for my MATLAB neural network. I was thinking to adopt the following strategy:

Loop for some values of hidden neurons, e.g. 1 to 40;
For each NN with a fixed number of hidden neurons, perform a certain number of   training (e.g. 40, limiting the number of epoch for time reasons: I was thinking to doing this because the network seems to be hard to train, the MSE after some epochs is very high)
Store the MSE obtained with all the nets with different number of hidden neurons
Perform the previous procedure more than 1 time, e.g. 4, to take into account the initial random weight, and take the average of the MSEs
Select and perform the "real" training on a NN with a number of hidden neurons such that the MSE previously calculated is minimized

The MSE that I'm referring is the validation MSE: my samples splitting in trainining, testing and validation to avoid overfitting is 70%, 15% and 15% respectively)

Other informations related to my problem are:
fitting problem
9 input neurons
2 output neurons
1630 samples

This strategy could be work? Is there any better criterion to adopt? Thank you

Edit: Test done, so the result suggest me to adopt 12 neurons? (low validation MSE and  number of neurons lower than 2*numberOfInputNeurons? but also 18 could be good...

A rule of thumb approach is:

start with a number of hidden neurons equal (or little higher) that the number of features.
In your case it would be 9. My suggestion is to start with 9*2 = 18 to cover a wider range of possibilities.
Be sure your test and validation sets are selected "fairly": a random selection and varying the seed some number of times to test different configurations would be ok.

In general, a number of neurons equal to the number of features will tend to make each hidden neuron try to learn that special thing that each feature is adding, so will could say it is "learning each feature" separately. Although this sounds good it might tend to overfitting.

Since your number of inputs and your dataset size is small its ok to start with a hidden layer size of the double (18) and start lowering down. When the training error and test error stabilize in a difference lower than a threshold then you could have found a better generalizing model.

Neural networks are very good at finding local optima by exploring deeply a solution from a starting point. However, the starting point it is also very important. If you are not getting a good generalization you might try to find good initial starting points with methods of Hybrid Neural Networks. A common one, for example, is using genetic algorithms to find an initial combination of weights and then start the neural from that point. Given that your search space would be better covered (in case your problem actually needs that).

As for every problem in machine learning is very important to clean your data before introducing it to the NN. Try to be very detailed in order to avoid the NN to learn things you already know. For example if you know how two features are correlated improve the input data by making this correlation explicit so less workload is given to the NN (that might actually get you in trouble).


Increase F1 score on a text corpus I'm doing some classification with a text corpus [professional emails], I've already done all the basic
pre processing steps (stemming, remove stop words, punctuation, low frequency words, words length ...)  which gives me a F1-score = 0.6.

I was wondering what could be the next step(s) to improve my results, I thought of "stacking" (use multiple classifiers) and maybe removing words which are common to the different classes (4 differents types of texts) [ not sure about this solution ]

Edit:

I tried with 15 differents classifier the best one is gradient boosting (gbm )with an F1-score=0.6002159.
The features have been extracted through a bag of words ( Document Term Matrix ) with Tf-Idf [I don't think taking into account the length of a text, punctuation or others is relevant] - [It is mainly professional emails]

Have you tried without the Tf-Idf weighting? What about a bi-gram analysis?

You can also review the pre processing, maybe without removing the low frequency words (wich the tf idf already does). Another idea is to remove one very high frequency word that appear on your corpora.


What are the ways to partition a large file that does not fit into memory so it can later be fed as training data? Is there any other way to partition a large file that does not fit into memory so it can be fed as training data other than using spark? or hadoop?

Yes, of cause. But, it's insignificant, because Spark and Hadoop are better.

This is my idea. Suppose that your memory can take in 100,000 examples. So splitting your data set to files with size lower than 100,000.

The key and most complex step is how to train classify with those data. Good luck, For Gradient descent series optimization algorithms (GB, SGD and so on), most algorithms (SVM, GBDT, Bayes, LR, deeplearn and so on) support this. You could
 load one file to RAM and fed them to classifier until to find the best parameter.

My code is very simple. Before each iteration, re-shuffling the order of simples and re-splitting data set will boost the classifier.

import numpy as np

X = np.random.random((100, 2))
y = [1 if x[0] > x[1] else 0 for x in X]

from sklearn.linear_model import LogisticRegression

lr_cly = LogisticRegression()

def stop_train(X_s, y_s, threshold):
    scores = [gnb.score(X, y) for X, y in zip(X_s, y_s)]
    return np.mean(scores) > threshold

def iter_train(cly, X, y, threshold=0.99, max_iter=10):
    X_s = [X[:50, :], X[50:, :]]
    y_s = [y[:50], y[50:]]

    iter_times = 0
    while iter_times <= max_iter:
        print "--------------"
        for X, y in zip(X_s, y_s):
            cly.fit(X, y)
            print cly.score(X, y)
        if stop_train(X_s, y_s, threshold):
            break
        iter_times += 1

iter_train(lr_cly, X, y)


Normalized output of machine learning For machine learning, we need to normalized the inputs (features) for good results.
Do we need to normalize the output as well during training?

First you do not always need to normalize (standardize) the input vectors (feature vectors), sometimes is good, sometimes is bad. In general you scale your feature vector when the magnitude of a feature dominates the others, so the model cannot pick up the contribution of the smaller magnitude features. Read here for a detailed explanation.

Second there are two general classes of machine learning problems: classification and regression. In a classification type problem the output (dependent variable) is discrete, so you do not need to normalize it.

In a regression type problem scaling the output do not affect the shape of your function see here. Moreover it does not affect error functions like Mean Squared Error, i.e. the error is also scaled.


Collapse a list to most common spellings I have a user-generated list of proper names.  For the sake of conversation, imagine it's pet names.

This list will have many variations of the same name: misspellings, alternative spellings, and even random use of punctuation or spaces.

eg

Fluffy
Fluffy the Dog
Flufy Dog
Fllufy

I can get a count of each spelling to figure out that, say, Fluffy is the most common at 2,000 occurrences while the others are all <100.

I tried to condense using the Levenshein distance package (python-Levenshtein) which had mixed success.  I would assign a name to a "more likely" name if it's Levenshtein distance was < 3 and the "more likely" name had a higher count.  When I manually search a few known common names, I can manually weed out the Levenshein winners from the losers, as well as include others Levenshein missed in my list (eg Fluffy the Dog and Fluffy have a huge distance of 8).

Are there better techniques I should be trying?

Credit to @Emre for setting me on this path with n-grams.

I discovered the library fuzzywuzzy which is well-built to solve this exact problem.  I'll give a brief synopsis, but a well-written tutorial and documentation made this pretty straight-forward.

I have a DataFrame where names are the index and there is a count column.
I loop through the index, finding high-similarity strings and then mapping the name to the name (including self) with the highest count.

names = df.index

#iterate through list find the top 10 (arbitrary) matching strings > 90 score, pop off the highest-volume order one
def find_top_match(name):
    my_matches = process.extract(name, names, limit=10, scorer=fuzz.token_set_ratio)
    my_matches = [t[0] for t in my_matches if t[1] >= 90]
    top_match = df.loc[my_matches].sort_values(by='count', ascending=False).index[0]
    return(top_match)

top_matches_set = []

for c in df.index:
    returned_match = find_top_match(c)
    top_matches_set.append(returned_match)

df['real_name'] = top_matches_set


How to decode encoded labels in Decision tree classifier I have some dataset with procurements of organization where actually i'm working. The aim is to find most important features that describe why some processes of purchases is succesful, and why not succesful.
To solve this, I'm using Decision Tree Classifier.
First, I convert boolen columns to int
for col in ['flag_prequalification', 'flag_procurement_category_strategy']:
data.loc[:, col] = data.loc[:, col].astype(int)

Then, convert object type to string. This for label encoding
for cat_feature in categorical_features:
    data.loc[:, cat_feature] = data.loc[:, cat_feature].fillna('unknown').astype(str)
%%time
encoder = defaultdict(LabelEncoder)
data.loc[:, categorical_features] = (
    data
    .loc[:, categorical_features]
    .apply(lambda x: encoder[x.name].fit_transform(x), axis=0)
)

After, let's use log(sum) instead of sum
for col in ['sum_tru_no_nds (lot)', 'price (lot)', 'count (lot)']:
data[col] = (data[col] + 1).apply(np.log)

And, in the end plot the tree:
from sklearn.externals.six import StringIO
from IPython.display import Image
from sklearn.tree import export_graphviz
import pydotplus

import graphviz
# DOT data
dot_data = tree.export_graphviz(clf, out_file=None,
                                feature_names=None,
                                class_names=target_col,
                                filled=True)

# Draw graph
graph = graphviz.Source(dot_data, format="png")
graph

The question is, how can I explain this tree to others, more precisely how can I decode this labels in tree?

First, setting feature_names will replace all the X[2] and such.  Using feature_names=data.columns is sensible, provided your feature names aren't too long for easy display.
Now, you'll still be comparing against the label encodings.  By saving the label encoder objects (your encoder dict), you can retrieve which levels correspond to the integer labels (with the classes_ attribute, or possibly the inverse_transform method).  If your features have many levels, then you probably don't want to display all the levels in each split, and just providing them off to the side somewhere might be best.  If your features have few levels, maybe hacking the resulting DOT code will work; see this related answer of mine, which concerns one-hot encoded variables.


How to apply K-Medoids in PySpark? the pyspark ml library does not provide any clustering methods for K-Medoids. So my question is, how can one apply K-Medoids in a pyspark context?

There is a k-medoids clustering for PySpark at spark-packages.org/package/tdebatty/spark-kmedoids and the source code is github.com/tdebatty/spark-kmedoids.

It can be installed with:

> $SPARK_HOME/bin/spark-shell --packages tdebatty:spark-kmedoids:0.1.2


Clarifying Probability Mass Function (PMF) I am currently reading Deep Learning book, and I want to get better understanding of probability theory. In chapter 3.3.1 of Deep Learning book it states that:

  Often we associate each random variable with a diﬀerent probability
  mass function and the reader must infer which PMF to use based on the
  identity the random variable, rather than on the name of the
  function;P(x) is usually not the same as P (y).

And not many paragraphs latter, it saids the following:

  Probability mass functions can act on many variables at the same time.
  Such a probability distribution over many variables is known as a joint
  probability distribution.P(x=x, y=y) denotes the probability
  that x=x and y=y simultaneously. We may also write P (x, y) for brevity

I am having hard time grasping these two paragraphs. What do they mean when they say that P(x) is USUALLY not the same as P(y). As I understood, random variable is basically a random phenomenon from a real world that we wish to model. And each random phenomenon has its own Probability Mass function. Does this mean that the first paragraph indicates that random variable y represents a different phenomenon, and in the second paragraph random variable y represents the same type of phenomenon as x and that is why we use the same probability mass function?

Thanks in advance!

I believe they mean the following: "We will use $P$ to denote the concept of a probability mass function (pmf). $P(x)$ is the pmf of random variable $x$, while $P(y)$ is the pmf of random variable $y$. The two variables are usually the results of different random experiments, hence we use two separate letters, $x$ and $y$. As such, they will probably have different pmfs. But somethimes, it might happen that they have the same."

I think the essential source of confusion is that $P$ is not used to denote any single pmf, but is the name used to refer to a pmf. As a parallel, height_of(Stefan) and height_of(Mark) may be different, or they may be the same. height_of is just the name of the thing we are measuring, not the thing itself.

P(x,y) is thus the pmf of the joint variable (x,y). P((x, y)) would be a more unambiguous notation, but I guess is omitted for brevity. If you need more clarification on joint variables let me know. I hope this helps.


Is the GA R package the best Genetic Algorithm package? I am learning about Data Science and I love the Healthcare part. That's why I have started a blog and my third entry is about using Genetic Algorithm for solving NP-Problems.
This post is https://datasciencecgp.wordpress.com/2015/01/31/the-amazing-genetic-algorithms/

I have some expertise with GA package solving problems like the TSP, but do you know any most powerful R package?

Thanks so much!

For such questions, I like to go to the Task Views on CRAN, since the packages noted there are, to a degree, pre-vetted by the R community. I'd trust those a tiny bit more than just googling myself.

The Machine Learning Task View at CRAN says:

  Packages rgp
  and
  rgenoud
  offer optimization routines based on genetic algorithms. The package
  Rmalschains
  implements memetic algorithms with local search chains, which are a
  special type of evolutionary algorithms, combining a steady state
  genetic algorithm with local search for real-valued parameter
  optimization.


What is the way to modify a neural network classifier to deal with sample points from outside of the label set? I am solving an image classification problem. However some photos may not belong to any category, and I'd like not to give any fake information, rather to capture this situation. What are the ways to do it. One idea I have in mind is to give an extra category "OTHER" but then I need to have some data for this category which would be difficult to gather as it's too general.

This question is a very common question.
How to handle samples belong to categories that are not included in the data set. There is no optimal way to handle this:

Creating a separate class is possible but not recommended. First, you need images to start with, second this class will have a very large variation within the class, a property that is not favorable which makes designing a powerful classifier to handle the other classes more challenging and usually results in a low quality classifier.

I prefer to handle this by measuring the confidence. If you are expecting to have anomalies/ outliers,  make sure to use classifier that gives both prediction and probabilities ( confidence).

CNN is good enough , you can only accept predictions if the confidence is a higher than a threshold, or the difference between the highest confidence ( label) and the next confidence (label) is higher than a threshold. Most of the probability based classifies can give you confidence values.

Some other classifiers can be changed to give confidence, for example, you can analyze the training data used to give a label for a leave in decision tree, these samples belonging to the same leave can be clustered to get some centroids. In prediction you can predict the label and measure the shortest distance between your target ( testing sample) and each of the centroids belonging to the same leave you end with, that shortest distance can be used as a measure of confidence and can be compared to a threshold to accept / reject the prediction.

Please note that this problem is still under research and there is no clear answer to it. It needs very deep understanding to each classifier to modify it / or add to it to measure confidence and to understand what  that measure means


Difference: Replicator Neural Network vs. Autoencoder I'm currently studying papers about outlier detection using RNN's (Replicator Neural Networks) and wonder what is the particular difference to Autoencoders? RNN's seem to be treaded for many as the holy grail of outlier/anomaly detection, however the idea seems to be pretty old to, as autoencoders have been there for a long while.

Both types of networks try to reconstruct the input after feeding it through some kind of compression / decompression mechanism. For outlier detection the reconstruction error between input and output is measured - outliers are expected to have a higher reconstruction error.

The main difference seems to be the way how the input is compressed:

Plain autoencoders squeeze the input through a hidden layer that has fewer neurons than the input/output layers.. that way the network has to learn a compressed representation of the data.

Replicator neural networks squeeze the data through a hidden layer that uses a staircase-like activation function. The staircase-like activation function makes the network compress the data by assigning it to a certain number of clusters (depending on the number of neurons and number of steps).

From Replicator Neural Networks for Outlier Modeling in
Segmental Speech Recognition:

  RNNs were originally introduced in the field of data compression [5].
  Hawkins et al. proposed it for outlier modeling [4]. In both papers a
  5-layer structure is recommended, with a linear output layer and a
  special staircase-like activation function in the middle layer (see
  Fig. 2). The role of this activation function is to quantize the
  vector of middle hidden layer outputs into grid points and so arrange
  the data points into a number of clusters.


SciKit Learn: Multilayer perceptron early stopping, restore best weights In the SciKit documentation of the MLP classifier, there is the early_stopping flag which allows to stop the learning if there is not any improvement in several iterations. However, it does not seem specified if the best weights found are restored or the final weights fo the model are those obtained at the last iteration. If not, is possible to restore the best weights found?

To restore the best weights you would need a way to monitor a metric of your choosing and keep track of the best weights up to a given point. This has nothing to do with early stopping, you could return the best weights regardless of how many iterations the classifier has performed. Unfortunately, as far as I know, this functionality is not supported by scikit-learn.

In early stopping you assume that the best weights are those that the point you stopped your training. Early stopping isn't as much a convergence speed technique as it is a regularization, one.


How to optimize the lambdas of a hybrid loss in a deep learning model I am using a generative adversarial deep learning model (GAN) with a hybrid loss represented by a linear combination of four losses with three $\lambda$'s, something like:

$total\_loss = loss_1 + \lambda_1\times loss_2 + \lambda_2\times loss_3 + \lambda_3\times loss_4$.

Is there a way to optimize these $\lambda$'s towards the best performance? provided that the computational complexity is not trivial. If my objective is for $loss_4$ to be minimum, should I use a high value for $\lambda_3$ compared to the other ones? But, how would that affect the overall performance, as all the models used in the GAN might play an important role in the final result.

NB. $loss_1$ is based on MSE measure, while the other losses are based on $l_1$ norm.

When there is an optimization problem involving more than one objective function to be optimized simultaneously, it is called multi-objective optimization.

Gradient descent methods can still be used to minimize the overall objective function. A paper entitled "Gradient-Based Multiobjective Optimization with
Uncertainties" goes into greater detail.

In PyTorch, autograd.backward can handle multiple objectives. You need to provide a list of tensors for backpropagation.


Bias and variance in the model o in the predictions? This topic confuses me. In the literature or articles, when talking about bias and variance in automatic learning, specifically in cross-validation, do they refer to the high bias (underfitting) and high variance (overfitting) in the model? Or do they refer to the bias and variance of the predictions obtained in the iterations of the cross-validation? How to handle each case?

In some cases, you may have a model that's a black box - you feed input features and get output predictions, without knowing or caring what happens in the middle. In those situations, the model is, in a way, defined by its output - two models that produce the same predictions are indistinguishable from one another, even though they may be entirely distinct models. In these situations, saying a model is biased or has high variance is equivalent to saying the predictions of the model are biased or have high variance. It's acceptable to describe both the model and its output in terms of bias/variance. Cross validation provides an unbiased estimate of bias/variance for your model/predictions.


neural network - target data For the simple AND learning with a perceptron, it is required to have two inputs x1 and x2 and one target data y.

Most AND examples (such as in the book "Fundamentals of neural network-fausett") have this network topology in the figure above, that is 2 inputs(x1,x2) and 1 output(o). My question is, how do we give the network the target data (y) if there is only two inputs(x1,x2)?

The target data is the desired output that happens to be in the real word. This information is known to you already(given the problem is supervised learning problem). So, for an 'AND' gate the target value is the vector stored in y on the right hand side as shown below:

x1    x2      y(desired output)
0     0          0
0     1          0
1     0          0
1     1          1

So this vector y = [0,0,0,1] is your target to these input types that you feed into the network as labels corresponding to your above input data.


Does gradient boosting algorithm error always decrease faster and lower on training data? I am building another XGBoost model and I'm really trying not to overfit the data. I split my data into train and test set and fit the model with early stopping based on the test-set error which results in the following loss plot:

I'd say this is pretty standard plot with boosting algorithms as XGBoost. My reasoning is that my point of interest is mostly the performance on the test set and until the XGBoost stopped training around 600th epoch due to early stopping the test-set loss was still decreasing. On the other way, overfitting is sometimes defined as a situation when train error decreases faster than test error and this is exactly what happens here. But my intuition is that decision-tree based techniques always drill down the training data (for example, trees in random forest kind of deliberately overfit the training set and it's a role of bagging to reduce the variance). I believe gradient boosting techniques are also known from drilling down the training dataset pretty deep and even with low learning rate we can't help it.
But I may be wrong. Therefore, I'd like to confirm that this situation is not really something to worry about and I do not overfit the data. I'd also like to ask what the perfect learning curves plot would look like with gradient boosting techniques?

You should worry about overfitting when the test error rate starts to go up again. Until then I would set it aside. Overfitting is rather about the number of parameters, e.g. When two models with the same performance have different number of parameters you would prefer the one with less parameters to preserve the generalisation power of the model.
Generally the training error will be lower than the test error for obvious reasons (the model specifically sees the training data but not the test data), but even this can be reversed in some cases. For example when there is very little test data, or the test data has a different distribution than the training data.
A "perfect" learning curve doesn't exist, the final performance is what counts.


Latent variable graph in Variational Autoencoder I followed this Keras documentation guide about Auto Encoders.
At the end of the documentation there is the graph of the latent variable z:

But I can not understand and how to interpret the plot, and how should the plot change as the hyperparameters of the model change?

As Nikos said in a comment, this is a graph showing the different classes (digits) in the space of the latent variable.
First a remark: from the point of view of visualization design I think the authors made a mistake by using a gradient of colours for the classes, it should have been a completely different colour for every digit: here the digits are a categorical variable, there is no notion of order between them. This is misleading because the reader of the graph would naturally expect some logic in the gradient, but there is none.
The space itself is not interpretable, in the sense that the values on the X and Y don't have a meaning (for a human). However what this graph can tell you is how close or how distinguishable are two classes: ideally the different classes are fairly distinct from each other, which means that the model is sucessful at separating them.
Unfortunately due to the mistake I mentioned above it's quite difficult to see things clearly in this graph, but if I see the colours correctly there are a few things observable:

the 1 is the big purple thing on the top right, it's mostly distinct from the rest.
The 0 is the smaller dark purple group on the top left, also quite distinct.

The other digits are much more mixed with each other, it's hard to see anything:

Apparently the 9 (yellow) spreads a bit everywhere in the centre, and there are many points for 7 (I think?) in the middle: this means that potentially the model could sometimes confuse 7 and 9 (it makes sense: the handwritten 7 and 9 can have the same shape).
The 6 seems to be the rather thin greenish thing which is closest to the 0, and I can imagine indeed that these two have a similar shape.

My eyes are not that good so maybe I made some mistakes, but I hope you see the idea.
Basically the more the different classes/colours are well separated on the graph, the better the model, so you should be able to see some difference between some very good hyper-parameters and some very bad ones. But it would be pointless to rely on the graph for that: a good old evaluation measure (or confusion matrix) is much more accurate.


What is Deep learning approach to count the number of Diamonds in an image? I am working on a project which involves counting the number of diamonds in the provided image. I have a set of images and a VIA annotated .json file which has all the annotations. How do I proceed with this problem.

what approach to start i.e Object detection vs Instance segmentation ?
what models to choose from MaskRCNN,U-Net, etc (Please suggest) ?

Please guide.
Thanks

I say that go with Object Detection. Because you already have annoted images. And segmentation applied to locate objects and boundaries (lines, curves, etc.) in images. . In Recent days EfficientDet Works very well in object detection. And for implementation look here. Best of luck.


Loss and Regularization inference I'm building a Matrix Factorization model for MovieLens dataset with batch-wise training. Loss function for the batch:
$$
L_{batch} = 1/|B|\sum_{(u,i)\in{B}}(r_{ui} - \mu - b_u -  b_i - p_u^Tq_i)^2 + \lambda(||p_u||^2 + ||q_i||^2)
$$
$$
L_{batch} = (L_{base\_loss} + L_{reg\_loss})/|B|
$$
$$
L_{base\_loss} = \sum_{(u,i)\in{B}}(r_{ui} - \mu - b_u -  b_i - p_u^Tq_i)^2
$$
$$
L_{reg\_loss} = \sum_{(u,i)\in{B}}\lambda(||p_u||^2 + ||q_i||^2)
$$
where $r_{ui}$ is the observed rating, $\mu$ is the global average rating, $b_u$ and $b_i$ are the average deviations of user $u$ and item $i$ from the global average rating respectively, $p_u$ and $q_i$ are the learned user embeddings and movie embeddings respectively, $B$ is the batch.

What should I infer from regularization loss going up? Model is not able to capture the underlying information using the current embedding size, or the scale of regularization loss is too less compared to the base loss?

There is no problem with the fact that your regularization loss is going up.

The cost function of your model is a weighted sum of the regularization loss and the base loss, so during training the model looks to minimize them together, but eventually it comes to a point where it has to choose and minimize one on the expense of the other. The fact that it is prefers the base loss means that your regularization hyper-parameter is not too big (eventually your goal is to reduce the base loss).

In general terms: The goal of the regularization loss is to simplify the model during training - demanding smaller weights forces the model to increase the size of just the important weights, and not all of them. So some wights converge to very small and insignificant values, effectively reducing the number of weights in the model. Smaller model means less overfitting and more generalization.

In your case: During the training of your model, it starts from reducing the regularization loss (because it is an easy task), but at some point it starts to notice crucial features which it wants to emphasize so it begins to increase the weights of those features, which results in higher regularization loss but a smaller total loss (a good thing, it is learning but is also under control).


Merging two datasets with unclear identifier in Excel I'm trying to merge two datasets on Mergers & Acquisitions. They both consist of c.10'000 observations with c.50-100 variables each. One contains information about the actual M&A deal whereas the other one contains info on how a deal was financed.

The problem is that there is no clear and unique identifier. For example, I could use the date that the deal was announced but that wouldn't be unique because on some days 10 deals were announced. Using company names is difficult since they mostly aren't identical in both datasets. For example if in one dataset I find "Ebay", in the other the same company could be called "eBay", "Ebay Inc", or "Ebay, Inc."

I've been working with the Fuzzy Lookup add-on for Excel, as well as concacenating various identifiers that are not unique but in their combination become useful (e.g. Date & Country & SIC Industry Classification Code, etc.). However I haven't been able to generate as many matches as I would have hoped.

I'd be grateful for any ideas or pointers towards resources that would help me merge the datasets more efficiently.

I typically proceed the following way,

convert to the same case with LOWER() or UPPER();
remove all non-alphanumeric characters with several SUBSTITUTE(), e.g. SUBSTITUTE(A1, ".", "");
trim white spaces TRIM();
extract the first word with LEFT(A1, FIND(" ", A1) - 1);
trim white spaces again;
perform a join on the key that you created.

Often this gives you a decent match.


Suitable metric choice for imbalanced multi-class dataset (classes have equal importance) What type of metrics I should use to evaluate my classification models, given that I have two imbalanced multi-class datasets (21 and 16 classes, respectively) where all classes have equal importance?
I am somehow convinced with macro-averaged-based metrics choice such as macro F1 and macro TNR, ...etc. Are macro-averaged-based metrics suitable for my problem based on the aforementioned inputs?

Yes, a macro-average measure is the standard choice in this context: a macro-average score is simply the mean of the individual score for every class, thus it treats every class equally.
With an strongly imbalanced dataset, this means that a small class which has only a few instances instances in the data is given as much weight as the majority class. Since the former is generally harder for a classifier to correctly identify, the macro-average performance value is usually lower than a micro-average one (this is normal of course).


LeNet for Convolution network? I keep seeing LeNet used to referring to a convolution network? I am wondering why LeNet is called LeNet? Is it the abbreviation of anything? Is there a difference between LeNet and convolutional neural network? Thanks!

LeNet is a family (LeNet-1, LeNet-4, LeNet-5) of convolutional neural network designed by Yann LeCun et al. The name is a play on his name and the French language, where "le" means "the", hence LeNet means "the network". I believe it was originally devised to recognize handwritten numbers on checks (cheques). LeNet is only one early instance of a convolutional neural network; many others exist today.


How to create a system to detect text structure of a file? Let's say I want to create a Machine Learning system that has a lot of log files of some few types (F1, F2,.. Fn) and I get a new Log file with maybe some errors or missing data.

How do I classify it into these class types or classify it is an anomaly if it doesn't belong to anyone of them.

I thought about anomaly detection but couldn't figure how to parse structure information from the text classes like (F1, F2... .etc).

Also what kind of structural information to extract from text files?

These input classes contain 100 - 1000 lines of code per document of each class type.

I looked into Linting or DeepCode ...

A sample log file looks like this:

11-02-11 16:47:35,985 +0000 E Activity class {com.trackingeng/LandingActivity} does not exist.
12-02-11 17:47:35,985 +0000 I Starting: Intent { act=android.intent.action.MAIN
.....

A log file may have stack trace like this also:

Error:
    Error detail 1
    Error detail 2
    ....
Non-Error:
    .....
Warning:
    .....
and similar to this.

Any help in which direction to look for is greatly appreciated.

Based on your current examples.

You have an ML that is doing work across multiple systems and users. Generating logs.
The logs can be grouped or classed (e.g. $F1, F2,...,FN$) by:

i) the operating System used (e.g. Android, Windows etc) and
ii) the error or message generated. The error messages are differentiated by tab white space.

If you cannot work out the log class using some logic, you could look to engineer features with regex to capture the source OS and the tabbed structural information in the error lines.

|---------------------|------------------|
| Error detail        |     n_tabs       |
|---------------------|------------------|
|  string_error       |         0        |
|---------------------|------------------|


Predict votes of future comments I have a database with a lot of comments, each comment has a vote, a vote can be positive or negative. ex : -2, -5, -90, +45, +20...

So based on this training dataset I want to predict votes of future comments. How is it possible ? Thank you in advance

If you are hoping to process the text of the comments this is definitely possible. Just NLP training and a single output score.

But this might be more machinery than you're looking for? If you have more data on the comments this will be do-able without a neural net.


How many epochs is enough? Is there any formula between the number of training inputs, the number of features and the number of epochs that's enough to learn the model?

As far I know, no formula.

But have some approaches(can be guidelines), which will help you out in analyzing your requirements.

"training inputs", higher the training data, better the results we can expect.
"number of features", If the number of features are high, model performance can be decreased due to over fitting. We dont have any formula for optimal features identification. If I'm correct l1(lasso) regularization, will do it the same internally and optimize the number of features by setting coefficient values as 0.
Epoch, generally we observe consecutive losses of "n" epochs. If it's not reducing, then we can terminate further training and you best model so far we observed. If model is still not acceptable, then we have to retrain the model with certain changes like review  features selection process, hyper-parameter tuning, algorithm selection and so on.


feature extraction for a pretrained model in keras Keras has a way to extract the features of a pretrained model, described here https://keras.io/applications/

from keras.applications.vgg16 import VGG16
from keras.preprocessing import image
from keras.applications.vgg16 import preprocess_input
import numpy as np

model = VGG16(weights='imagenet', include_top=False)

img_path = 'elephant.jpg'
img = image.load_img(img_path, target_size=(224, 224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)

features = model.predict(x)

I tried this with some sample images. My vector x is of shape (100, 3, 224, 224) for 100 observations, 3 for RGB and 224x224 pixel size.
the preprocess_input reshapes this for the VGG model (it expects a different order).

However, the output shape of features is (100, 512, 7, 7). What is this shape?
I want to use the features as input for a logistic regression. So I need a shape like (100, n): one row for each observation and the features in the columns.
How do I reshape the output to this dimension?

Say I now want to build my own simple Convnet:

from keras.models import Sequential
from keras.layers import Convolution2D, MaxPooling2D
from keras.layers import Activation, Dropout, Flatten, Dense

model = Sequential()
model.add(Convolution2D(32, 3, 3, input_shape=(1, 299, 299)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Convolution2D(32, 3, 3))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Convolution2D(64, 3, 3))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

This model expects grayscale images as input, hence the shape.

What kind of layer do I have to add to get features of this model (something I can input in a logistic regression or random forest).

Thanks

The features variable contains the outputs of the final convolutional layers of your network. The final convolutional layer of VGG16 outputs 512 7x7 feature maps. All you need to do in order to use these features in a logistic regression model (or any other model) is reshape it to a 2D tensor, as you say.

reshaped_features = features.reshape(100, 512*7*7)

This will flatten out the feature maps to a long one-dimensional vector for each instance.

Tip: If you can't be bothered working out the actual dimensions for your reshape, you can replace 512*7*7 with -1 and numpy will figure out how big the final dimension should be.


Why is this Binning by Median code wrong? I was working on Binning by Mean, Median and Boundary in R.

# R CODE
a=c(20.5, 52.5, 62.6, 72.4, 104.8, 63.9, 35.3, 83.9, 37.4, 71.6, 74.6, 44.5, 66.6, 56.1, 45.3, 37.2)
a=sort(a)
binsize=4
median(a[1:4])
# BINS ARE
for(i in 1:length(a))
{
  if(i%%binsize==0)
  {
    print("HI")
    print(a[i-3])
    print(a[i-2])
    print(a[i-1])
    print(a[i])
  }
}

# BINNING BY MEAN
sum=0
for(i in 1:length(a))
{
  sum=sum+a[i]
    if(i%%binsize==0){
      avg=sum/binsize
      sum=0
      print(rep.int(avg,binsize))
    }
}

# BINNING BY MEDIAN
i=1
for(i in 1:length(a)){
  if(i%%binsize==0)
  {
    print(rep.int(median(a[i-3:i]),binsize))
  }
}

Can anyone tell me why binning by median is giving me wrong output.
The median(a[i-3:i]) for 1st bin returns a value which is not same as median(a[1:4]) for 1st bin. Why?

The error is due to the well-known R gotcha that the : (colon operator, which calls seq()) takes higher precedence than arithmetic. Always parenthesize arguments to : if they involve arithmetic or are expressions: a[(i-3):i]

Your code a[i-3:i] doesn't do what you want it to do a[(i-3):i], it does a[i - (3:i)]). So the medians you are printing are for these slices:

4-3:1 # i.e. 1:3
8-3:1 # i.e. 5:7
12-3:1 # i.e. 9:11
16-4:1 # i.e. 13:15

PS some coding-style tips

You don't need to iterate over all possible values of i and check them modulo binsize, just do:

for(i in seq(binsize, length(a), by=binsize)) { ... }

so in your median case:

for(i in seq(binsize, length(a), by=binsize)) {
  print(rep.int(median(a[(i-3):i]),binsize))
}
[1] 36.25 36.25 36.25 36.25
[1] 48.9 48.9 48.9 48.9
[1] 65.25 65.25 65.25 65.25
[1] 79.25 79.25 79.25 79.25

But in fact you can replace even that with:

split(a, ceiling(seq_along(a)/binsize))

as per the "Split a vector into chunks in R"

To make it even clearer, you could define a helper function chunk <- function(x, binsize) { split(x, ceiling(seq_along(x)/binsize)) }
Then you can replace the for-loop with sapply:

.

sapply(split(a, ceiling(seq_along(a)/binsize)), mean)
sapply(chunk(a,binsize), mean)
     1      2      3      4
32.600 49.600 66.175 83.925

sapply(split(a, ceiling(seq_along(a)/binsize)), median)
sapply(chunk(a,binsize), median)
    1     2     3     4
36.25 48.90 65.25 79.25

Much cleaner, easier to read, and prevents errors, right?

The generalization of the colon operator is the seq() function, give it a read, it's pretty useful.


How to best estimate the coefficients of a confusion matrix in case of strong class imbalance? I have a trained binary classifier (forget about how this was trained and think of it as a magical black box) and I would like to measure its classification performance (e.g. compute a confusion matrix) on a strongly imbalanced population/test dataset.
Suppose the composition (in terms of the two classes) of the population / test set is also known - say 99% of class $A$ vs 1% of $B$.

I also would like to reduce as much as possible the uncertainty in the estimate coefficients of the confusion matrix computed on the entire (large) test set under the constraint of doing predictions for a max of $N$ examples of the test set (suppose that the predictions are expensive to run).

What's the best approach to do this?

The best thing I can come up with is doing something like "stratified sampling":

I create from my test set (which is arbitrarily large for simplicity)  a balanced sample with $N/2$  examples belonging to $A$ and $N/2$ examples belonging to $B$.
I run the classifier on the balance dataset, and compute the resulting confusion matrix.
I "normalize" the so-obtained confusion matrix so that its rows sum to 1, obtaining $\hat{\mathbf C}$.
To estimate what would be the confusion matrix on a large representative sample (say of $M$ items taken from my large test set) I compute

$$
\mathbf C =
\left(
\begin{matrix}
0.99M & 0 \\
0 & 0.01M
\end{matrix}
\right)
\cdot \hat{\mathbf C}
$$

P.S.: To be really happy, I would like to estimate also the uncertainty of the components of the final confusion matrix $\mathbf C$.
To do that, can I just compute e.g. the credible intervals of the coefficients of $\hat{\mathbf C}$ starting from the balanced sample, and "rescale" them using the definition of $\mathbf C$ given above?

Bonus track: if somebody could send me with a reference about how to do some kind of "power test" (e.g. "how large should $N$ be to "bracket" the values of $\mathbf C$ within given tight intervals?") I'd be glad to review it.

Bonus$²$ track: any tips on how to extend the estimation of the uncertainty to nonbinary classifiers (more than 2 classes) are also welcome! $\ddot\smile$

There are many useful metrics which were introduced for evaluating the performance of classification methods for imbalanced data-sets. Some of them are Kappa, CEN, MCEN, MCC, DP, etc.

Disclaimer:

If you use python, PyCM module can help you to find and calculate these metrics.

Here is a simple code to get the recommended parameters from this module:

>>> from pycm import *

>>> cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})

>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]

>>> score = cm.Kappa


Which ML algorithm to use if we have categorical data, numeric data, derived data (derived from) other variable in our data set? I am a beginner in Data Science. I have a data set which contains numerical data, categorical data and derived data (derived from other columns). The target column (dependent) is binary. Which Machine Learning algorithm will predict my target column most accurate?

This question is way too vague, you just describes pretty much a standard situation. Which algorithm / method you use depends entirely on your objective and data. You should provide more information if you want a specific answer. Since your dependent variable is binary you are interested in classification, popular algorithms are Random Forests, Logistic Regression, K-means, etc.


How to find similarity/distance matrix with mixed Continuous and Categorical data? Say I have a dataset like this:

Hotel  HasPool AvgPrice
 1        1      $123
 2        0      $234
 3        1      $200

Currently I have broken down the dataset into 2 (one containing all continuous, other all categorical). The continuous one, I am calculating euclidean distance b/w all observation whereas for the categorical one I calculate Cosine distance . Is there a way to combine both scores effectively. Or, is there a distance function which works for both data types ?
My final output should look like this:

Hotel  1   2   3
  1    1  0.3  0.7
  2    0.3  1  0.5
  3    0.7  0.5 1

Any help will be appreciated! Thanks

Similarity measures are subjective and so are they ways to combine them.
You should decide what is your subjective definition of similarity and then find a way to combine them that fit your definition.

In general, I like to reduce similarity problems into classification problems. Given the dataset of items you have, create a new dataset of item pairs. The concept should be whether the two items in a pair are similar. Each similarity measure you have is a feature of the pair. Note that now you are in the good old classification framework. You can evaluate the similarity measures by computing the mutual information/accuracy/your chosen metric given the concept.

In your case, I would build the dataset this way. First choose pairs of items on which both your measures agree on. Choose pairs that are close by both the Euclidian distance and the cosine distance or pairs that are far by both measures. Note that since the similarity measures agree on these pair, combining them usually  leads to agreement too and not that important. However, you should use such pairs so they will stabilize your weighting function. Other than that, you can easily generate these pair, which is always an advantage.

Next, you should choose pair on which your similarity function disagree. Choose some that are considered close by the first and far by the second and vice versa. Manually label the pairs in order to decide whether they are close or not.

Once you built the dataset, you can present your subjective definition of similarity as a set of pairs and distances. Now run any classifier (in your case a linear regression might fit what you look for) and use the result as the combined similarity measure.


What techniques are used to understand call patterns? I have customer data since 2013 and there is a file which has the customer unique id, a timestamp, and the reason for the call (a drop down from the person who handled the call).

I did some cumulative counts based on customer ID and the timestamp and I saw that one customer called in over 1000 times alone. What's the best way to make sense of the call driver data when I'm looking at millions of rows and around 200 categories of call types?

Is there a broader topic which looks into 'downstream'  issues or predicting the probability of future calls or events?  The end goal would be to visualize these calling patterns and focus on reducing the call backs. This is a specific problem but it seems like it should be common and I can learn about addressing it in a bigger picture manner.

When you said "a broader topic", did you mean what algorithms to use to examine the event log with the goal of reducing future calls from the same customer on the same topic? In other words, a customer may call for help for a different topic. If a customer calls in for the same topic repetitively, something can be improved.

You may get ideas from a Coursera class Processing Mining since the issue you're solving is similar to the example of a spaghetti process in lecture 6.7:

"Spaghetti process describing the diagnosis and treatment of 2765 patients in a Dutch hospital. The process model was constructed based on an event log containing 114,592 events. There are 619 different activities (taking event types into account) executed by 266 different individuals (doctors, nurses, etc.)."

By the way, you can click the drop down menu under "Sessions", choose "April 1, 2015 to May 19, 2015" then you can register and view the lectures. On the right of each lecture, there are some icons. The first icon is to download slides for the lecture. You may find reading the slides is faster than listening to the lecture.

Suppose a customer called for installation of software release 1.5 then called a day later for running the new features of software release 1.5. Are these two issues logged as two of the 200 call categories? If so, we can use a time period (say one week) to judge whether it is about the same topic. Within a short time period, a customer is likely to work on the same topic, especially with the same key words such as "software release 1.5". We can coach the call center employees on solving possible follow up questions by saying something like "now that we finished installation, let me show you a couple of new features. It'll save you time." This will reduce the number of calls on the same topic from the same customer.


PCA - what do I do with its results? I have a data set with more than 20 features, and I applied PCA:

M.fit_transform(all_data)
variance = M.explained_variance_ratio_
var = np.cumsum(np.round(M.explained_variance_ratio_, decimals=3)*100)
plt.ylabel('% Variance Explained')
plt.xlabel('# of Features')
plt.title('PCA Analysis')
plt.ylim(30,102.5)
plt.plot(var, marker="s")
plt.show()

Printing the var variable, I get

array([ 89., 100., 100., 100., 100., 100., 100., 100., 100., 100.])

I understand this tells us that the variance is explained by 2 features.

So I calculated it again, now the 2 components:

from sklearn.decomposition import PCA
M = PCA(n_components = 2)
X = M.fit_transform(all_data)
plt.scatter(X[:,0],X[:,1])

And this gives a "random looking plot". I understand that the data was changed during the PCA process.

What can I do with this information? How will this help me understand the data?

Is it useful per se? Is it useful as a preparation method for other methods? Which ones can I try?

What can I do with this information?

You can do a lot of things with this data. You can visualize it, you can use the vectors for prediction or regression, whatever the task at hand. However, there are a few restrictions of PCA that you need to keep in mind. For eg. its very memory intensive, so you need to have a "lot" of RAM to use PCA on certain data-sets.

How will this help me understand the data?

You can visualize the data like this (image is taken from http://www.nlpca.org/pca_principal_component_analysis.html):

With reference to the above image, you can see that the data-points can be clearly separated into different clusters. Using this, you can apply K-Means and get different cluster centres. Using these cluster centres, you can further investigate and find additional insights.

Is it useful per see?

PCA is a dimensionality reduction technique which is very memory intensive.
If you have the required memory, you can easily reduce the number of features by 50-80 %, while still retaining a good amount of information. For eg, we can reduce 100 features to 20-30 features which contain maximum amount of information.
While performing PCA, it's important to check if the matrix computation can be done with the RAM that you have, otherwise, you can check out Iterative - PCA.

Is it useful as a preparation method for other methods?

It is very useful as a preparation method for clustering and visualization.
Please refer this link : https://qiita.com/bmj0114/items/db9145a707cb6ed13201

Which ones can I try?

You can try the example given in the link above.


A Simple Explanation of ZooKeeper in Hadoop Can Anybody Explain me the role of ZooKeeper and ZooKeeper Quorum in a BigData structure?

I'd appreciate if you explain me with a simple practical example instead of abstract theoretical definitions.

Please

As you would explain for a non-technical person

I noticed you are asking several questions related to the Hadoop ecosystem. It is hard to judge how much you know while explaining the things you are asking in a very simplistic manner without knowing that. Might be worthwhile giving some background of what you are trying to understand. But I digress.

Unless you are building an application that requires coordination (think race conditions) you more than likely won't need Zookeeper directly. Many tools within the Hadoop ecosystem do use it for this exact purpose (e.g. HBase).

This is probably the best and most simplistic answer I can give.

  In a nutshell, ZooKeeper gives you the tools to help build distributed applications. For example, suppose you have a distributed web server application running on 10 nodes. Say, you want to get total real-time hit count. One way to do this is to write an application which connects to the 10 nodes, gets count from each and present the sum. Alternatively, you can have each web server application write their hit counts to ZooKeeper on regular intervals and then query ZooKeeper to get the count.

Source: https://10kloc.wordpress.com/2013/12/21/quick-intro-to-apache-zookeeper/

ZooKeeper Quorum is "A replicated group of servers in the same application is called a quorum, and in replicated mode, all servers in the quorum have copies of the same configuration file."

Source: https://zookeeper.apache.org/doc/r3.1.2/zookeeperStarted.html

A more detailed answer of what Zookeeper is and how it is beneficial can be found on SO (https://stackoverflow.com/questions/3662995/explaining-apache-zookeeper).


Build text complexity model based on complex examples I try to build the user specific model which predicts whether arbitrary English text is complex for particular user or not. Having the complex and easy text samples allows to build such model but what if I have only complex samples. How can I build the model in such case?

I can detect whether the given text is different (find the "outlier") from those which user marked as difficult. But that information does not tell me in which way it's different. The text could be easier or more difficult.

Currently I see only one way - make an assumption about how the easy text could look like. But it's kind of unsafe since different people might have own unique areas which they do not understand in the text.

There have been many ways to measure text complexity proposed in the literature, I don't have any particular survey to recommend but google is your friend.

Many of these measures are heuristics, i.e. they work in an unsupervised way. I don't remember the details but I've seen some works using a combination of several of these measures to obtain more accurate results.

A basic way would be to be build a language model on the complex text, measure the complexity against this model for any new text and assume that if it's not similar then it's not complex, but as you rightly noticed it's not a very safe assumption.

At the most basic level, you can use the type token ratio (TTR): divide the number of types (unique tokens) by the total number of tokens. The TTR is a quite good indicator of lexical diversity, so complex text is likely to give a high value. It's a very crude measure but it's useful as a baseline: whatever system you try, if it doesn't give better results than a threshold on the TTR then it's not a good system :)


Sentiment Analysis model for Spanish I barely know about Data Analysis tools and techniques, so bare with me if I'm asking something too trivial.

I'm looking for a Sentiment Analysis tool to process comments in Spanish. I do know some options for Sentiment analysis but those all work for English.

Is there a model/tool that already works with Spanish?

I'm language agnostic so it does not matter if it's a Java, Python or even Go code.

The Indico.io API supports Spanish (and
Chinese (Mandarin), Japanese, Italian, French, Russian, Arabic, German, English).

eg in Python:

>>> import indicoio
>>> indicoio.config.api_key = <YOUR_API_KEY>
>>> indicoio.sentiment("¡Jamás voy a usar esta maldita aplicación!  No funciona para nada.")
0.02919392219306888
>>> indicoio.sentiment("¡Es patrón!  La mejor que he visto.  Punto.")
0.8860221705630639

If this isn't your area, then that's probably the easiest sort of solution to integrate.


Why are HMMs called linear-chain? I found in many sources that Hidden Markov Models are linear-chain networks(e.g. in Predicting Structured Data book by MIT). However, as I understand it, HMMs can have any edges in its graph. Even simple example of HMM in wikipedia has non-linear graph:  .

So, the question is: what is the formal definition linear-chain structure and in which case forward-backward and Viterbi algorithms can give precise results.

I have also taken into consideration this picture, taken from CRF tutorial, which says, that linear-chain CRF is "generative-discriminative pair" to HMM.

It is so called because it classifies the linear sequence, and not because the structure of the graph.


Implementation of Gaussian Mixture Model for clustering when dealing with multidimensional hyperspectral data in python I have a python numpy array of size (800,800,4) which is my hyperspectral camera data.

When I try standard GMM methods from scikit-learn I get an error saying that the expected dimension of the data should be less than or equal to 2.

Current method:

gmm=GaussianMixture(covariance_type="full")

gmm=gmm.fit(data)

Are there any python packages or functions available to process higher dimensional data using gmm?

If I understand your type of data correctly, what you have is essentially an image with $800\times800 =640000$ pixels and each pixel has a vector of 4 values.

If this stands, I suppose you could then transform your data to a $640000\times4$ matrix, so as to conform with scikit-learn's data representation schema of inputting matrices of shape ($\#samples\times\#features$) and then you could use the GMM class implemented by the package.

And you could transform the data back if needed.

P.S.: On a second thought, I don't know if the spatial distribution of pixels plays a role on the generation of the GMM components. Maybe try it out and let us know?

Ok, after you have trained your model you can ask for it to predict the mixture model from which each pixel in your dataset has been generated.

That is if you run again:
$clusters=gmm.predict(newdata)$ you will get an array with values $0-2$, denoting the cluster that each pixel belongs to. You can, then map these values in the image at hand, applying a mask of different color for each pixel in order to see if your model correctly identifies the three classes.

Here is an example of what I mean:

from scipy.misc import imread, imshow
from sklearn.datasets import load_sample_image

X = load_sample_image('china.jpg')
old_shape = X.shape
X = X.reshape(-1,3)
gmm = mixture.GMM(covariance_type='full', n_components=5)
gmm.fit(X)
clusters = gmm.predict(X)
clusters = clusters.reshape(old_shape[0], old_shape[1])
imshow(clusters)