Statistics - Train and test data split How much data should we use during training, and how much in testing? Can anyone explain why does it always seem to be 70:30 or 80:20 ratios?

Those rule of thumbs for ratio make no sense. The use case for a test set is to measure your performance in a real scenario on unseen data. How much you need for an accurate measure depends on the level of accuracy you need, the amount of variance you expect, not so much on the size of the original dataset. Something to be aware of is that you do decrease the size of your training set, which is a cost of testing. But when you have 10 million rows of data it makes no sense to validate on 3 million, when 10,000 could be more than enough. Outside of the cost of losing training data you should look at absolute numbers in your test set.


Limits of Hellinger distance values I am calculating Hellinger distance for different vectors. I initially assumed that the value returned by it in in the range of 0 to 1.

However for the following two vectors I received Hellinger score as 1.0488088481701514, which is > 1.

vector_1 = [0.0,0.5,0.7]
vector_2 = [1.0,0.0,0.0]

Now, I am curious to know the range of Hellinger distance values. Please explain me why that value exceeded 1.

It is bounded by unity, but your first vector does not encode a probability mass function, since 0.5 + 0.7 > 1.0. If the 0.7 had been 0.5 or the 0.5 a 0.3, the distance would have been 1.0 since the distributions are maximally separated, having no overlap.


How to find coherence between a large number of sentences I have a list of sentences returned as a result of a document search algorithm. I want to determine if the results returned are semantically close/similar/coherent using some sort of metric. For a starting point, I'm using Word Movers Distance (WMD) and calculating the similarity between the sentences. But my list of sentences is too long, and doing a pairwise comparison for all the items within a list (document) would be computationally infeasible. What might be the best way to solve this?

You could use clustering with a more basic similarity measure, for example cosine or even simply the proportion of words in common (e.g. Jaccard, overlap coefficient). This should gives you groups of sentences which are "quite similar" against each other, whereas sentences in different clusters are supposed to be very different. This way you would only have to compute WMD distance between smaller groups of sentences. By increasing the number of clusters the clusters would be smaller so there would be less WMD computation needed, however there is more risk to miss a pair of sentences since they could end up in different clusters.


Does gensim use Negative sampling in Word2vec? When I train a word2vec model in Gensim on a huge amount of words/data (let's say hundreds of thousands of word vectors), is gensim using negative sampling automatically?

Alternatively, is there a way to make it use it?

Gensim's word2vec can use negative sampling. When calling class gensim.models.word2vec.Word2Vec, there is a parameter negative. The default amount is negative=5.

  negative (int, optional) – If > 0, negative sampling will be used, the int for negative specifies how many “noise words” should be drawn (usually between 5-20). If set to 0, no negative sampling is used.

The documentation is here.


Why is my test data accuracy higher than my training data? I'm using four years of data, training on the first 3 and testing on the fourth. Using LSTM w/ Keras. My test data set (which has no overlap at all with the training) is consistently performing better than my training data. How should I interpret this? It seems very unusual. Here's the trail end of the model output. You can see my training accuracy for a given epoch hovers around 80%, but test output jumps to about 86%:

Epoch 8/10
9092/9092 [==============================] - 9s 964us/step - loss: 0.9870 - acc: 0.8185
Epoch 9/10
9092/9092 [==============================] - 9s 1ms/step - loss: 0.9670 - acc: 0.7996
Epoch 10/10
9092/9092 [==============================] - 9s 937us/step - loss: 0.9799 - acc: 0.7895
Test Set Accuracy: 85.96%

predicted     0    1
actual
0          2639  238
1           211  111

Edit: Here's my code to create & compile the model:

embedding_vector_length = 32
days = 30

model = Sequential()
model.add(Embedding(2080, embedding_vector_length, input_length=days)
model.add(LSTM(100))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
print(model.summary())
model.fit(train_x, train_y, epochs=3, batch_size=64,class_weight={0:1.,1:1})
scores = model.evaluate(test_x, test_y, verbose=0)
print("Test Set Accuracy: %.2f%%" % (scores[1]*100))

Your test-set is unlucky.

Use Cross-Validation

Also make sure to handle proper/random split for test and train data. So that it's not biased


Python - Feature Selection - Should I remove bad variables? I've this code to print the importance of each variable on my model:

importances = trained_model.feature_importances_
std = np.std([trained_model.feature_importances_ for trained_model in trained_model.estimators_], axis=0)
indices = np.argsort(importances)[::-1]

# Print the feature ranking
print("Feature ranking:")

for f in range(training_features.shape[1]):
    print("%d. feature %d (%f)" % (f + 1, indices[f], importances[indices[f]]))

I print a lot of variables with feature ranking as 0.0. Should I remove that variables? I can I do it using Python?

Like this:

df = df.drop('Col_A', 1) WHERE importances[indices[f]] = 0

Thanks!

Assuming your features are df[X] and your target is df[y], I would just do the following:

keepfeatures = X[trained_model.feature_importances_ > 0]
keepfeatures.append(y)

df = df[keepfeatures]


Feature Selection Algorithm for Attributes with Logical Relationships (like "AND") I'm looking at datasets where the the attributes and the target class have a logical relationsships. All attributes and the target class are binary.

Here's an example: Neither feature#1 nor feature#2 have a significant correlation with the target class. But the conjunction feature#1 AND feature#2 is correlated with the target class.

Are there any Feature Selection Algorithms able to cope with situations like that? I'm thinking that frequent itemsets could be useful.
It would be tremendously helpful, if anyone could point me to a related paper.

Cheers!

Emergent pattern mining could be useful for you. In classification, emergent patterns are subsets of features which differentiate one class from the others. Look at Example 1.1 in the paper. Two feature subests, denoted $X$ and $Y$, are EPs because $X$ has large support in one class and small support in the other, and vice versa for $Y$. The nifty thing about EP mining is that it considers subsets of features instead of individual features.

Mining patterns in this way could address your problem of feature1 and feature2 having no correlation with the target class alone, but the combination feature1 AND feature2 having high correlation with the target class.

Unfortunately, I don't see any open-source software implementations on the web. However, there are a number of papers out there which implement some sort of EP mining.


weightage attributes If I have a dataset with binary classification and has many attributes with value of  (0 or 1) means the occurrence of attribute is represented by 1 and absence is represented by 0, can I add weight of that attributes as separate column in dataset before applying classifier model?

Yes, it is possible. You don't really need a new column, you could just convert your existing columns to their representative weights.

I would suggest for you to:

Convert your [0,1] to [-1,1];
Multiply by a float weight in the range [0.0, 1.0].

I suggest you change the range from [0, 1] to [-1,1] because of the following example where it is not possible to differentiate the two:

attribute_value = 0
weight = 1.0
final_value = attribute_value × weight = 0.0

attribute_value = 1
weight = 0.0
final_value = attribute_value × weight = 0.0

However, if you change the range, you get:

attribute_value = 0
attribute_value_new = -1
weight = 1.0
final_value = attribute_value_new × weight = -1.0

attribute_value = 1
attribute_value_new = 1
weight = 0.0
final_value = attribute_value_new × weight = 0.0

A full example would be:

X = [
    [0, 1, 0, 1],
    [0, 0, 1, 1]
]

X_transformed = [
    [-1,  1, -1, 1],
    [-1, -1,  1, 1]
]

column_weights = [1, 0, 0.5, 0.25]

weighted_X = [
    [-1, 0, -0.5, 0.25],
    [-1, 0,  0.5, 0.25]
]


How many neurons a deep learning model (specifically a CNN) does have? I found total number of neurons of ResNet-50 model is 26,560 and 94,059 in two different papers. Their titles are below:
1: DeepXplore: Automated Whitebox Testing of Deep Learning Systems
2: Testing Deep Neural Network based Image Classifiers
I found another article (link) that clearly describes how many neurons a CNN does have. If I follow this technique then the neuron count does not match with the count mentioned in those papers.
Could anybody please help me how I can calculate number of neurons in a DL model? Also which layers have neurons and which don't have.

I hope you are talking about the parameter count in a convolutional neural network.
Below is basic understanding about calculating the parameters. After getting this, it would be self explanatory.
kernel count = k
kernel size = (a, a)
depth of input = d
Total number of parameters: k*(a*a)*d + k

Now, for below scenario which has one input layer with one con2d layer.

To cross check the number of parameters, we need k,d,a:
 k = 64, d = 3, a*a = 3*3

Total parameters: 64*(3*3)*3 + 64 = 1792

Hope its clear now.


BERT Model Evaluation Measure in terms of Syntax Correctness and Semantic Coherence For example I have an original sentence. The word barking corresponds to the word that is missing.

Original Sentence : The dog is barking.
Incomplete Sentence : The dog is ___________.

For example, using the BERT model, it predicts the word crying instead of
the word barking. How will I measure the accuracy of the BERT Model in terms of how syntactically correct and semantically coherent the predicted word is?

(For an instance, there are a lot of incomplete sentences, and the task is to evaluate BERT accuracy based on these incomplete sentences.)

In other words, how will I measure the distance in terms of semantics in terms of model between the two words barking and crying.

Please help.

You just stumble over one big problem in the NLP field : finding the perfect metric..

Most traditional metrics (BLEU, ROUGE, ...) simply does not take into account the distance in terms of semantics between barking and crying.

So according to these metrics, The dog is crying is as similar as The dog is salmon to the reference, the dog is barking.
From a human viewpoint, this is not correct, the first sentence is closer to the reference, because for example the second sentence makes no sense.

People recently tried to provide better metrics in this sense. You might be interested in BERT score.

The idea is simply to use a BERT model (that have been pretrained, therefore have some linguistic knowledge) to compute how similar 2 sentences are.


Train loss vs validation loss I have a few basic questions about tracking losses during training.

If I am using mini-batch training, should I validate after each
batch update or after I have seen the entire dataset?
What should be the condition to stop the training to prevent overfitting? Do you save the model at that point?
In case I use mini-batch training losses fluctuate a lot, depending on the random choice of training data, and sometimes validation loss is less than training loss. Is this normal? I think my confusion about this point will be answered in 1 by itself.

Both approaches can be done: I recommend to validate after every batch when your are just playing around with your gradient method, to see if the validation accuracy goes up or down and to figure out how everything is going.
In this setting, you can adopt early stopping, although there are many ways to prevent from overfitting. Early stopping checks, at the end of an epoch, your validation accuracy, and saves the model if it is the best so far. If the validation accuracy does not increase in the next $n$ epochs (and here $n$ is a parameter that you can decide), then you keep the last model you saved and stop your gradient method.
Validation loss can be lower than training loss, this happens sometimes. In this case,  you can state that you are not overfitting.


Why does English ELMo model give embeddings for non-English words? Here's the code from my notebook:
%tensorflow_version 1.x
import tensorflow as tf
import tensorflow_hub as hub

elmo = hub.Module("https://tfhub.dev/google/elmo/2", trainable=True)
tf.logging.set_verbosity(tf.logging.ERROR)

def elmo_vectors(x):
    embeddings = elmo(x, signature="default", as_dict=True)["elmo"]
    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        sess.run(tf.tables_initializer())
        return sess.run(embeddings)

Output for non-English language: (Hindi in this example)
words = ['गोकुल']
v = elmo_vectors(words)
print(v.shape) # (1,1,1024)
print(v[0][0])
# Output: [ 0.3731584   0.5700774  -0.48072845 ... -0.1241736   0.5961436 -0.6986947 ]

The documentation of the pre-trained ELMo on Tensorflow Hub shows that it was trained only on the English language.
That is, the dataset from 1 billion word benchmark is based on monolingual English data. (Source)
So, how/why am I getting embeddings for non-English vocabulary words from ELMo using the TF Hub model?

While ELMo was trained on English data, it does not know whether the data you give it as input is English or not.
The input of ELMo is received at character-level. It may happen that the 1B Word data had hindi characters intermixed, case in which your characters would be encoded as they are or, most probably, your characters are encoded as unknown characters (just like the unknown token <unk> for word-level NLP but for characters).
ELMo is just a bunch of mathematical operations, so it takes whatever it receives and computes its operations on it, first, taking the character embedding with the characters you pass to it, then with a char-CNN followed by two highway layers and finally a bidirectional LSTM.


Dynamic learning rates in XGBoost cross-validation XGBoost's xgb.train() method takes a learning_rates parameter, which can take a custom function to apply a dynamic learning rate, depending on the current training round.

I recently posted a paper explaining how I'm using it to both speed up training in the beginning, and making more precise towards the end.

However, there's a problem with this method: it tends to overfit on the eval set because there's no cross-validation.

I noticed that xgb.cv() method has no learning_rates parameter and therefore appears to not allow for a dynamic learning rate while doing CV.

So my questions:

Is there a possibility to use dynamic learning rates with xgb.cv()?
If not, is there a rationale behind this or is this just not implemented yet?

There isn't as far as I can tell by looking at the source code of the version I'm working with (April 2016). But it's definitely worth adding and it isn't impossible. xgboost.train loops over the iterations extracts corresponding learning rate, sets it on booster via eta parameter and calls booster update. xgboost.cv calls fold update (ie CVPack.update) which calls booster update. Just need to add a learning rate (aka eta) parameter to CVPack.update


Neural Network Model using Transfer Learning not learning I am a beginner in Deep Learning and working on Road Crack detection using transfer learning. I am working on binary classification with two classes , crack and no crack.

My distribution of two classes is as follows:

Cracks - 600 images

No cracks - 480 images

I have used data augmentation also :

 train_generator = train_datagen.flow(trainX, trainY, batch_size=16)

 val_generator = test_datagen.flow(testX, testY, batch_size= 16)

I am using VGG16 and I have frozen the lower 4 layers like this :

vgg = vgg16.VGG16(include_top=False, weights='imagenet',
                                 input_shape=input_shape)

output = vgg.layers[-1].output

output = keras.layers.Flatten()(output)

vgg_model = Model(vgg.input, output)

for layer in vgg_model.layers[:4]:

     layer.trainable = False

After that, I added two hidden layers :

model = Sequential()
model.add(vgg_model)
model.add(Dense(256, activation='relu', input_dim=input_shape))
model.add(Dense(256, activation='relu'))
model.add(Dense(2, activation='softmax'))
model.compile(loss='binary_crossentropy',
          optimizer=Adam(lr = 1e-6),
          metrics=['accuracy'])

But after 1-2 epochs nothing seems to change, neither validation accuracy nor loss. I tried using SGD optimizer also but that also didn't help. I added more layers also but didn't have any effect on accuracy and loss.The maximum validation accuracy achieved is 62%.

I tried testing an image from my dataset, for that also model gives wrong prediction. For every test image it predicts as crack, i.e label 1.

Could someone suggest how i can improve this?
Thanks!

Just take 2 images from your training data. One from class 'crack', another one from class 'not crack'. Now, check if your model can get a training accuracy of 100% which means it can overfit on the training dataset or not. If it is not able to do that, something is extremely wrong about the model.


When and how to use bagging? Can all types of ML methods benefit from bagging? Decision Tree Classification seems always be the go-to example of bagging, what about other classifiers or regressions?

When it's suitable to do bagging, how to pick the size and number of bags?

Update:

I am looking for something mathematically more rigorous, such as, for each model (single learner) we can break its total estimation error into:

$Error^2 = Bias^2 + Variance^2 + Irreducible^2$

If we can have a rough estimation of $Variance$ and the correlation between the predictions from all the single learners, then we can know about how low we can push variance to through ensemble.

Bagging main goal is to minimize variance of your model.  Basically, if you have a model that is on average pretty accurate but inconsistent (meaning, it does well for a given data set, poorly generalizations) then bagging may be a way to produce a more consistent estimators.  Decision trees are the common example of this because they are the canonical high variance machine learning algorithm.

As for your last question, the size of each new training set needs to be the same size as the original training set.  The way you achieve this is by random sampling of the original dataset with replacement (meaning the new dataset may have duplicates).  The number of new training sets is dependent on the problem.  Sometimes 100 is fine other times you need 1000 or so.  There isn't a way to just know how many sets you need.  It is a parameter that needs to be tuned.


Dataset to give same eigenvectors? Any suggestion on what kind of dataset lets say $n \times d$ ($n$ rows, $d$ columns) would give me same eigenvectors?.

I believe it should be the one with same absolute values in each cell. Like alternating +1 and -1. But it seems to work otherwise.

Any pointers?

I will restrict myself to the finite-dimensional situation.

First, if we are talking about normal eigenvalues and not generalized ones, we need a square matrix. If $M\in\mathbb{R}^{n\times n}$ (or $\mathbb{C}^{n\times n}$) is a diagonizable matrix, it has an eigendecomposition
\begin{equation}
M = V\Lambda V^{-1}
\textrm{,}
\end{equation}
with $V:=\begin{bmatrix}v_{1},v_{2},\ldots,v_{n}\end{bmatrix}$ the matrix with its $n$ eigenvectors $v_{i}$ as columns and $\Lambda :=\operatorname{diag}\left(\lambda_{1},\lambda_{2},\ldots,\lambda_{n}\right)$ the diagonal matrix with the corresponding eigenvalues on its main diagonal.

As you can see from this equation if you want to get matrices with the same eigenvectors, i.e. $V$ fixed, you can only change the eigenvalues $\lambda_{i}$. In general the relationship between the eigenvalues and the entries of the original matrix $M$ is non-trivial.

The easiest situation arises if the eigenvectors $v_{i}$ form the standard basis, i.e. only $1$ at the $i$th position, $0$ otherwise. Then $M=I\Lambda I^{-1}=\Lambda $ ($I$ the identity matrix) and you can change every entry on the main diagonal without changing any eigenvector.

Edit

I understood the question differently than victor. I thought the question asks what kind of different datasets can be described by the same eigenvectors.


Machine learning algorithm that can use many instances to predict 1 continuous outcome per person I am trying to use movements identified from accelerometers during sleep to predict gait speed (continuous). I am trying to figure out what the best machine learning algorithms/ feature extraction methods can be used to use as much of the information as possible in this prediction.

I am currently identifying every movement that occurs during sleep for each night measured and then extracting features from the accelerations of each movement. I then average all of these features over each movement identified during the night and then each night measured for that person, resulting in 1 set of features per person. This can be easily used in machine learning algorithms to predict gait speed as a continuous variable.

However, I think by averaging over all of the movements per night and then over all of the nights collected, I am losing a lot of information about the individual movements of each person. Is there a way to include all of the movements (~200-1000) from one person as instances that can be used to predict the 1 set of outcomes per person? If not, is there a better way to condense the number of movements in into 1 instance per person without losing a ton of information from taking an average? Maybe using clustering?

Thanks for any input!

I think you would benefit from keeping your data as a time series and use a sequence model. Commonly you would use an RNN, the most popular flavor would be a LSTM.

Here is an example using Keras that I think is relevant for you

Multivariate example from same blog (but forecasting instead of classification)

LSTM with regression


Is there a model-agnostic way to determine feature importance? Sklearn has a feature_importances_ attribute, but this is highly model-specific and I'm not sure how to interpret it as removing the most important feature does not necessarily decrease the models quality most.

Is there a model-agnostic way to tell which features are important for a prediction problem?

The only way I could see is the following:

Use an ensemble of different models
Either start with a big set of features and remove one at a time. To find the features "uplift", compare the ensembles quality with the full feature set against the ensembles quality against the removed feature set.

(What this can't do is to find connected features: Some features might be not exactly the same, but have a common underlying cause which is important for the prediction. Hence removing either of them doesn't change much, but removing both might change a lot. I ask another question for that.)

Ways to "determine feature importance" are normally called feature selection algorithms.

There are 3 types of feature selection algorithms:

Filter approaches: they choose variables without using a model at all, just looking at the feature values. One example is scikit-learn's variance threshold selector.
Wrapper approaches: they use whatever prediction algorithm to score different subsets of features and choose the best subset based on that. These use a model but are model agnostic, as they don't care about which model you use. One example is recursive feature elimination.
Embedded approaches: in these approaches, the variable selection is part of a model, hence the feature selection and the model are coupled together. This is the case of the feature_importances_ in random forest algorithm.

From the question, I understand that both filter and wrapper approaches are suitable for the OP needs. A classic article that covers both very well is this one by Kovavi and John.

Here you can see an overview of scikit-learn feature selection capabilities, which includes examples of the three aforementioned types of variable selection algorithms.


Estimating test AUC using k-fold CV for imbalanced classification problem I have an imbalanced classification problem.

I first partitioned my data into a training set (Dataset A) and a test set (Dataset B).

I then used the R package ROSE to simultaneously undersample the majority class in Dataset A and oversample the minority class. This produced a balanced set (Dataset C) which has as many rows as Dataset A.

I have trained my model on Dataset C and computed training AUC. I now want to estimate (i.e., not using Dataset B) the test AUC using k-fold cross-validation. How should I best do this?

The test AUC is the AUC you find after predicting the held out test set (dataset B).

You have split your dataet into A = training set and B = test set. You then used downsampling (which is questionable, but I won't get into this) on dataset A to get what you refer to as dataset C. You should then train your classifier on dataset C and evaluate its performance on dataset B.

If what you are referring to is a validation set (i.e. not the test set = dataset B) then what you need to do is as follows: split your dataset into A = outer train, and B = test. Then, split A again into C = inner train, D = validation. Downsample dataset C, train your classifier on C only, then validate on D. Once you have optimized to D, then downsample A, retrain your classifier on A (using the exact same model building process as you did with C and D), and report your final, unbiased performance measure on B. Repeat this entire process if AUC scores are volatile to get more estimates of final model performance.

Since you have unbalanced data, I strongly recommend stratified sampling when creating your validation and test sets.


Extract segment from document scan I need to extract some "valuable" information from document scan. For example, document's number, incoming date, organizations, persons, etc.

Example document:

I'm trying to extract highlighted segment of the document. Original scan doesn't have that highlighting. And value can be handwritten or typewritten.

I tried U-Net and Mask RCNN for my dataset (~100 examples). Without any success.

Any ideas?

Priviet, feeper!

I created some simple program to extract data from documents.
Works pretty well.

https://gist.github.com/fuwiak/780cb4abbe01aa5d1438269dfa0a3cfc

Best


Why can't Policy Gradient Algorithm be seen as an Actor-Critic Method? During the equation deducing in policy gradient algorithm(e.g., REINFORCE), we are actually using an expectancy of total reward, which we try to maximize.

$$\overline{R_\theta}=E_{\tau\sim\pi_\theta}[R(\tau)]$$

Can't it be seen as an Actor-Critic Method since we are using V(s) as a Critic to guide the update of Actor π? (Here we've already introduced an approximation)
$$\nabla \overline{R_\theta} = \sum_{n=1}^N R(\tau^{(n)}) \nabla \log p(\tau)$$
If not, what's the clear definition of Actor and Critic defined in Actor-Critic Algorithm.

In RL we have:

Actor-only methods such as REINFORCE in which the output is a probability distributions over actions. REINFORCE is a policy gradient method but doesnt use a critic.
Critic-only methods such as Q-learning in which the output is the expected reward for every available action ($Q(s,a)$ $ \forall a\in A $)
Actor-Critic methods that involve both Actor and Critic estimations. For example the popular DDPG and A3C algorithms. Both algorithms are policy gradient methods. By reading the papers you will start getting a sense on why the simple REINFORCE introduces variance in gradient estimations and how a critic can reduce it.

Policy Gradient methods are based on the Policy Gradient theorem. A standard implementation is an Actor-Critic algorithm, and we use both Actor (probability distribution over actions) and Critic (Value functions) in order to trade-off bias and variance in your gradient estimations.


Any idea to what $\bar{X}$ refers to in association rules? Is it "not-$X$"? Any idea to what $\bar{X}$ refers to in association rules? Is it "not-$X$"?

It's in the context of interestingness measure for rule $X \rightarrow Y$.

Do you have a link to the actual source you are referring to? I haven't myself seen it before in the context of association rules and interestingness.

As was pointed out by @Sean Owen in his comment, it is one of a few standard notations for saying something is not in a set - i.e. the complement. This would make most sense, as we are generally referring to itemsets when computing the associations.

Regarding the complement and set notation, the Wikipedia article contains this:

  The absolute complement of A is usually denoted by $${\displaystyle  A^{\complement }}$$

  Other notations include $$ {\displaystyle A^{\text{c}}}, \hspace{3mm} \overline A, \hspace{3mm} A',\hspace{3mm} \complement _{U}A \hspace{3mm} and \hspace{3mm} \complement A$$

Of course a bar can also mean things such as the mean or expectation value of some values (I have even seen people using it to mean the unobserved or predicted value!), but with the information you provide, the complement seems most likely.

Here also from the same Wiki page linked above for those who are unsure, a visual definition of the complement:


Hyperparameter tuning XGBoost I'm trying to tune hyperparameters with bayesian optimization. It is a regression problem with the objective function: objective = 'reg:squaredlogerror'
$\frac{1}{2}[log(pred+1)-log(true+1)]^2$
My dataset consists of 20k vectors, each vector has length 12 (twelve features). Every vector has a corresponding Y value.
I want to find the set of hyperparameters that minimize the loss function.
This is how it is implemented in code:
def evaluate_model(learning_rate, max_depth, nr_estimators, min_child_weight, min_split_loss, reg_lambda):
    model = get_model(learning_rate, max_depth, nr_estimators, min_child_weight, min_split_loss, reg_lambda)
    model.fit(X_train, Y_train)
    pred = model.predict(X_val)

    error = np.array([])
    for i in range(len(pred)):
        prediction = np.maximum(pred[i],1)
        error = np.append(error, (1/2)*(np.log(prediction+1)-np.log(Y_val[i]+1))**2)
    err = np.mean(error)
    return -err

My question is if anyone has any problem with how I've constructed the evaluate_model function. Do this optimize the squared log error when bayesian hyperoptimization is being implemented? The maximum(pred[i],1) is there in case a negative prediction is produced. Also, I get bad results even after the hyperparameter optimization.
These are the hyperparameters I evaluate:
pbounds = {'learning_rate': (0,1), 'max_depth': (3,10), 'nr_estimators': (100, 5000), 'min_child_weight': (1,9), 'min_split_loss': (0,10), 'reg_lambda': (1,10)}

The optimization is ran for 100 iterations and 10 init points. The package I've used for the bayesian optimization is bayes_opt

Instead of looping through the predictions why don't you import mean_squared_error from scikit learn's metrics module?
I'm guessing you are optimising using the mean squared log error objective.
Firstly, Log transform the Actual and predicted values and find the mean squared error
Example:
from sklearn.metrics import mean_squared_error
mean_sq_error = mean_squared_error(np.log1p(yval), np.log1p(pred))
np.log1p(x) does the same as np.log(x+1)


Can we look just at the other features when we have a missing vaue? For my classification task i have 3 features a,b,c.
Feature c is missing for some datas.
I can have already a good score for my classifier training with the two other features a,b and even better if I don't look at the missing data and I train with a,b,c.

Is it possible to train my classifier with features a,b,c on all the data which don't have missing values, and in my prediction phase, if the feature c is missing, to predict the class like if I had train my classifier with a and b?

Try a few options and pick the best.

Fill in the mean/median from known $c$'s in the $a,b,c$ model
Create a linear regression model to predict $c$ from $a,b$, fill the estimate $c$ from this model in the $a,b,c$ model
Train both models and use the applicable one

Personally, I think 2. will work best in most cases, but 1. can already be sufficient when $c$ is not correlated to $a$ or $b$.


XGBoost probability distribution tending towards the extreme I am using an XGBoost classifier to make risk predictions, and I see that even if it has very good binary classification results, the probability outputs are mainly under $0.05$ or over $0.95$ (like 60% of them).

I have tried calibration methods (from the sklearn API) but it reduces the problem only slightly.

My dataset has 1800 training points and I test it on around 500 datapoints. It is quite balanced. I also use Bayesian optimisation to tune the hyperparameters of the model. There are 19 features for my model.

Does anyone know a solution to get more regularly distributed probabilities? Does the problem dwell in the fact I have too few datapoints? Should I set my hyperparameters differently? Do I have too many/few features?

Since you're seeing probabilities concentrated near 0 and 1 (as is expected in gradient boosting), make sure you're not using Platt calibration; isotonic calibration is a better choice, or if you're willing to depart sklearn, beta calibration sounds promising.  Keep in mind that all these calibration methods should use a separate training set.  Also, have a look at calibration plots (which might show that Platt is unsuitable, and would also indicate whether the next paragraph is a possibility).

There's also a (slight) possibility that these scores are actually well-calibrated: e.g. if the data was generated in two clusters, and each cluster's points choose classes uniformly with probabilities close to 0 in one cluster and close to 1 in the other cluster, then "the right" answers are these probabilities, with the tree finding those clusters.

One approach to obtain "more regularly distributed probabilities" without actually calibrating them would be to make your model more conservative, using fewer trees and/or lower learning rate, more regularization, etc. but this risks hurting the model's performance.


Automatic Feature Engineering I came across a Machine Learning software which I won't name, that claims that completely automates the Feature Engineering process from any source without domain knowledge, implying pretty much that turns Data Scientists unnecessary for this step. Now I find that claim a bit suspicious since it goes against the common sense of what I learned Feature Engineering consists of.

But considering that it might be a lack of knowledge from my part to what point we can automate the featuring engineering process? I can think of some general approaches like statistical methods (median,etc..) or several types of coding (binary, polynomial,etc..) for categorical variables, but could anyone give me an overview state of the art approach regarding this subject or point me in the right direction?

In my experience, when people claim to have an automated approach to feature engineering, they really mean "feature generation", and what they're actually talking about is that they've built a deep neural network of some sort. To be fair, in a limited sense, this could be a true claim. Properly trained deep neural networks can handle any number of pairwise correlations between individual features or groups of features. That said, without a great deal of up-front data pre-processing tools that know how to intelligently handle different types of input data (e.g., free text, images, etc.), none of this would be possible. Bottom line, it takes a great deal of manual effort to do something automatically.


Feature selection for data with both continuous and categorical features? I am working on a classification problem with 4 ordinal classes to predict, labelling/predicting samples as either a number from 1-4. My training dataset has 284 features by ~40,000 samples and I am looking to explore feature correlation and variance and relate that to using a filtered feature selection method. I've been looking to learn from this guide: https://machinelearningmastery.com/feature-selection-with-real-and-categorical-data/

However, my dataset has predominantly continuous features, but it does have ~5 categorical features. How do I consider both types in my statistics and feature selection method? Is it common to partition the features and explore correlation separately (e.g. select out my categorical feature and use Chi-square, then select my continuous and use ANOVA?). Or should I be transforming either my continuous or categorical variables into the other and applying one statistical method to all features?

I aim to be systematic and explore different statistics and selection methods, but I am not sure how I should be accounting for the continuous and categorical variables together or apart. I am new to machine learning (with a biology background) so any help or resources to learn more would be appreciated.

I recommend using the Lasso for feature selection. I think the main advantage is that it is a multivariate method. However, it is more suited for linear models than for tree-based ones.

There are some broad theoretical and practical resources that explain why it is used. I wrote one article comparing it to univariate methods and another one comparing it to feature importance.

You can use the glmnet package in R or use the Lasso() class in scikit-learn.


Identify document's owner by content Can anyone suggest a machine learning algorithm that would be useful for identifying the organization a document belongs to? I'm working with a relatively large number of organizations (on the order of 1000s). Naturally, the input data is unstructured, but most documents look almost, but not quite, the same. Also, there are some organizations that follow more then a one template, again, in that "almost but not quite" manner.

Documents are in PDF format, and are reports of some kind and can contain several rectangular regions of text and tables. Most of them are not more than five pages long.

New organizations can eventually appear - how does interfere with algorithm?

Additional to the content of the document, the form could contain even more valuable information. I can't comment on the formatting of pdf files, but it should be possible to exploit the pdf metadata. Author, Title, subject or keywords could provide some clue pointing to a common origin in the same organization.


gold price detection using data mining I have dataset of gold prices and after modifying and some preprocessing i ended up with dataframe below:

There is 50,000 record in dataset and all columns expect date are int type and date is datetime object.
i need to predict price per unit in some specific dates. but somehow i baffled with so many methods.

My question is what data mining algorithm/method is results good prediction for this kind of data ?

Recurrent Neural Network (RNN) are quite efficient at dealing with sequential information such as time-series.
LSTM (Long Short Term Memory) is a type of RNN that can learn long-term dependencies and overcome some problems that vanilla RNNs have.
If you want to try it on your problem, I'd suggest this tutorial on LSTM for time-series prediction.


How do I compare columns in different data frames? I would like to compare one column of a df with other df's. The columns are names and last names. I'd like to check if a person in one data frame is in another one.

If you want to check equals values on a certain column let's say Name you can merge both Dataframes to a new one:

mergedStuff = pd.merge(df1, df2, on=['Name'], how='inner')
mergedStuff.head()

I think this is more efficient and faster then whereif you have a big data set


Negative Feedback to GradCam method I use EfficientNetB0 for performing image classification with one of the class as "stone countertop table", the other class is "not stone countertop table". I use the reddest part in gradCAM to put a point saying that this is the "stone countertop table". But in some cases, it is pointing to the basin (as most of the stone countertop accompany with a basin, and lead efficientNETB0 focus on the wrong thing). Is there a way to tell EfficientNetB0 not to use basin as a hint to look for "stone countertop table", and therefore the reddest part in gradCAM will always point to the surface of "stone countertop table"?
This is what is desired:

This is not:

This is not as well:

From these 2 pictures alone, I feel that the algorithm is focusing on the table color. Your second table is black but the basin is silver. Can you add more black table images?


Pre trained vehicle detection network I'm looking for a pre-trained net recognizing vehicles, something like the Inception network for images.

If that is impossible to find which vehicle detection algorithm would you suggest, possibly implemented in tensorflow?

You can take a look at the YoloNet which detects Objects based on Pascal VOC 2012 dataset.
Here is the link to it : https://github.com/pjreddie/darknet/wiki/YOLO:-Real-Time-Object-Detection

You should be able to detect vehicles with this pre-trained model.


Clustering time series based on monotonic similarity Context

I am involved in a task of clustering 1500 time series of 500 observations into a few number of clusters. The time series share all the same observed property at different spatial locations, but responding to the same exogenous variables. However, for each time series, the magnitude of the response is very different. For a time series of reference $X$, I would like to be grouped in the same cluster series that are alike $X^a$ for all $a > 0$.

Tryouts

So far, my interpretation of the problem is that I want to cluster time series sharing a strong monotonic relationship. My first tryouts used hierarchical agglomerative clustering by defining a distance based Kendall's tau rank coefficient since it measures the strength of a monotonic relationship. By visual interpretation, best results were obtained using Ward's linkage method. However, this approach seems unorthodox, non-robust, or doubtful for several reasons.

First, Scipy documentation mentions here that Ward's method is only correct when the Euclidean distance is used. Secondly, I couldn't find any detailed application of time series clustering based either on Spearman or Kendall's tau coefficient. Furthermore, I was very surprised that I couldn't find any paper or reference aiming at clustering based on a monotonic criterion.

I am willing to consider other approaches, though I cannot measure their benefits. For instance, rescaling all time-series to map them to a standardized gaussian distribution (e.g. Box-Cox) and then using the Euclidean distance. Another possibility is to turn the first difference of time series into a boolean vector (1 if $\Delta X >0$, $0$ otherwise) and then use the Euclidean distance or another distance metric.

Questions

Since I am new to time series clustering, I have some troubles to picture by myself what would be the best approach(es) (or the worse) for this specific purpose. Hence, I have two related questions:

Specifically, is using Hierarchical Clustering based on Kendall's tau and Ward's linkage method a wrong way to go and why?
Generally, what is the best way to cluster time series based on monotonic association?

Some references on the topic are also welcome.

The way Ward's linkage is computed really only makes sense with squared Euclidean type of measures. Only then the Konig-Huygens theorem can be used.

Why don't you consider average linkage? Why Ward?


Predicting future order dates and order amounts for customers with online retail data Using the online retail II dataset (https://archive.ics.uci.edu/ml/datasets/Online+Retail+II), I'm trying to predict when each customer will place subsequent orders, and if possible, the monetary value of those orders.  This will be a proof of concept for a real online retail store, in which I'll have access to even more data/features.

This current dataset contains the following attributes:

InvoiceNo: Invoice number. Nominal. A 6-digit integral number
uniquely assigned to each transaction. If this code starts with the
letter 'c', it indicates a cancellation.
StockCode: Product (item) code. Nominal. A 5-digit integral number
uniquely assigned to each distinct product.
Description: Product (item) name. Nominal.
Quantity: The quantities of each product (item) per transaction.
Numeric.
InvoiceDate: Invoice date and time. Numeric. The day and time when a
transaction was generated.
UnitPrice: Unit price. Numeric. Product price per unit in sterling.
CustomerID: Customer number. Nominal. A 5-digit integral number
uniquely assigned to each customer.
Country: Country name. Nominal. The name of the country where a
customer resides.

My current intuition is to model the data as a daily time series that would output which customers are likely to order for the day and the value of those orders (if possible?).  The output would either be a sparse, one-hot encoded matrix of shape num_customers X num_customers, or a more dense vector of size num_customers (which could have thousands of customers).  I'm not quite sure how, or even if the model would be able to also output the amount of the order.  This model will likely be fed into an LSTM network or similar. I suppose this would be considered a multivariate time series forecast (still learning here)?

From this, I'm hoping to be able to run the prediction for each day in the following year to forecast the next order date, customer value and churn risk

Since this would be an attempt to model human behavior as a time series, without a direct correlation from one day to the next (in contrast to predicting daily temperature or a stock price), and with so many variables, I'm also worried it would just come off as unpredictable noise.

Anyway, am I on the right track? (any insights would be greatly appreciated)
Should this be broken into multiple models, and if so, how many?
Does anyone have any tips, or know good articles or tutorials that could point me in the right direction?
What are the merits of the one-hot encoded matrix, vs the multi-hot
encoded vector (not sure if that's the proper name)?

Thanks!

To answer your points in order:

I can generally understand what you are trying to do. However, a few considerations you should bear in mind:

1.1. Given that you are trying to make time series predictions for all customers, then you are most likely interested in a panel data modelling solution - i.e. one that takes into account the fact that the data is both cross-sectional (at one point in time) and time series (across several time periods). The alternative is simply running a separate time series for each customer, which could prove to be highly inefficient.

1.2. This would be considered a multivariate time series forecast - provided that price is in fact dependent on variables other than itself, i.e. quantity, country, etc. Are you planning on incorporating these as explanatory variables into a time series model?
Hard to say how many models should be used off the bat, but I might be inclined to construct a single panel data model for this case - you could always construct it on a subset of the data as a proof of concept and then work from there. Construction of multiple LSTM models would appear too much effort for too little reward (in terms of forecast accuracy).
I would be inclined to take a look at linearmodels - it is a Python library designed for working with panel data.
These techniques are used for representing categorical data - you might find the following answer useful.

Hope this helps, and please let me know if I can offer any further clarity on the above.


Which machine learning problem is this? I am not able to figure out what kind of machine learning is this:
Training set: consists of sentences with object labels for object phrases
Example:
"This is a black chair. It is next to a large bed."

Phrases 'This', 'black chair', 'it' were annotated with the label "chair" and phrase 'large bed' was annotated with the label 'bed'. There are 18 available labels that can be assigned.
For an unannotated sentence, I want to predict the labels for each phrase in the sentence.
For example:
There is a study table in the corner of the room, behind it is a small chair.

I would like the model to predict a label (available 18 labels) for each phrase that represents an object in the above sentence.
Expected output:
'study table', 'it' -----> label 'table'

'small chair' -----------> label 'chair'

Is this a classification, regression, or a different kind of problem?

Technically this is sequence labeling, the most common application being Named Entity Recognition.
However it looks like in this case you're trying to solve a problem of coreference resolution, which is a quite difficult task in general. I think this usually involves a more complex model than simple sequence labeling, but I'm not an expert in this. You might want to search around this topic, there are certainly some relevant papers and tools about it.


Time Series Continuous Classification Intro

I'm quite new to all the subjects in this question. This is my very first shot at Keras, Tensorflow, NNs or Time Series. If you're not a newbie, you'll notice that immediately.

Problem

I have 15 samples from an industrial process that reads temperatures, weights and heatflows every 2 seconds for roughly an hour (1876 rows). Samples look like this:

      temp   weight     heat
t
0  31.9661  9.09606 -0.834733
2  31.9905  9.09503 -0.839906
4  32.0967  9.09374 -0.854042
6  32.3021  9.09315 -0.877126
8  32.5371  9.09390 -0.917037

Each of the 15 samples has been analysed by an expert that has produced QA, rating 5 of them as lousy (0), another 5 as medium (5) and the third lot as perfect (10). My goal is to analyse further samples and produce an equivalent assessment.

Statistical analysis on the data revealed that the only feature showing differences is heatflow, so I'm focusing on that column. The column name is calor (heat in Spanish). I also detected very synchronous activity around the tails but big discrepancies over the middle section, so I'm also focusing on that section (rows 1250 to 3250).

Solution

I'm reading the CSVs into a dictionary and then passing it to a Keras network with the following code:

sequences = [data[k][['calor']].loc[1250:3250].values for k in data.keys()]
final_seq = np.array(sequences)

targets = [5, 5, 5, 5, 5, 0, 0, 0, 0, 0, 10, 10, 10, 10, 10]
groups = [1, 1, 1, 2, 3, 1, 1, 1, 2, 3, 1, 1, 1, 2, 3]

train = np.array([final_seq[i] for i in range(len(groups)) if groups[i] == 1])
test = np.array([final_seq[i] for i in range(len(groups)) if groups[i] == 2])
validation = np.array([final_seq[i] for i in range(len(groups)) if groups[i] == 3])

train_target = np.array([targets[i] for i in range(len(groups)) if groups[i] == 1])
test_target = np.array([targets[i] for i in range(len(groups)) if groups[i] == 2])
validation_target = np.array([targets[i] for i in range(len(groups)) if groups[i] == 3])

# Building the model
model = Sequential()
model.add(LSTM(11, input_shape=train[0].shape))
model.add(Dense(11, input_shape=(11,), activation='softmax'))

adam = Adam(lr=0.001)

chk = ModelCheckpoint('best_model.pkl', monitor='val_accuracy', save_best_only=True, mode='max', verbose=1)

model.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy'])

train_binary = to_categorical(train_target)
validation_binary = to_categorical(validation_target)
model.fit(train, train_binary, batch_size=len(train), epochs=100, callbacks=[chk], validation_data=(validation, validation_binary))

model = load_model('best_model.pkl')
test_preds = model.predict_classes(test)
print(f'test_target: {test_target}\ntest_preds: {test_preds}')
acc = accuracy_score(test_target, test_preds)
print(f'Accuracy score: {acc}')

I get different results, but usually around the following:

Model testing...
test_target: [ 5  0 10]
test_preds: [10  0 10]
Accuracy score: 0.6666666666666666

Accuracy of 66% is the best I ever got so far.

Questions

Am I doing this right?

I'm not sure about some hyperparameters. I kind of arbitrated 100 epochs, the batch size as the total sample, the categorical_crossentropy loss function and the groupings as per the above groups list because of the targets (which are fixed).

I'm also not sure about the LSTM thing. My categories are not only numeric but ordinal. I could actually have a regression result here.

Should I even be using NN for this? Basically, any input is welcome.

I'm going to need much more data, correct?

I have 5 samples each. How many more should I have?

Thank you very much, experts!

With only 5 samples per class you probably should not be using any Neural Network. If you can find a way to summarize your long time-series into a couple of numerical features, then you can try with a simple Logistic Regression model. Exploring the data is the key here, as well as understanding the problem domain. What is the physics of the process for example, or how would experts describe the requirements to get a particular score.

Even so I would try to collect some 10x more data.


Pricing decisions using neural network I have a big list of spare parts with several parameters (material, weight, size, manufacturing complexity, ...). For some parts in this list, a price has either not been set or has to be adjusted in order to be in line with other parts. There are a few obvious and simple correlations in this dataset, for example:

if material and complexity is the same, bigger parts are more expensive;
if size and material is the same, more complex parts are more expensive;
for equal size and complexity, more expensive material leads to higher price.

Trying to figure out all these rules by hand and sticking them together seems to be an endless endeavour, so I thought about training a neural network with the priced parts' parameters (input) and prices (output) and let it figure out the prices for the parts which don't have a price yet. The decisions of the NN could be supervised by an expert who knows the parts and could manually figure out a price.

Question 1) Is this a good idea in general?
Question 2) If yes, what type of NN would be most suited for such a problem?

Question 1) Is this a good idea in general?

Solving this problem as a supervised learning regression problem is a fantastic idea and is the type of solution that will greatly benefit your company since it will translate to other similar and dissimilar problems much more easily than deterministic methods.

However using a neural network to solve this supervised learning regression problem is probably a very bad idea. Neural networks can add great value to certain problems, are among the very best algorithms for complex problems where computing power is not an issue, and have fascinating implications for the future of machine learning and artificial intelligence. But... they can be very tricky to train, require a lot of computing power, require a lot of data, and perform poorly in many cases.

I suggest you scope the problem using linear regression and then try a support vector regressor (SVM, SVR) or naive Bayes regressor. The analogue methods in SVR work very well with limited data and provide surprisingly accurate results.

Question 2) If yes, what type of NN would be most suited for such a problem?

If you must use a neural network then try playing with the problem.  Start with a feed forward neural network. Note that it will likely underperform an SVR.  Then think about moving toward a convolution neural network. Again, this is probably a very bad way to go. Try linear regression followed by other methods first.

Moving forward

The documentation for either Scikit-Learn in python, H20 in Java, or Weka provide very shallow learning curves to jump on the merry-go-round and take a spin or two. Please make sure that you thoroughly understand cross validation and scoring metrics as this is essential to continued, adequate progress.

Hope this helps!


Why keep vocabulary and posting list separate in a search engine I am taking a class in information retrieval. We learned that the index of a search engine has (possibly among other things):

A vocabulary mapping terms to their statistics (frequency, type, ...) and
A posting list mapping terms to the documents were they are stored (with or without positions, fields, ...)

These are separate data structures. I understand why those information is needed and what for. But I don't understand why we want to keep them separate. Why can't we have one data structure that maps terms to statistics and documents?

I am currently thinking it might be because the vocabulary would be much smaller and we could read it from memory. So we could use the statistics to remove certain query terms, which are likely not useful or to try to find misspellings in the query without having to touch the large posting list.

Is this correct or is there another reason to keep vocabulary and posting list separate?

It has many reasons to be (performance, design, storage, compression, evaluation of data structures). The principal reason is that all structures are verified in the practice, but you can make you own data structure and show a new mode to do.

Even the google has a paper for verify that he works fine, if you has your own data structure and design, I suggest you choose a database with the correct size for your experiment, search for the precision and recall values for your own collection and make it.

Other reasons that you can see is that the information has different requirements about performance, compression, storage and hardware.

The cost of gigabyte for main memory is very different between RAM and HD, when you have a lot of servers your strategy of cost reduction is a reason of improve low cost storage and no high hardware requirements.

It's clear when you have a collection with Terabytes of data, many cultures or many countries. (google has a paper about his cluster).

Think simple, a collection with 50GB is a small collection, but how much cost a server with 60GB of RAM and how much cost a server with 50GB of HD?

The answer is clear and the it's reflects in the data structures (B-tree is good for secondary memory and simple hastables are good and fast in main memory).

But you decide.

A old good reference is the chapter 5 of Managing Gigabytes (https://www.amazon.com/Managing-Gigabytes-Compressing-Multimedia-Information/dp/1558605703/ref=sr_1_1?s=books&ie=UTF8&qid=1474990622&sr=1-1&keywords=managing+gigabytes)

The chapter shows a table with different results for many data structures and types of memory.

The google's paper shows the mode for verify your implementation and design:
http://infolab.stanford.edu/~backrub/google.html

Think about the cluster and jobs:
http://static.googleusercontent.com/media/research.google.com/pt-BR//pubs/archive/43438.pdf


What is the best to identify the proper hierarchy of this data? So I worked on a hierarchical clustering algorithm to be able to determine which items are most similar, and what attributes are most important. I have two tables:

Table 1: contains a bunch of item codes, and it's attribute (brand, flavor, sales, and so on). It looks something like:

Item_code | Brand | Flavor   | Caloric_content | ... | sales
006891313 | Coke  | Original | 0               | ....
002349823 | Fanta | Orange   | 200             | ...

The other table that I have is what i run my clustering algorithm on. It's an NxN matrix, (where N is the number of distinct item_codes in the previous table) An entry [i,j] in the matrix, corresponds to the number of times j was bought after i was purchased on a previous trip. So more clearly, in the matrix below, what the number 1223 means is that, 1223 times after item 003428734  was purchase on an initial trip to the store, item 003428734 was purchased on the next trip.

          |003428734 |009849328 | 09840202 |....
003428734 |  1223    |    13    |    0     |
009849328 |    12    |   945    |    34    | ....
   .
   .
   .

I apply a hierchical clustering on that matrix, using Ward 2, squared euclidean distance and standard Z scores. The final output is a dendrogram, with all the item_codes on the branches of the dendrogram.

This is where the tedious process is. The ultimate goal, is to have a hierarchy for our products, and see which attribute (brand, flavor, size...) falls where on the dendrogram. The only way i can think of doing it, is organizing the item codes in the first in the same order that they are in the dendrogram, put a picture of the dendrogram side by side, and eyeball which attribute clusters the most. You'll see in the link below, it looks like brand A and B are clustering together, so I would say that products cluster by brand first. I would then take a closer look at the smaller cluster within the dendrogram and try to identify where the other attributes fit in the hierarchy (the picture is just a snippet of the dendrogram, it is in reality much, much bigger)

This eye balling process is very tedious and annoying. Is there a way to run a similar cluster, which shows a hierarchy within the products, as well as indicate which attribute fit where in that hierarchy? Or is this even the best way to approach this?

I should mention that I'm very new to these clustering algorithm, so apologies if this is a dumb question

I don't think your clustering makes sense in the first place. That you actually get some output is more of a coincidence than a result.
Just invoking some functions in a way they don't fail is not enough to get a trustworthy result...

Your matrix is vaguely resembling a similarity matrix, except that it is not symmetric, and the diagonald are not necessarily the maximum values.

Now Ward usually needs a squared Euclidean distance matrix, something very different. The reason why your program doesn't fail supposedly is because it now computes such a distance matrix based on the rows from your similarity matrix. That will usually give somewhat okay looking results (as similar items should have similar rows) but causes subtly problematic bias based on the number of product types.

Later on you can evaluate each hierarchy level based on the agreement with some known label (e.g., manufacturer or product class) and you then can argue which agreement comes earlier. But on a result that isn't valid this doesn't make much sense.


Recommendations for storing time series data As part of my thesis I've done some experiments that have resulted in a reasonable amount of time-series data (motion-capture + eye movements). I have a way of storing and organizing all of this data, but it's made me wonder whether there are best practices out there for this sort of task.

I'll describe what I've got, and maybe that will help provide some recommendations. So, I have an experiment that requires subjects to use their vision and move their body to complete a task. Each task is one trial, and each subject performs multiple trials to complete the experiment. During a trial I record the movement and the eye tracker (~200 channels) at regularly sampled time points (~100Hz). I store these in a CSV file (one file per trial), with one row per time point, and one column per variable (e.g., left-fingertip-x, left-fingertip-y, left-fingertip-z, etc. for the mocap, and left-eye-x, left-eye-y for the eyes).

Associated with each trial is some metadata such as the experimental condition of the trial (e.g., how fast a target in the trial is moving, say). I store these values in the CSV filename itself, using a "key=value" sort of syntax.

While this works well enough for my purposes, it's really ad-hoc! I'd like to get a sense of whether other people have solved problems like this, and, if so, how?

There are two solutions that are worth looking at:

InfluxDB is an open source database platform specifically designed for time series data. The platform includes many optimized functions related to time and you can collect data on any interval and compute rollups/aggregations when reporting. The company recently launched a query app called Chronograf. I have not used this - but if its no good, you can also check out Grafana (which is very widely used and stable).

The alternative strategy you may want to pursue is an elasticsearch index. Elasticsearch is great at running aggregations and other mathematical functions on data. Many use it to store server log data and then query said data using Kibana.


Supervised learning model for extracting terror attack motives from attack summary (NLP) I have a terror attack (tabular) data set. Each row is one attack and there are columns like:

Date of the attack (daily resolution)
Location of the attack (long/lat, as well as city/country)
Number of casualties
Attack/weapon type
A few boolean columns like whether it was a suicide attack

Furthermore, I have a text column that holds a 2-3 sentence description of the attack. This is the main column I want to use for training/predicting.

There are several target columns of the form "is_left_wing", "is_right_wing", etc. The values are 0, 1, and -1. Here 0 means the attack didn't have the respective motive, 1 means it had the motive and -1 means it is unknown.

In short, my goal is to build a model that is trained on the 0 and 1 values in the target columns and makes predictions about the -1s.

The main thing I'm stuck on is how to extract features from the text column with the attack description. I have limited NLP experience and I want to use something more sophisticated than a simple bag of words model.

I would appreciate suggestions about the general approach to this problem (also some good readings on the topic).

Use word embedding and encode the entire sentence into one fixed feature vector by using vanilla RNN or more sophisticated model like attentional LSTM.
see Sentiment analysis using RNNs(LSTM)
Concat the other features with the fixed length representation of the sentence and append on top of them either dense layer.
the architecture
 feedforward(concat(other_features,RNN(sentence)))


On coursera what exactly does Andrew Ng say in videos Lectures 60 & 61 of machine learning? Model Selection and Train/Validation/Test Sets - Stanford University | Coursera:

At 10:59~11:10

  One final note: I should say that in the machine learning as of this practice today, there aren't many people that will do that early thing that I talked about, and said that, you know...

Is my comprehension correct?
Because English subtitles on coursera sometimes are not correct.
As I konw, here what Chinese subtitle means is opposite to what English one does.
So I am not sure whether Andrew Ng said "there aren't" or "there are"

Thanks for your reading.

I would like to ask another one.

Diagnosing Bias vs. Variance - Stanford University | Coursera:

At 02:34~02:36, what Andrew Ng said is not quite clear as well as the English subtitle.

My comprehension is as following

  If d equals 1,.... to be high training error.

It's not that complete.

Would anyone like to identify that?

Thank you...

No, he actually says the opposite:

  One final note: I should say that in the machine learning as of this practice today, there are many people that will do that early thing that I talked about, and said that, you know...

Then he says (the "early thing" he talked about):

  selecting your model as a test set and then using the same test set to report the error
  ...
  unfortunately many people do that

In this lesson he explains about separating the data set:

training set to train the model;
cross validation set to find the right parameters;
test set to find the final generalization error (of the function with the best parameter values found during using the cross validation set).

So Andrew Ng is complaining that many people us the same data set to find the right parameters, and then report the error of that data set as final generalization error.


the error occurred while selecting feature using recursive feature elimination in sklearn I tried to rank the feature using recursive feature elimination in sklearn. However, I got this error when using RFE. here are the error and code information.
  from sklearn import svm
  x_vals = data['all_data'][:,0:320]

 y_vals_new = np.array([0 if each=='Neg'  else 1 if each =='Neu' else 2 for each in data['all_data'][:,320]])

 clf = svm.SVC(decision_function_shape='ovo',kernel='rbf')

rfe = RFE(clf, 200)
rfe = rfe.fit(x_vals,y_vals_new)

print(rfe.support_)
print(rfe.ranking_)

clf.fit(DEAP_x_train, DEAP_y_train)

print("###Valence###")
print("when the kernel function is rbf")
print('The mean square error %10.9f ' % np.mean((clf.predict(DEAP_x_test)-DEAP_y_test)**2))  # The mean square error

print('the mean accuracy on the given test data and labels %10.9f'% clf.score(DEAP_x_test, DEAP_y_test))

Did anyone know the reason of this error？ some opinion showed that RFE（recursive feature elimination) only works with SVC when the kernel is chosen to be linear. Is that correct？ Thanks a lot！

Recursive feature elimination works by considering co-efficient value of feature column (in case of linear models) or variable importance exposed, by say, random forest classifier.

For example, in case of a linear model, it will eliminate feature with smallest (absolute) value of co-efficient (i.e. less impact on outcome).

In case of SVMs with non-linear kernels, the classification is happening in a transformed space of features and not original features. So it is not possible to assign importance measure to original features. So RFE doesn't work with RBF kernel here.

Related question on variable weight for non-linear kernel


Chose among highly correlated variables I am working on a Kaggle dataset and I am trying to build a predictive model for the "Chance of Admit" (dependent variable) of students to the university of their interest.

Below you can find the correlation among all (independent and dependent) variables. We can quickly observe that only "GRE Score", "TOEFL Score" and "CGPA" considerably affect the "Chance of Admit" variable. So, it makes sense to eliminate all other variables from the predictive model.

Now among the "GRE Score", "TOEFL Score" and "CGPA" variables, we can see that they are all highly corellated (this also makes sense in real life as you always expect a good student to get good grades in these tests). I cannot decide which variables to keep for my final model. Could I keep all of them ? or how do I decide which one to exclude ?

Any help is appreciated.

I agree with @peter's detailed answer on points #1 through #5 and would like to supplement that with some more details:

Perform a PCA or MFA of the correlated variables and check how many predictors from this step explain all the correlation. For example, highly correlated variables might cause the first component of PCA to explain 95% of the variances in the data. Then, you can simply use this first component in the model.
Random forests can also be used for feature selection by looking at the feature importances of the variable. However, correlated variables can cause misleading feature importances. You can use permutation feature importance (https://scikit-learn.org/stable/modules/permutation_importance.html)
You might be overfitting when using all the features. Therefore, it is essential to use a validation set to check for overfitting.
You can also drop useless features by adding a feature with random values and dropping any feature that has lower feature importance than that feature.

For this topic, I highly recommend the book: Feature Engineering and Selection: A Practical Approach for Predictive Models; Open-sourced here: http://www.feat.engineering/ And also available on Amazon.


Comparing 2 columns from separate dataframes and copy some row values from one df to another if column value matches in pandas Im relatively new to pandas andpython . I have a simple problem but I can not find the solution . I have 2 dataframes DF1 and DF2 as shown in image
Both dataframes have ID coluumns. I wannt to check if the ID for a row in DF1 matches any of ID in DF2, then I want to copy the diameter value from DF2 and paste it to DF1 in a new column. Eventually what I want to achieve at the end is DF3 shown in image below.
Help will be highly appreciated.

You may merge two dfs:
df1.merge(df2, on=['id'], how='left')


What makes a graph algorithm a good candidate for concurrency? GraphX is the Apache Spark library for handling graph data. I was able to find a list of 'graph-parallel' algorithms on these slides (see slide 23). However, I am curious what characteristics of these algorithms make them parallelizable.

Two words: associative and commutative

In other words, the operations that the algorithm does need to be independent of how you order or group your data...this minimizes the need for cross-talk in the algorithm and leads to more efficiency.


From where does BERT get the tokens it predicts? When BERT is used for masked language modeling, it masks a token and then tries to predict it.
What are the candidate tokens BERT can choose from? Does it just predict an integer (like a regression problem) and then use that token? Or does it do a softmax over all possible word tokens? For the latter, isn't there just an enormous amount of possible tokens? I have a hard time imaging BERT treats it like a classification problem where # classes = # all possible word tokens.
From where does BERT get the token it predicts?

There is a token vocabulary, that is, the set of all possible tokens that can be handled by BERT. You can find the vocabulary used by one of the variants of BERT (BERT-base-uncased) here.
You can see that it contains one token per line, with a total of 30522 tokens. The softmax is computed over them.
The token granularity in the BERT vocabulary is subwords. This means that each token does not represent a complete word, but just a piece of word. Before feeding text as input to BERT, it is needed to segment it into subwords according to the subword vocabulary mentioned before. Having a subword vocabulary instead of a word-level vocabulary is what makes it possible for BERT (and any other text generation subword model) to only need a "small" vocabulary to be able to represent any string (within the character set seen in the training data).


AttributeError: 'numpy.ndarray' object has no attribute 'nan_to_num' I'm trying to run a Random Forest model from sklearn but I keep getting an error: ValueError: Input contains NaN, infinity or a value too large for dtype('float32').
I tried following steps in ValueError: Input contains NaN, infinity or a value too large for dtype('float32')
fillna(0) on my pandas dataframe still gave the ValueError.
So I tried working with my numpy array:
val = setTo.ravel().nan_to_num(0)

But I keep getting an error: 'numpy.ndarray' object has no attribute 'nan_to_num'
I'm wondering how I can deal with the nan values if I have ndarray?
Thanks so much!
Update
Thanks so much to @Beniamin H for all the help, as per suggested, I rescalled the data, which I based on https://stackoverflow.com/questions/34771118/sklearn-random-forest-error-on-input and it worked!

You are using the right method but in a wrong way :)
nan_to_num is a method of numpy module, not numpy.ndarray. So instead of calling nan_to_num on you data, call it on numpy module giving your data as a paramter:
import numpy as np
data = np.array([1,2,3,np.nan,np.nan,5])
data_without_nan = np.nan_to_num(data)

prints:
array([1., 2., 3., 0., 0., 5.])

In your example:
import numpy as np
val = np.nan_to_num(setTo.ravel())


Python + Neural Nets + Large dimension Large Dataset I am building a NLP application. My dataset has 0.6M datapoints each of 0.15M dimensions. My feature vector is highly sparse - mostly 0s, atmost 20 1s in any feature vector. I am using a 32 GB machine on AWS.

Owing to size, Cant load entire dataset in one go in the memory. I am looking for a python based neural net library that supports (if not all then atleast some) of the following [Owing to the dataset size and dimension, i defined below mentioned abilities]:

Ability to train the net on small batches
library being able to exploit underlying multicore architecture (if available)
A way to directly feed the feature vectors in compressed representation (like scipy.sparse) [in my case

I have had a good look at the following libraries - PyBrain, Pylearn2, NeuroLab, FANN. To the best of my understanding, none of them has any of the desired abilities (Pybrain facilitates batch leaning. Even this takes close to 55 hrs).

Any suggestions ? (cant reduce dimentionality)  Or in case there is a shortcoming in my understanding and the any of the above libraries have (some) above capabilities, can you pls point out me to some sample code implementing/exploiting the same ?

I experimented with PyBrain and PyLearn2
I rather find Keras [http://keras.io/ ] as an excellent choice. It way ahead of PyBrain for sure. PyBrain development has been stopped


Compare job ads with a given set of categories (which each consists of terms) For a recent research paper, I plan to perform the following, for which I'd kindly ask for your advice.

I obtained a set of a few thousand job ads. I now want to analyse how and whether these job ads include 'content' that has been previously specified in another research paper as individual 'categories'. To make things more precise, there are about 15 existing categories, each of which contains descriptions that explain the category in 2-4 sentences.

Now, I want to understand, which and how many job ads cover the aspects described in each of the 15 categories. A result could be, for example, job ad #1 contains content that matches (or comes close) to the descriptions of categories 2, 5, 8 but misses content that would allow any reference to the remaining categories.

In case you got any references or advice how to approach this task, please let me know. I would suspect that the best approach would be a supervised learning approach.

Best,

Spooz.

In terms of data understanding, I would recommend an unsupervised approach first. For example you could ...

.. perform tf/idf vectoring and build a simple tag cloud
.. perform Latent Dirichlet Allocation to get an overview of latent topics in your data (like clustering and interpret the categories)
.. train a word embedding (e.g. word2vec) on top of your data. Then perform a dimension reduction (PCA) to visually explore your data. (You could find semantically and syntactically clusters)

After the unsupervides data exploration you could perform a text classification based on classical approaches (e.g. bag of word models) or with neural nets (=supervised).


Actual number of iterations I am currently using TF 2.1. When using model.fit with tf.data.Dataset, it shows me that the number of iterations is 2044/2043. So my question is, how many batches of data are actually produced? 2044 or 2043?

Number of batches = total number of samples in dataset/batch size.
If the total samples are not completely divisible by batch size, then it will take the remaining in the next batch.
Eg: 15x66+10=1000 which means it will take 66 batches of size 15 and for the final steps it takes only 10.


Sentiment Analysis algorithm in Weka I have an assignment, i have to collect movie reviews and classify them with poz neg notr tags, and turn them info arff files. I did all of this stuff with regex etc.

But for experiment i have to choose ten classify algorithm. I did some research but couldn't figure out which algortihm should i use.

Can you please help?
Thanks.

There is a good tutorial here:

http://www.unal.edu.co/diracad/einternacional/Weka.pdf

Basically you want to use a StringToWordVector filter to convert your string into word vectors (!) and then train a Bayes classifier on that data.


Is it possible to create a rule-based algorithm to compute the relevance score of question-answer pair? In information retrieval or question answering system, we use TD-IDF or BM25 to compute the similarity score of question-question pair as the baseline or coarse ranking for deep learning.

In community question answering, we already have the question-answer pairs to collect some statistics info. Without deep learning, could we invent an algorithm like BM25 to compute the relevance score of question-answer pair?

could we invent an algorithm like BM25 to compute the relevance score of question-answer pair?

It depends:

BM25 (actually cosine with BM25 weighted vectors) is a simple similarity measure, ultimately based on counting words in common. Proposing a different similarity measure is easy, for instance there are various measures used for MT evaluation (including some quite sophisticated ones) which could be used as well. Of course, these measures don't actually measure the relevance, they just offer a crude approximation.
However if there was such a rule-based algorithm which would be able to actually measure the relevance of an answer in any context, then for all means and purposes we would have solved AI: judging the semantic relevance is much more subtle than counting words in common. In particular if there is such an algorithm, then the problem of question answering is solved: you can just generate all the possible answers and loop until one is found relevant to the question.

People have tried to do "intelligent" rule-based algorithms in NLP for decades, before realizing that ML is more efficient and performs much better in most tasks. So it's extremely unlikely that a rule-based algorithm would suddenly outperform ML on a non-trivial task like this.


Chinese word segmentation using neural networks Chinese text uses a character set containing tens of thousands of characters. Words in Chinese are most commonly made up of 1, 2 or 3 characters. There are no spaces or other markers between words in Chinese text since native speakers can easily segment words from text. In order to assist language learning tools, it's helpful to have an automatic way to segment words within blocks of text.
Some approaches use dictionary based greedy algorithm methods but are prone to failing due to the common problems that most greedy algorithms present.
I want to try using neural networks, but my question is, How would one go about encoding the characters to the input neurons of the network?
I am not talking about OCR. The characters are already known and encoded in unicode, but how would I present the characters to the inputs of the network?
One method I can imagine is to have the network look at the text in portions of say 100 characters at a time, and have one neuron for each character. But how would I represent the character as a number to the network? Using the unicode integer number value for the character doesn't seem like a good idea.

So, the main question is about how to represent the Chinese characters in your Chinese word segmentation task.
Since effectively these characters are non-ordinal categorical variables, we would represent these as one hot encodings (https://medium.com/@michaeldelsole/what-is-one-hot-encoding-and-how-to-do-it-f0ae272f1179) of dimensionality n = number of unique Chinese characters in your dataset.
Sometimes n can be very large. So in this instance, we typically have an Embedding layer, which can reduce data sparsity and collapse the number of dimensions. (https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding#:~:text=Arguments%20%20%20%20input_dim%20%20%20,the%20embed%20...%20%202%20more%20rows%20)


Is it possible for a model with a large amount of data to perform very well and reach an extremely low cost within a single epoch? I am working on a project to detect human awareness levels using this dataset.
I preprocessed the video data as the following:

Convert video to frames(taking a frame every 5 seconds.
Rotate the frames to be vertical.
Apply OpenCV DNN to extract the faces from the images.
Split the data into 90% train, 5% validation and 5% test.

All in the dataset has a size of about 570,000 images.
I am using the model on a mobile device so I used transfer learning with MobileNetV2.
The model classification is extremely good but it feels odd seeing it do so well and reach a very low loss so fast.
Is this even possible on a dataset this big? I am feeling that I did something wrong cause when I try to use the model on the mobile device with Tensorflow.js it does not perform well at all. After doing some research I realized that I should be using a model that combines a CNN and a LSTM as this is video data. But I am bit strapped for time to redo the whole preprocessing of the data to convert the images into a sequence of frames and then do the training once more.
What I was planning to do was make an average of the predictions on the mobile device to improve the accuracy there but I am wondering if I messed up the process anywhere.

A few things come to mind here:

If you are using a pre-trained MobileNetV2 on a task which is similar to the pre-training, then you may not need much fine-tuning to get good results. This may explain why you are seeing good training results.
For the poor testing results, are you transforming your frames in the same way you did for training? Any differences between the training pipeline or phone testing pipeline you can think of? Is it possible to test some of your good training results with the phone testing pipeline as a sanity check?


Can't get Keras functional model to work: "could not broadcast input array..." For a really trivial MWE, here is some code that I expected to work and does not:

import tensorflow.keras as keras
import numpy as np
input_ = keras.layers.Input((256, 768))
layer = keras.layers.Lambda(lambda x: x[0, :], output_shape=(768,),
                            input_shape=(256, 768))
model = keras.models.Model(inputs = [input_], outputs=layer(input_))
model.predict(np.zeros((1,256,768)))

I expect this to return np.zeros((1, 768)), but instead it throws the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/holmes5/anaconda3/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 1078, in predict
callbacks=callbacks)
  File "/Users/holmes5/anaconda3/lib/python3.7/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 370, in
model_iteration
aggregator.aggregate(batch_outs, batch_start, batch_end)
  File "/Users/holmes5/anaconda3/lib/python3.7/site-packages/tensorflow/python/keras/engine/training_utils.py", line 169, in aggregate
self.results[i][batch_start:batch_end] = batch_out
ValueError: could not broadcast input array from shape (256,768) into shape (1,768)

What am I doing wrong?

Fixed it, sorry. I just needed to say lambda x: x[:, 0, :], so it included the sample index.


Why K-harmonic means is less sensitive to random initialization than K-means? As we know that K-Means is a powerful clustering, however it often stuck with local minimum problem because of bad initialization . One of the solution is K-Harmonic means that use Harmonic Average as the performance function instead of minimum distance btw the data and cluster's centroid. I am curious that why using harmonic function makes less sensitive to init centroids than ordinary K-Means.

Harmonic mean is less sensitive to outliers.

Say that you have 3 numbers: 1 1 and 30001

Their arithmetic mean (average) is 10001 and their harmonic mean is 2 that is much closer to the majority of the points.

So in a way, using an harmonic mean is more forgiving.

A Bayesian look at the harmonic mean

Assuming that the closer a point is, the more likely it is to be in a cluster.

One can choose to model the distribution of distances from an origin point, as an Gamma-Exponential model with a priors $\alpha=m$ and $\beta=\frac{m}{\mu_0}$ where $\mu_0$ is the initial mean and $m$ is the strength of the prior.

And given observations of $\{d_1,d_2,\dots,d_n\}$, the updated model's mean is:
$$\mu'=\frac{m+n}{\frac{m}{\mu_0}+\frac{1}{d_1}+\frac{1}{d_2}+\dots+\frac{1}{d_n}}$$
We got our beloved harmonic mean.

You could argue that the harmonic mean in k-means context is an assumption on the distribution of distances.


Linear Discriminant Analysis, which parameters can be tunned in cross validation set up? I am implementing Linear Discriminant Analysis in R, which parameters can be tunned in cross validation set up? In regularized mode called penalizedLDA there are parameters which are optimised but I want to know which parameters are turned in case of simple LDA method?

LDA has a closed-form solution and therefore has no hyperparameters. The solution can be obtained using the empirical sample class covariance matrix. Shrinkage is used when there are not enough samples. In that case the empirical covariance matrix is often not a very good estimator.


Field Aware Factorization Machines Can anyone explain how field-aware factorization machines (FFM) compare to standard Factorization Machines (FM)?

Standard:
http://www.ismll.uni-hildesheim.de/pub/pdfs/Rendle2010FM.pdf

"Field Aware":
http://www.csie.ntu.edu.tw/~r01922136/kaggle-2014-criteo.pdf

Suppose (prior to one-hot-encoding) you have predictors/fields from a set $Z$ (say movie genre, user gender, and user race).  Suppose further, each predictor $z \in Z$ can take on one of $k_z$ values.  After one-hot encoding, you will have a new set of binary features $X$ of size $K := \sum_{z \in Z}k_z$.

In a model with all interactions, you must estimate a matrix of interaction coefficients $Q$, which has $K\times (K+1) / 2$ unique terms.

The factorization machine puts structure on the matrix $Q$, and assumes that $Q \equiv W^{T} W$, where $W$ is of dimension $l \times K$, with $1\le l \le K$ some number specified by the user.  We estimate $W$ instead of $Q$.

The field-aware factorization machine puts structure on $Q$ as well.  It partitions $Q$ into blocks based on $z$ (the original features).  If $q_{z_i, z_j}$ denotes the $z_i,z_j$ block of $Q$, we assume that $q_{z_i,z_j}$ comes from the $z_i,z_j$ block of $W_j^{T} W_i$, where $W_i$ is of dimension $l \times K$.  As with the FM, we estimate the $W_i$ instead of $Q$.

The FM factorization of $Q$ has $K \times l $ parameters.  The "feild-aware" FM has $K\times l\times |Z|$ parameters.  A model with all interactions has $K \times (K+1)/2$ parameters.


Using sigmoid in binary DNN output layer instead of softmax? For a binary DNN, the output is $y_0 + y_1 = 1$ since they are the probability distribution, hence the sum must equate to 1. However, I've been told that $y_1$ is sufficient to represent the output of the DNN where:

$y_1 = \frac{e^{h_1}}{{e^{h_0} + e^{h_1}}}$

Hence, $h = h_1 - h_0$, why is it that $y$ which is the final output can now be represented as $y = \sigma(h)$, where $\sigma$ is the sigmoid function. Could anyone please explain why? How did it derive to the point where $y = \sigma(h)$?

In a binary classification problem you have only 2 classes, let's call them the negative and the positive class.
You only need to ouptut 1 number which corresponds to the probability of your input point to belong to the positive class.
The sigmoid activation function is good for that because it maps any input value to the range ]0,1[ which is what we want for a probability (it is not a real issue that 0 and 1 are excluded).

Since you have only one output number, it makes no sense to use a softmax activation.
Softmax activation is used in the multiclass problem where you must predict 1 of N classes where N is greater or equal than 3 and in this case the number of outputs is N (1 probability by class).
The softmax function makes that all your outputs sum to 1 and it amplifies the gap between high and low probabilities.


Understanding what is going on I have a collection of 1000s of bottles of wine. I want to understand what could / likely to be driving the price either up or down. Below is an example of the data

Wine      Country       Area        Grape        Class    Year    Price
A         France        Burgundy    Pinot Noir   Grand    2014    +6%
B         France        Burgundy    Chardonnay   1er      2014    -1%
C         France        Burgundy    Pinot Noir   1er      2014    +4%
D         USA           California  Pinot Noir            2013    +1%
E         USA           California  Blend                 2014    +0.5%
F         USA           California  Chardonnay            2014    -4%
G         USA           California  Chardonnay            2013    -5%

From these fictitious numbers i would assume the reason bottle A is up is because pinot is up but more importantly France is up V USA wine. To the same extent Chardonnay is doing poorly but USA again is doing worse than France.

Here this is rather easy but if this was extended by 1000s of rows with more categories added, the exercise becomes less easy to see the trend.

My original thought would be to use the importance of a feature from a decision tree. But is there a better way?

Thanks

What you are looking for is feature contributions to the final score of an observation having a positive or negative price. Feature importances from a decision tree or random forest is not going to help you because a feature's importance is fixed across all observations (wines). They tell a story about the overall model, but nothing about the individual observations. So if you want to know what caused a wine having a negative or positive price, you should look at feature contributions.

Let's elaborate a little bit.

Let's forget about decision trees, and go for ensemble of them, say random forest, for all the good reasons. (I highly recommend you to read why ensemble models uniformly beat the base learner, and how they help with the high bias - high variance problem base learners might have.)

The first step would be to label your dependent variable, price, as $1$ and $0$, depending on the price being positive and negative ($y = 1$ iff $price > 0$). After the typical exploratory analysis (where you can look at boxplots and stuff as a beginning study) and feature engineering, you can fit your random forest to predict a wine having a positive or negative price (i.e. $y$ being $1$ or $0$).

Once you do that, you will have an estimated probability for each and every wine, representing the classification model's estimation of that wine's price being positive or negative. Something like this:

Wine Prob
A    0.4
B    0.7
C    0.9
D    0.3

These probabilities actually consist of the summation of a bias term and the individual contributions of all features to that observation only:

$$P(y_i = 1) = bias + \sum_{i=1}^{m}(contributionOfFeature_i),$$

where $m$ is the total number of features. This is how you can analyze each and every feature's contribution on the final probability of a particulat wine's price being positive or negative.

Regarding how to code this, if you are using scikit, you can use this very easy and convenient tool:

https://github.com/andosa/treeinterpreter


Generate new images based on a dataset of images Is it possible take a training set of one million 120 x 120 pixel tiles, feed these tiles into a machine learning algorithm. And then make it synthesize images that look like the original training set with a certain, adjustable level of entropy?

What ML techniques should I consider?

Yes, there are plenty ways to do this, what you basically want is to interpolate between the two.

This is current state-of-the-art for solving problems like yours: https://github.com/carpedm20/BEGAN-tensorflow

Also keep an eye out for my question here: 2D Image-based Avatar Animation With Expressions / Image-To-Image Translation

If I end up using the ML approach, you could very well also use it with tiles instead of faces and be able to adjust the likeness. I will link to my project in my question.


Why do good word embeddings come out of maximizing cosine similarity? I have an understanding of the technical details of word2vec.

What I don't understand is why semantically similar words should have high cosine similarity.

From what I know, goodness of a particular embedding is seen in shallow tasks such as word analogy. I am unable to grasp the relationship between maximizing cosine similarity and good word embeddings

Training word embeddings does not rely on optimizing the cosine similarity of words. It usually relies on prediction problems. Take for instance the skipgram model: you are predicting the context of a word, given the word. Such models, project words in geometric spaces (for example the commonly used ~300 dimensions). In other words, a word is associated with 300-dimensional dense vectors. Due to the way that these vectors are learned, they capture the semantics of the words and hence similar words are close to the induced space.

Cosine similarity (or dot product) captures exactly this semantic closeness. Intuitively, we want similar words to be close, because they are similar and we hope that the space models some of the word properties and meaning.


Accuracy Testing & Training I have 102 observations. I made standard scale for dataset. I have found the accuracy training and accuracy testing values, but training score is 1.00 and testing score is -217.541. I have run with MLPRegressor score. What does negative accuracy testing score mean?

FileX_train, FileX_test,FileY_train,FileY_test = train_test_split(FileX,FileY,test_size=0.33, random_state=0)

sc = StandardScaler()
X_train = sc.fit_transform(FileX_train)
X_test = sc.transform(FileX_test)

sc = StandardScaler()
Y_train = sc.fit_transform(FileY_train)
Y_test = sc.transform(FileY_test)

from sklearn.neural_network import MLPRegressor
from sklearn.metrics import accuracy_score
mlp = MLPRegressor(solver='lbfgs',activation='tanh', hidden_layer_sizes=(100,), max_iter=1000000000, learning_rate='constant')
mlp.fit(X_train, Y_train.ravel())
print('Accuracy training : {:.3f}'.format(mlp.score(X_train, Y_train)))
print('Accuracy testing : {:.3f}'.format(mlp.score(X_test, Y_test)))

Results:

Accuracy training : 1.000
Accuracy testing : -217.541

You haven't actually used accuracy_score, and the default scorer for sklearn regressors is R^2:
https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html#sklearn.neural_network.MLPRegressor.score

A negative value for R^2 generally means a very bad fit:
https://stats.stackexchange.com/q/12900/232706