How to compare performance of Cosine Similarity and Manhatten Distance? I'm doing clustering of documents by applying k-Means on the word-vectors. To measure the cluster quality, I calculate David Bouldin Index for different k's. I tried two different distance measures, Cosine Similarity and Manhatten Distance, and get quite different values:

Cosine Similarity: ~0.8 to ~0.6
Manhattan Distance: ~0.3 to ~0.2

Can these values be compared directly? (Does Manhattan really performs a lot better here?) Or is there another way to compare the clustering results of the two different measures?

Comparing distance values of different distances is nonsense.

Consider this distance function:

$$d(x,y)=0$$

Clearly, this gives smaller values, but it also is useless.

Consider this trivial variation of Manhattan distance:

$$d(x,y) = \sum_i |x_i-y_i|/100$$

Clearly, this distance is equivalent to Manhattan, but will yield much smaller values.

Choose the distance because it is "the right thing to use", not because of some number. Cosine is good if you desire length normalization and have sparse vectors.


Dataset Made of Multiple Datasets I recall a tweet by Andrej Karpathy sometime a while ago about a ML dataset made up of cifar-10, mnist, ImageNet, and maybe others. I want to use that dataset, but I can't find it. Does anyone know if this still exists?

I found the dataset. Its called the "Visual Domain Decathlon"

Heres a link: https://www.robots.ox.ac.uk/~vgg/decathlon/


What is the right way to keep track of the different things we try? In Machine Learning we usually try many combinations of different features, filters we apply to the data, transformations on the features or the target variables and different versions of hyperparameters.

This fact makes it difficult to keep track of what works and what doesn't if we are not exhaustive with how we keep track of the different combinations we try.

I am wondering if there are any best practices around this problem. My current approach is to keep track of the different combinations naming files making reference to the parts that compose it, for example a hyperparameters pickle file I would name it booster_params_{}_{}_{}_{}.pickle'.format(filter_name, features_name, model_target, params_iteration)

where filter name is the set of filters I'm applying to the data, features name refers to the set of features used, model target to the target I'm modeling and params iteration refers to the version of the hyperparameters.

This seems like an overkill, and that is why I'm looking for ideas on how to tackle this problem.

You can maintain multiple versions of booster_params.pickle either:
1) through a version control system
or
2) manually, e.g. booster_params_v1.pickle, ...v2, etc. and a separate file where you would describe each version


Best Julia library for neural networks I have been using this library for basic neural network construction and analysis.

However, it does not have support for building multi-layered neural networks, etc.

So, I would like to know of any nice libraries for doing advanced neural networks and Deep Learning in Julia.

MXNet Julia Package - flexible and efficient deep learning in Julia

https://github.com/dmlc/MXNet.jl

Pros

Fast
Scales up to multi GPUs and distributed setting with auto parallelism.
Lightweight, memory efficient and portable to smart devices.
Automatic Differentiation

Cons

Doesn't have yet low level operations for algorithm implementation. But they are working on this issue (https://github.com/dmlc/mxnet/issues/586)


Use cases for graph algorithms and graph data structures in finance and banking I work in a bank and most data is in tabular format in relational databases. I have been reading about graph algorithms (page rank), graph libraries (spark graphx) and graph databases (neo4j). I would like to pick a use case from my field (finance). What use case would be suitable for graph algirithms and databases?

There are many use cases of graph theory in Finance industry and it is a very broad question. As Emre said can be used for Fraud Detection, Risk Modelling, Economic Networks etc.

These below links can give you better understanding of different application, please go through for better understanding:

Applications of Graph Theory In Finance
 Graph Theory for Systemic Risk Models
Graph theory: connections in the market
Analysis of Equity Markets: A Graph Theory Approach
Portfolio Diversification
From Graph Theory to Models of Economic Networks. A Tutorial

Do let me know if you need any additional information.


Is it OK to train a binary classifier using all the extremely imbalanced data if the majority class is negative? I'm training a neural network as a binary classifier for text classification. The data is very imbalanced, where the ratio of TRUE:FALSE is approximately 100:10000

Intuitively, it feels like using all the negative samples would prevent the classifier from learning invalid patterns (that might otherwise be learned using undersampling, for example).

Am I underestimating the effect of the imbalance on classifier performance?

An imbalance of this magnitude is definitely a problem. Because machine learning algorithms penalize model complexity and try to maximize fit, a model that always predicts "False" will often outperform any other model because it predicts the right outcome for 99% of the data and is super simple!

What one should do in this case is downsample (http://www.simafore.com/blog/handling-unbalanced-data-machine-learning-models), train the model, and then scale the resulting predictions back up to reflect the imbalance in the original data.


AWS SageMaker Model as endpoint size limit Is there a size limit imposed on models deployed on AWS SageMaker as endpoints? I first tried to deploy a simple TensorFlow/Keras Iris classification model by converting to protobuf, tarring the model, and deploying. The size of the tarred file was around 10KB, and I was able to deploy that successfully as an endpoint. However, I tried the same process with a Nasnet model where the size of the tarred file ended up being around 350MB, and I got the following error:

  The primary container for production variant AllTraffic did not pass
  the  ping health check. Please check CloudWatch logs for this
  endpoint.

Could it be because the model is too large to deploy? I tried increasing the instance type from 'ml.m4.xlarge' to a higher tier but that did not work either.

It doesn't seem that it is about the size of the model. I am no SageMaker expert, but the error message suggests that the model was deployed, but that something went wrong when the health check was run.

This could be caused by many different things, but the most probable would be that there is a bug in the code. Please check the following:

Can the model be loaded properly?
Can the model make a prediction?
Can the model make multiple predictions?


Prioritized Experience Replay - why to approximate the Density Function? I am reading about Prioritized Experience Replay, and can't understand the following:

On page 4, every transition can be selected from the table with its own probability.
Here is the cumulative density function (if I understood correctly):

$$P(i) = \frac{p_i}{\sum_k{p_k}} $$

where:

$$p_i = \frac{1}{index In Table}$$

Aterwards, the paper says:

  For the rank-based variant, we can approximate the cumulative density function with a piecewise
  linear function with k segments of equal probability. The segment boundaries can be precomputed
  (they change only when N or α change). At runtime, we sample a segment, and then sample uniformly
  among the transitions within it.

My question is, why do we have to approximate the density if it can be achieved with the following:

roll a dice between 1 and N (with a dice that is exponentially more likely to roll a '1' rather than a '2', etc)
select an item from index according to the dice.

In c++ we have std::exponential_distribution [source] so there is no need to approximate anything. ...If we maintain our table sorted in a descending order.

That's because the probablity density function might not follow an exponential curve, especially if formula is not $p=1/rank$ but is instead proportional to the amount of error. This will be a byproduct of us not keeping the array sorted, because we want to simplify our life. The curve could then look like this:

Here, area under the curve is one, but the curve is not exponential or linear or linear-horizontal.

Roll a dice from 0.0 to 1.0, which by the way has the same probability to roll 0.71 as 0.92 etc. Select an element from the array (doesn't have to be sorted!), scanning the array from left to right.

Consider array with 4 numbers:  {5, 15, 25, 10}. Each represents a portion from  the total sum:

|0.1|     0.3     |           0.5           |0.2|
you got to be pretty lucky to land on 0.1 (within first 10% segment)

The likelyhood of selecting an element is dependant on its "height under the curve". The taller the element, the more likely for our 0-to-1 counter to land on it.

You can't represent such "landing probability mechanism" by the "exponential dice approach" mentioned in my original question.

However, selecting by the "exponential dice" - albeit not precise might nonetheless work. The only caveat is the array's would have to be kept ordered, which is a pain in the rear. So the benefit isn't really there, just keep the array unsorted and go with the "segmented tree" aka "sum tree" approach O(logn) cost on all operations. This would allow you to avoid scanning the array left-to-right, helping you get from O(N) to O(logn) sampling performance

Now please help me with this question instead :)


Aggregating small values in a frequency bar plot I have a pandas Series of sorted percentage values like this :

A -> 0.001
B -> 0.0012
C -> 0.0015
...
H -> 0.02
I -> 0.03
J -> 0.041
...
X -> 0.12
Y -> 0.31
Z -> 0.4

(I typed those by hand, I am not familiar with how to type those python outputs in DSSE, sorry! If anyone has a hint on that, it would be nice. If this pandas series would be named "content_distribution", content_distribution.X would return the float 0.12 .)

I want to obtain a Series which looks like this :

Other -> 0.07
I -> 0.03
J -> 0.041
...
X -> 0.12
Y -> 0.31
Z -> 0.4

because those small values are irrelevant to display in a plot that I want to make, and I want to keep the big values on display. Is there any pandas-ic or pythonic way to do that? I can write my own script, but anything I try out looks non-pandas-ic (even though it sort of works).

(I also don't know if "aggregating" is the correct pandas term for this, if anyone has a better one, he or she is welcome to edit my title.)

Here is an example where I create a new row called Other, which contains the sum of all values below a given threshold.

I then remove the rows that were below that threshold (and so included in the sum), so the final Pandas Series only has values above the threshold, plus the new Other row.

This is all performed on a randomly generated Pandas Series, as follows:

In [1]: import pandas as pd, numpy as np

In [2]: X = pd.Series(np.random.random(20))

In [3]: X
Out[3]:
0     0.507151
1     0.366259
2     0.444196
3     0.027280    this will be in the sum
4     0.132785    this will be in the sum
5     0.993170
6     0.614712
7     0.942894
8     0.516255
9     0.107436
10    0.710416
11    0.512221
12    0.502685
13    0.753515
14    0.894735
15    0.780213
16    0.998315
17    0.971558
18    0.504525
19    0.224767
dtype: float64

In [4]: threshold = 0.2

In [5]: X.loc['Other'] = X[X < threshold].sum()

In [6]: X.drop(X[X < threshold].index, inplace=True)

In [7]: X
Out[7]:
0        0.507151
1        0.366259
2        0.444196
5        0.993170
6        0.614712
7        0.942894
8        0.516255
10       0.710416
11       0.512221
12       0.502685
13       0.753515
14       0.894735
15       0.780213
16       0.998315
17       0.971558
18       0.504525
19       0.224767
Other    0.267501
dtype: float64

Notice that the indices still match the original values, i.e. they are not continguously numbered - indeices 3 and 4 have been dropped/removed.
If you want to get continuously numbered indices back, you can do the following:

X.reset_index()

This creates a new index and puts the original one shown above in a new column called index, which can then itself be dropped if desired:

X.drop('index', axis=1)

That will remove the label Other, which was in the original index.


Do we need the outer discount term when implementing REINFORCE algorithm I am learning the REINFORCE algorithm, which seems to be a foundation for other algorithms.
I saw the $\gamma^t$ term in Sutton's textbook.

But later when I watch Silver's lecture on this, there's no $\gamma^t$ term.

I read several implementations of the REINFORCE algorithm and seems no one includes this term. (However, when calculating return $G_t$, they do use $\gamma$ to do discounting.)
I am wondering, is it OK to just omit the outer $\gamma^t$ while still using it to calculate the return ($G_t$ or $\text{v}_t$)? Does the outer $\gamma^t$ term have nothing to do with the inner $\gamma^{k-t-1}$ term?
Thanks.
FYI, this is the implementation I studied: https://github.com/pytorch/examples/blob/master/reinforcement_learning/reinforce.py

The answer to your question is already in the text which you copied from Sutton's and Barto's book.
Sutton's and Barto's pseudocode describes the general, discounted, case. While the lecture by Silver talks about the undiscounted case. And in the undiscounted case $\gamma=1$ and therefore $\gamma^t=1$. So you can just omit it. Equation (13.8) in Sutton's and Barto's textbook also provides this simplified equation for the undiscounted case.
Also note that Monte Carlo methods are applied to episodic MDPs only. Therefore, in many cases you might not need the discounted version.


How to customise cost function in Scikit learn's model？ For example, when I have a problem that false negative should be penalised more, how can I incorporate that requirement in the algorithm such as SVM?

There are several ways in which you can achieve your desired result:

Implement the make_scorer function from Scikit learn
Make modification to the class_weight argument

In regards to your SVM question take a look at the below code:

 class sklearn.svm.SVC(C=1.0, kernel='rbf', degree=4, gamma=0.0,
 coef0=0.0, shrinking=True, probability=False, tol=0.003,
 cache_size=300, class_weight=None, verbose=False, max_iter=-1,
 random_state=None)

In the above example, the class_weight function can be changed to 'auto' or you can pass dictionary values which have the user-defined class weights.


Why does averaging a sentence's worth of word vectors work? I am working on a text classification problem using r8-train-all-terms.txt, r8-test-all-terms.txt from https://www.cs.umb.edu/~smimarog/textmining/datasets/.  The goal is to predict the label using a Random Forest classifier.

Each text sentence as been vectorized using the GoogleNews word vectors. The embedding source can be found here: https://github.com/mmihaltz/word2vec-GoogleNews-vectors

In the example I am following along with there is one step that irks me - there is a step that converts my array of vectorized tokens to a single vector by taking the mean over the tokens e.g.

def transform(self, data):
        v = self.word_vectors.get_vector('king')
        self.D = v.shape[0]

        X = np.zeros((len(data), self.D))
        n = 0
        emptycount = 0
        for sentence in data:
            tokens = sentence.split()
            vecs = []
            m = 0
            for word in tokens:
                try:
                    vec = self.word_vectors.get_vector(word)
                    vecs.append(vec)
                    m += 1
                except KeyError:
                    pass
            if len(vecs) > 0:
                vecs = np.array(vecs)
                X[n] = vecs.mean(axis=0)    # take the mean of the vectors?  what does it mean?
            else:
                emptycount += 1
            n += 1
        print("Number of samples with no words found: %s / %s" % (emptycount, len(data)))
        return X

I am leaving out some boilerplate but later on I run the model and the results are surprisingly good:

model = RandomForestClassifier(n_estimators = 200)
model.fit(XTrain, YTrain)
print("train score:", model.score(XTrain, YTrain))
print("test score:", model.score(XTest, YTest))

> train score: 0.9992707383773929
> test score: 0.9378711740520785

I understand that the random forest model expects to have one row per example so it is unable to consume a sequence of embeddings like a RNN might.  So you are required to convert to single row (1-D array).

My question is: WHY does it work?  It seems at odds to me that the averaged word vectors would be able to capture anything about the context or meaning of a sentence by merely averaging over the encodings.

Best case scenario I would expect this technique breaks down for larger blocks of text because you would tend to squash all your examples into the same neighborhood of your input space.

It would be great to get some clarification on this.

It works for the same reason why the good old bag-of-words + TF-IDF works. Despite loosing some word ordering information, a text can be still classified by the typical keywords. Since texts on different topics differ a lot with respect to the vocabulary used, simply putting together the words' embeddings might work surprisingly well.

Here is a paper that shows that a simple sentence embedding methods beats sophisticated supervised methods including RNNs and LSTMs. Their method is just a weighted average of the word vectors, modified a bit using PCA/SVD. Section 4.3 tells that the word ordering plays a role, but not too much.


B.Tech Project for final year of College I am an Engineering student in NSIT, Delhi, India. In our last year of B.Tech degree we have a B.Tech project, aka, BTP. I am in ECE(Electronics and Communication Engineering) dept.

Usually, my ECE batch mates would perform the project majorly regarding electronics, but I want to do in data Science or data manipulation under other departments(COE or IT). I approached a COE mentor under whose supervision I would like to do the project. But I have been struggling in choosing the right topic for project.

There are a lot of suggestion online, and many are preferably done by students who just want to complete a project instead of it been already done and executed by others. I want some decent approach suggestions for choosing the right project topics as I am quite confused right now.

  What student level data Science or, specifically, Data manipulation
  projects can I do in my final year of College?

Do machine learning on microcontrollers or other edge devices (link1, link2, link3, link4, link5). This is a hot topic, new emerging technologies, and it will make your profs happy from both electronics and IT departments.

There was a recent news article (link) where a team from the Fraunhofer IMC did a handwritten digits recognition on an Arduino Uno. They claim that they not only run a trained neural network (which would be easy) but actually train the network on Arduino.

However, they don't show their code. On their license page (link) they only say some blah-blah-blah about discussing possibilities with partners and customers and boast 30 years of experience in the development of microelectronic circuits. This is lame. I am sure that this can be done by an undergraduate student without 30 years of experience and that student will write a blog post about it.

If you develop a machine learning code that would do such tasks on an Arduino or even something smaller, like ATTINY85 that would be cool. Your model should be able to be trained on a microcontroller given the restrictions of its memory and computational capacity. Then upload your code on a github or a blog, put your link here. If your code is good, it will get lots of likes, you will showcase your abilities to your employers, and will put those Fraunhofer dinos who hide their "great discoveries" to shame. Besides, it is very useful because of emerging technologies with IoT, 5G, etc.

By the way, it does not have to be a neural network. Other models, like SVM, KRR would also be great. Also, it does not have to be Arduino. Other microcontrollers, like PIC or ST are also OK.


The problem might arise because of the meta-text in the .csv or .txt file that is not really written there but is copied when its content is loaded somewhere.

I think it is better to first import your text in an array or a string and then split it and save into the dataframe specifically when your data is not too large.

import csv
arrays = []
path = "C:\\Users\\Souro\\Downloads\\AXISBANK.csv"
with open(path, 'r') as f:
   reader = csv.reader(f)
   for row in reader:
       row = str(row).replace('\\', '') #deleting backslash
       arrays.append(row)

Then take a look at arrays[:10] to find where the meta data ends and delete the unwanted data (meta data)
and then converting the 'arrays' array into the dataframe.
for instance:

arrays = arrays[9:]
df = pd.DataFrame(arrays[1:], columns=arrays[0]) #arrays[0] is the columns names

about your comments:

if you look at the text in each row (print each row), you would find out that a backslash is at the end of each row, so by replace('\',' ') we are substituting each backslash with nothing(''). why two \? It is the way that we declare backslash, otherwise, it won't be recognized.

row=str(row).replace('\\',' ')

and about

open('text.txt','r')

It opens the file 'text.txt' in reading mode (r).


why leave-one-out encoding? This question is from this post asking about what is the difference between one-hot encoding and leave-one-out encoding.

I now understand how to calculate the leave-one-out encoding for the training set. But how about the test set, where is Y (label) is blinded? If the encoding for test set is always 0.5, then what is the point of this encoding?

The point is to learn useful variations of data instead of just splitting by large categorial variable. Each new row after encoding becomes immediately related with the output, while original categorial variable may be related only in indirect, latent manner. Plus, the interactions between output and the original variable are represented too, by definition. Think of it as of adding this interaction explicitly to add more intuitive justification to this method.

So I think this method will be perfect not only for divide-and-conquer RF-style methods but also for plain LR as well.

  if the encoding for test set is always 0.5, then what is the point of this encoding?

That's okay, as soon as you know how this should be related to the output.
0.5 in this example is just a rough approximation of input, but given that you have learned model even this approximation is still meaningful, as it instructs model -- which path to output to choose. If your model of choice is RF then 0.5 will land at about same set of leaves as more granular distribution ( if you were able to know it )

Meanwhile much stronger approximations are possible, for instance, one can use KDE to estimate response variable distribution and then draw samples from it at test time. Adding uniform random noise is just a hint in this direction. Maybe you want to follow it?

Knowing only particular level of categorial variable is way less informative, than this 0.5, isn't it?


What is the meaning of the parameter "metrics" in the method model.compile in Keras? I don't have very clear the meaning of the parameter metrics of the compile method of the class model in Keras:

model.compile(..., metrics = ['accuracy'], ...)

the documentation states:

  List of metrics to be evaluated by the model during training and testing

what I don't understand is:

are these the metrics used to evaluate the performance of the network at the end of each training epoch, i.e. at the end of each epoch the code makes the network predict on the training set and calculates the metrics passed

OR

are these the metrics used to train the network, i.e. is the network trained with the goal of obtaining the best possible value for these metrics (I don't know how it could do this, something like: if the recall is low then it corrects the weights more when working with positive samples)?

(or it has another meaning?)

The argument metrics is meant to define your criterion for training evaluation. Let me make an example: if you are training a classifier, you want to evaluate your model based on how accurate (in percentage) it is. Therefore, your metric is accuracy (experessed as a float in the [0, 1] range). The higher the accuracy, the lower the loss, the better the model.

metrics must not be confused with loss. The loss function is something you need in order to "punish" your model when it makes mistakes. The loss function is at the basics of backpropagation and of weight update, it's the loss object what Neural Networks use to learn. Metrics instead is something that humans watch to understand how good a model is and communicate it.

The definition of some metrics is optional, you can evaluate a model based on the loss value only if you want. Sometimes you don't need to specify it. For example in regression tasks, i.e. when you have to predict continuous outputs, you speecify a loss (usually MSE or RMSE) and also evaluate your model based on that.


Setting best SVM hyper parameters I have a non linear data set, and I am using SVM (RBF kernel) to build a classification model, but not sure how to set the best hyperparameters of the SVM, C and gamma in Matlab fitcsvm.  And is it acceptable if it is trial and error approach setting arbitrary values, until finding the best performance?

Well, there is a bunch of articles that tries to tackle this problem but basically, to guarantee a good solution you will need to do Grid Search (sklearn tutorial on it)

You can use various techniques for that, for example:

Binary Gridsearch: Try the maximum and the minimum value in the interval you want, and the middle point, check which side is best and set the minimum or the maximum as your previous middle point according.

That can help you find an average solution fast, but remember that this comes without warranty.

Exhaustive Grid Search
Randomized Parameter Optimization
Genetic Algorithms

Check the link for sklearn for more information. I research on kernel methods and this is aways a bit annoying thing to tackle.

Note: If you're only going to try RBF I would advise to use $\sigma$ as a value betwen the norm of your training samples $\pm 80\%$. You could also normalize you vectors to norm 1 and limit the search to $0.2$ to $1.8$. Usually too small $\sigma$ gives high sensibility to noise while too big aproximates a straight line (loses non-linear power)


Keras RNN (batch_size I created an RNN model for text classification with the LSTM layer, but when I put the batch_size in the fit method, my model trained on the whole batch instead of just the mini-batch _size.
This also happened when I used GRU and Bidirectional layer instead of LSTM.
What could be wrong?
def create_rnn_lstm():
    input_layer = layers.Input((70, ))
    embedding_layer = layers.Embedding(len(word_index) + 1, 300, weights=[embedding_matrix], trainable=False)(input_layer)
    embedding_layer = layers.SpatialDropout1D(0.3)(embedding_layer)
    lstm_layer = layers.LSTM(100)(embedding_layer)
    output_layer1 = layers.Dense(70, activation="relu")(lstm_layer)
    output_layer1 = layers.Dropout(0.25)(output_layer1)
    output_layer2 = layers.Dense(2, activation="softmax")(output_layer1)
    model = models.Model(inputs=input_layer, outputs=output_layer2)
    model.compile(optimizer=optimizers.Adam(), loss='binary_crossentropy')
    return model

LSTM_classifier = create_rnn_lstm()
LSTM_classifier.fit(X_train_seq, y_train, batch_size=128, epochs = 10, shuffle=True)

In Keras, with verbose=1 (default parameter of the fit method) will display the total number of samples, not the batch number.

If your batch size is 128, then the progress bar will jump by multiples of 128. You can try to change batch_size parameter to 13714, and you will see the progress bar jumping straight from 0/13714 to 13714/13714, since you would have defined a batch size of the size of your whole training dataset.

Finally, if you want to train your model only on one mini batch, you can extract it from your training set and fit your model on it.

import random
indices = random.sample(range(13714), 128)

X_train_seq_one_batch = X_train_seq[indices]
y_train_one_batch = y_train[indices]
LSTM_classifier.fit(X_train_seq_one_batch, y_train_one_batch, batch_size=128, epochs = 10)

The above code will train your network only on one random batch only. random library is used to draw a random batch of indices in your training set.


Difficulty interpreting word embedding vector similarity (spaCy) I calculate vector similarities like this:

nlp = spacy.load('en_trf_xlnetbasecased_lg')
a = nlp("car").vector
b = nlp("plant").vector
dot(a, b)/(norm(a)*norm(b))
0.966813

Why are the vector similarities so high for unrelated words for the embedding? This is not the only pair for which they are abnormally high. I also had a similar experience with fastText, so I am wondering, am I misunderstanding something?

Also I am able to get vectors for non-words like "asdfasfdasfd" or "zzz123Y!/§zzzZz", and they differ from each other. How is this possible?

Why are the vector similarities so high for unrelated words for the embedding?

For the specific example you give, I would argue that it makes sense that car and plant  have high similarity. This is likely due to phrases such as car manufacturing plant

  Also I am able to get vectors for non-words like "asdfasfdasfd" or "zzz123Y!/§zzzZz", and they differ from each other. How is this possible?

For your specific case, since you use the en_trf_xlnetbasecased_lg, the answer is straightforward. Embeddings provided by XLNet are contextual, meaning that even if the word itself isn't a word, you'll get an embedding given the words in its context. Also, it is likely that HuggingFace's implementation uses Byte-Pair Encoding as tokens, making it much more robust to out-of-vocabulary situations.


Why can't I use 2D-arrays as features for CCA (Canonical Correlation Analysis) classifier? The Problem
When using fit of the scikit learn CCA classifier it won't allow me to use arrays as features. The error ValueError: Found array with dim 3. Estimator expected <= 2. can be produced whit the following code
from sklearn.cross_decomposition import CCA
    CCA_model = CCA(n_components = 3, max_iter=20000)

    input_arr = [[[k*-1+j*-i*-1 for k in range(125)] for j in range(2)] for i in range(189)]
    input_arr = np.array(input_arr)
    print("INPUT SHAPE:", input_arr.shape)
    input_lbl = [[(-(-1+(-1)**(1+k+j)))/2 for k in range(3)] for j in range(189)]
    input_lbl = np.array(input_lbl)
    print("LABEL SHAPE:", input_lbl.shape)

    model = CCA_model.fit(input_arr, input_lbl)

>>INPUT SHAPE: (189, 2, 125)
>>LABEL SHAPE: (189, 3)

The question
Why is it so, shouldn't it be allowed to use arrays as single features? Is there any parameter I need to modify to do this?

No, coding-wise, it cannot use multi-dimensional arrays as single features. You can flatten the array though. Essentially, you want to reshape your input to be (189, 250).


Data weight averaging say I have 2 datasets and need to get an average but with a weight based on the number of students.

school 1  = 98.1 . (50 students)
School 2 = 95.4 .   (169 students)

How would i get the average of both with a weighted average reflected the average of the student totals i think it would be around   = 96.ish

thanks for any insight!


How to forecast product/item sales next one week using Xgboost regressor I have trained xgboost algorithm to predict the number of items sale on a given day and got pretty good results, now I would like to forecast sales ahead of one week.

I tried re-training the algorithm by adding date as one of the independent variables, however, got an exception on date type.

So can someone please suggest how to predict forecast item sales count next one week.

In-sample predictions code:

train_x,X_test,train_y,y_test=train_test_split(data,data_y,
                                               test_size=0.30)
xgb = xgboost.XGBRegressor(n_estimators=100, learning_rate=0.08, gamma=0, subsample=0.75,
     colsample_bytree=1, max_depth=7)
xgb.fit(train_x,train_y)
predictions = xgb.predict(X_test)

Yes, you cannot pass in a datetime object as a feature to XGboost Regressor, because it isn't time-series forecasting tool such as ARIMA.

Instead, you create various features from the timeseries object such as year, month, day of month, day of week, number of days until public holiday etc. All of these are of type int. Drop the timeseries column.

When you're passing in the test data, you convert the datetime object into the above features.


How to replace levels with certain value? I have a categorical variable in my dataset. I want to replace the levels which are present in test dataset and not present in training set with a value called "others"

Here is how it looks:

levels(training$var1) has levels as "1"  "2"  "3"  "Others"

levels(testing$var1) has levels as "1"  "2"  "3"  "4"  "5"  "6"  "7"  "8"  "9"  "10"

I want to replace all the levels in testing data which are not training data. To achieve this I take the difference between the levels first.

a <- setdiff(levels(levels(testing$var1),training$var1)).

and I get output as

levels(a) as "4"  "5"  "6"  "7"  "8"  "9"  "10"

Now I need to replace all the above difference values with "Others". Kindly note that I do not want to drop out these levels but I want to replace with "Others".

For this I tried

testing$var1[testing$var1 == "4" <- 'Others"
testing$var1[testing$var1 == "5" <- 'Others" ==> these works.

However I want to make it in run time, something like this:

testing$var1[testing$var1 == a[1,] <- 'Others"

But this is not working.

I could be able to achieve this with the following code:

testing$var1 <- as.character(testing$var1)
a <- data.frame(a)
testing$var1 [testing$var1 %in% a[1,] <- "Others"
testing$var1 <- as.factor(testing$var1)

In case if there is any other better/effective solution/function to achieve this, please let me know.

Thanks all.


Reading Persian Characters in R I have a csv file with characters in Persian and I cannot view them in R correctly. Also, I cannot subset based on Persian characters values. Here is a sample code:

list1 <- c("x","y")
list2 <- c("ب","الف")

list1
list2 ##OK-readable

writedf <- as.data.frame(cbind(list1,list2),encoding="UTF-8")
write.csv(writedf,"test.csv")
testdf <- read.csv("test.csv",encoding="UTF-8")

testdf
## not readable

testdf[,testdf$list2=="ب"]
## data frame with 0 columns and 2 rows  ???

make sure the files are saved in UTF
try Sys.setlocale("LC_ALL", locale_code) and have a look at the documentation of this function


Open source solver for large mixed integer programming task? I'm currently using General Algebraic Modeling System (GAMS), and more specifically CPLEX within GAMS, to solve a very large mixed integer programming problem. This allows me to parallelize the process over 4 cores (although I have more, CPLEX utilizes a maximum of 4 cores), and it finds an optimal solution in a relatively short amount of time.

Is there an open source mixed integer programming tool that I could use as an alternative to GAMS and CPLEX? It must be comparable in speed or faster for me to consider it. I have a preference for R based solutions, but I'm open to suggestions of all kinds, and other users may be interested in different solutions.

Never done stuff on that scale, but as no-one else has jumped in yet have you seen these two papers that discuss non-commercial solutions?  Symphony and COIN-OR seem to be the dominant suggestions.

Linderoth, Jeffrey T., and Andrea Lodi. "MILP software." Wiley encyclopedia of operations research and management science (2010). PDF version

Linderoth, Jeffrey T., and Ted K. Ralphs. "Noncommercial software for mixed-integer linear programming." Integer programming: theory and practice 3 (2005): 253-303. Compares performance


count_vectorizer.vocabulary_.items() and count_vectorizer.vocabulary_ Inconsistent number of returns # create a count vectorizer object
count_vectorizer = CountVectorizer()

# fit the count vectorizer using the text data
count_vectorizer.fit(data['text'])

# collect the vocabulary items used in the vectorizer
dictionary = count_vectorizer.vocabulary_.items()

To my understanding, after count_vectorizer fits to data['text'], it generates a list of features. In my case, it generated 25,257 features and these are mapped as dict data type when I call count_vectorizer.vocabulary_. Which is still 25,257 tuples. It means, it used all the features.

Problem is, when I call count_vectorizer.vocabulary_.items() it returns 15,142 tuples as  dict_items. Why the number has been reduced here? Should't all the features be used to make the dictionary?

Here are the lengths I'm talking about:

len(data['text'])  #19579
len(count_vectorizer.get_feature_names())  #25257 items
len(count_vectorizer.vocabulary_)  #25257 items
len(dictionary) #15142 items (??????)

For future readers: as Ben wrote in the comments, it is really hard in general that len(my_dict) != len(my_dict.items()). When these kind of strage behaviours happen, it is always a good practise to perform some routine checks:

Clean your environment from every variable, even better restart the kernel and then run again your code.
Check your code for variables with same name, be sure you're not overwriting stuff, it's easy also to end up defining functions with same name of other imported library functions.
Try your code on different datasets, to be sure the mistake is not due to some external issue with the data.

If none of the above checks solve the mystery, then you can start thinking about a bug, a corrupted installation of a library or some other more nested problems.


Can regression trees predict continuously? Suppose I have a smooth function like $f(x, y) = x^2+y^2$. I have a training set $D \subsetneq \{((x, y), f(x,y)) | (x,y) \in \mathbb{R}^2\}$ and, of course, I don't know $f$ although I can evaluate $f$ wherever I want.

Are regression trees capable of finding a smooth model of the function (hence a tiny change in the input should only give a tiny change in the output)?

From what I've read in Lecture 10: Regression Trees it seems to me that regression trees basically put the function values into bins:

  For classic regression trees, the model in each cell is just a constant estimate of
  Y .

As they write "classic" I guess there is a variant where the cells do something more interesting?

Regression trees, particularly gradient boosting (essentially many trees), tend to do very well on continuous predictions, often outperforming models that are truly continuous like linear regression when. This is especially true when there are variable interactions and when you have a sufficiently large dataset (over 10,000 records) so that overfitting is less likely. If your primary objective is simply predictive power, then whether the model is 100% continuous or pseudo continuous should be irrelevant. If making your regression trees more continuous enhances out of sample predictive power, then you can simply increase tree depth or add more trees.


Which neural network to choose for classification from text/speech? I am considering two tasks:

Dialog Act Classification from Text (e.g. classify to: question; opinion; ...)
Emotion Recognition from Speech (e.g. happy; calm; sad; ...)

Which DL model should perform better for such tasks? I am planning to use CNN which should work for both of them, however not sure how well. Can I apply LSTM or some other methods? I used Keras before.

Is it good to apply attention mechanism or some other approaches for these 2 tasks?

Welcome to the site. I'm a little disturbed by the other two answers you received here. It sounds like you are skipping a whole lot of steps and wanting to jump right into modeling - that's a massive mistake!

You are a scientist! Your role is to create the most fair, unbiased environment possible to let the data speak to you (not the other way around!). What worked before (LSTM) may or may not be the best approach to this completely new data set. Therefore, after doing your EDA phase, you should keep an "open field" view to the multiple models that you will examine and test prior to making any decisions about which model to proceed with. The answer may not even be a neural network, it may be a whole different approach.

Please, be responsible your data science practice. You cannot jump into modeling right away. Let the data speak to you.


K-means and LDA for text classification I hope to explain in a clear way what I would like to do. I have more than 50000 tweets and I would like to add some labels on topics. So I have used LDA for doing this. I have also used k-means to group them and try to predict the cluster (but not the topic). I would like to know if it is possible to associate to clusters the topics found from LDA or if the two approaches are worthless when used simultaneously.

Try to look at it from a broader perspective, what you're essentially doing is wanting to build a sort of multi-class classification model based on your clusters and the topics LDA assigned.
It's like wanting a clustering model and a classification model to do the same thing, which would ultimately depend upon the dimensional space the data points are in and their respective vectors.
I assume you must be using some sort of tweet to vector conversion method to represent a tweet, like bag of words or glove doc2vec or something. Now making clusters on this vector space and hoping this does the same segregation as LDA does to allot topic is quite farfetched. I don't think so, your vector space will be so much in line with the topics LDA produces for your clusters to reflect the same. Also clustering puts data points in same clusters based on certain attributes, but determining why a cluster is made based on these attributes if the data dimensions are is very difficult.


How to auto tag texts Suppose we have predefined list of tags Tag #1, Tag #2, ..., Tag #N
and we want to assign tags to sample texts based on keywords or semantic analysis. A text can have multiple Tags. I don't want to consider it as classification problem assuming each tag as a separate class. What algorithm we can use here, Any suggestions would be of great help.

This is a multi-label classification task. Several labels can be assigned to each text. You can do this task using LSTM and sigmoid functions. The sigmoid functions are in the output layers and number of them is equal to number of your tags.


When does fitting happen in KNN? In training session, model fitting happens to reduce error. But does KNN do this?

Reducing error only happens due to changing K value and number of features, isn't it?

So training set and test set is only for do things below, right?

Train model with training set
Given test sample, model finds K-nearest neighbors in training set
Do classification or regression for test sample
Find accuracy with MSE or RMSE

Short version: this is just terminology, but arguably K-NN doesn't actually fit a model.

Conceptually, K-NN is a lazy learning model. This means that, in a sense, there's no fitting until a new instance arrives to be classified or a value has to be predicted (depending if you're using K-NN for classifying or regression, both are possible). Even then, using the term "fitting the model" is a bit out-of-place, in my opinion.

I think your confusion is in the sentence "model fitting happens to reduce error". No, it does not. Model fitting is simply getting a model (for example, a family of data distributions) and fitting the model by finding the parameters that better describe the data (thus choosing a member from that family).

Maybe you're thinking of neural network's epochs as "fitting"? What happens there is that there's a family of possible models (all the possible values for each weight in the network), and fitting the model is just finding the best possible values. The fact that neural networks do so in an iterative manner does not mean that fitting is an iterative process, fitting is the end result.

  So training set and test set is only for do things below, right?

Well, that's true for every model, yes. Training and test set separation serves the only purpose of evaluating the model on the chosen metrics. At the end of the day, the final model you choose to deploy, be it lazy or eager, will use all the available data (or a portion/transformation of it, in some cases), not just the training set.


How to do Predictive Modeling for Different Companies There is Company X

Company X wants to predict if its opportunities will Win or Lose

I trained my model based on the data of company X
and did the prediction of new opportunities

Now Company Y and Company Z wants to buy this product

Should I predict their opportunities based on my trained model of (Company X) ?

OR

Do a new model that is trained based on (Company: X+Y+Z)

OR

Do 3 separate models for each company

What is the best approach if I want other companies to use my model? In this case its 3 only, what if I have 20 companies

Thanks

Interesting question and the answer is: it depends.

You would have to provide more information about the exact inputs that are needed for the model and you would have to be 100% certain that each input means the same thing for each company. So, for example, you would have to be sure about things like timelines - if you are looking at "sales" and one company measures sales monthly, but the other quarterly then that's a problem for your model.

Also, you would have to consider the scale of the inputs. If one company has sales in the millions of dollars while a smaller competitor has sales in the thousands of dollars, that's also a problem for your inputs.

If you can think about issues above and still decide that it's OK to continue, then yes, you can use the same model. This is called transfer learning and you can take a trained model and then "update" it with the data from the new company.

But if you do run into issues with things like the above (or some other variation on issues with inputs) then I would recommend that you use your third option and just develop a model for each. If you have a framework that works for one company (i.e.  you have a model and you have a sense for the data you need) you should find that you are able to create similar accuracy numbers, even with brand new data.

Personally, I think there is a 4th option. You could just handle all of this from the data side. I would develop a new model for each company, but I would develop Company Y's model based on Company Y data but then also augment my training with Company Z data. You can choose the mix you want, but by including records like this you are still leveraging the power of writing a single model for a company with the added industry/competitor knowledge.


kNN(classifier) - Disadvantages So I recently came along kNN k nearest neighbour.
When looking at its disadvantages, most of the literature mentions it is costly, lazy, requires full training data plus depends on the value of k and has the issue of dimensionality because of the distance.
Other than that I have following hypothesis.

1- It ignores the fact that dimensions can be inter related and instead assumes they are independent (as we are just calculating distance)
2- Has the issue of normalization of data... if the data is not normalized distance can be biased towards a specific dimension

I will like to have a comprehensive analysis on the disadvantages of kNN apart from those mentioned above and if they are wrong then why.

It doesn't handle categorical variables very well
It doesn't handle 'soft' boundaries - i.e. areas where some cases appear on either side of a boundary.
See also mathbabe here: https://mathbabe.org/?s=nearest+neighbor - for extended criticism.


Use TSFRESH-library to forecast values Have some issue with understanding how to use TSFERSH-library (version 0.4.0) to forecast next N-values of particular series. Below my code:

    # load data train/test datasets
    train, Y, test, YY = prepare_train_test()
    # add series ID
    train['TS_ID'] = pd.Categorical(train['QTR_HR_START']).codes
    test['TS_ID'] = pd.Categorical(test['QTR_HR_START']).codes
    # add ordered id for concrete event of series
    for id in sorted(train['TS_ID'].unique()):
        train.ix[train.TS_ID == id, 'TIME_ORDER_ID'] =  pd.Categorical(train[train.TS_ID == id]['DATETIME']).codes
    for id in sorted(test['TS_ID'].unique()):
        test.ix[test.TS_ID == id, 'TIME_ORDER_ID'] = pd.Categorical(test[test.TS_ID == id]['DATETIME']).codes
    # perform feature extraction for my signal
    extraction_settings = FeatureExtractionSettings()
    extraction_settings.IMPUTE = impute  # Fill in Infs and NaNs
    X = extract_features(train, column_id='TS_ID', feature_extraction_settings=extraction_settings).values
    XT = extract_features(test, column_id='TS_ID', feature_extraction_settings=extraction_settings).values

    # there should be as example
    # model = xgb.DMatrix(X, label=Y, missing=np.nan)
    # model.fit()
    # model.predict(XT)

However, after line X = extract_features(...) I see at debugger following results

It's mean that initial X-dataset/features (shape=(722,10) were transformed to shape (80, 1899).

Where does '80' come from? I guess from train.TS_ID comes. But my XT-dataset still contains 722-rows (9 days * 80 different series per day).

So, how can I predict for 9 days in advance? or is there only forecast for next period?

TSFRESH is already supporting Time Series Forecast.
See details and example here and here


High level understanding of residual blocks I've been wondering what the residual blocks actually do and why they are a necessity for deep convolutional networks. Specifically, why are they so useful in some generator networks? To better extract features? I want an intuition for this concept, but can't find an understandable explanation online.

And what does the summation at the end of the block mean? It shows skipping around the block but in the end it adds it separately anyway? What does x and F(x) really represent in this context? Is F(x) an image feature of a given input and x is the input?

To me it looks like it's for feature detection/recognition because of the way it individually adds that separately calculated sub-results, but I could be wrong. And it doesn't entirely make sense to me this way either, because how does this particular method help to achieve that end? Is it a principle of reaffirmation - the net separately calculates some of the features and through these means knows for sure that a certain texture is what it is instead of something else (less mistakes everywhere)?
But then - why not have this for the discriminator as well? It fundamentally needs this same tech, doesn't it?

Is it just that the generator needs to improve more than the discriminator does? In other words - if the discriminator had these blocks it would start classifying pretty much only the exact replicas of the input image and the generator would never get fool the discriminator? But then - what importance would that be anyway - the generator would get just as good at image generation anyway.

Suppose we want to fit a function $f(x)$.  We can either try to learn a neural network model $F(\cdot)$ so that $F(x) \approx f(x)$.  Or, in the residual network approach, we try to learn a neural network model $R(\cdot)$ so that $x+R(x) \approx f(x)$.

Why is the latter easier to learn?  There's no fundamental reason why it should necessarily be so, in general.  It depends on the specific data set.  However, one possible intuition is that we might expect that in some settings, $f(\cdot)$ might be approximately linear: i.e., as a first-order approximation, $f(x) \approx x$ might be a reasonable first-order approximation.  Then we want to model the error term: i.e., suppose $f(x)=x+r(x)$; then we want to model $r(x)$.    In some settings, the error term $r(x)$ might be simpler to model or smaller than $f(x)$.  In that case, a residual network architecture might work better.

If you like, you can think of this as being akin to a Taylor-series approximation.  The Taylor series for $f(x)$ is

$$f(x) = c_0 + c_1 x + c_2 x^2 + \cdots$$

Suppose $x$ is small (i.e., $|x| \ll 1$).  Then $f(x) = c_0$ is a zero-th order approximation; $f(x) = c_0 + c_1 x$ is a first-order approximation; and so on.  At each step, we expect that the residual/error term is probably smaller than the approximation.  You can think of a residual network as learning a Taylor series for $f(x)$, in the special case where $c_0 = 0$ and $c_1 = 1$.

There are other more sophisticated explanations / intuitions, but hopefully this gives one possible perspective.


Does produced data fall under the "data collection" category? I am not sure whether data governance and policy is covered by this community. I'm giving it a try and let me know if I need to adjust.

While surveying all the data of my current employer, and developing a data governance policy, I am able to categorize two different sets of data on our customers:

data we collect on customers
data we produce about customers

Example in the context of my organisation (the data producer and data collector) being a school:

Collected Data: the student's parent's names and date of birth
Produced Data: the student's grades.

However, from the general definition of data collection, it seems that the data produced also falls under the "data collection" term. Is there a generally accepted term to dissociate these two categories?

There are probably many ways to think about this, but in the statistical world view I think the word you are looking for is dependent variable.

Statistician see students (or any other object for that matter) as a stochastic process that produce outcomes (dependent variables) given traits that we can observe (independent variables).

Data scientists tend to call those things target outcomes and features respectively.

None of these terms however mean anything about who owns the data. Philosophically a dependent variable is produced by the stochastic process (the outcome is governed by the givens), and are in a sense a characteristic of the process. Policywise this doesn't carry much weight though. GDPR for instance is mainly concerned about data that can be used to identify persons. That isn't likely the case with outcome variables, but it can be!


Welcome to the site, @intellion! I can recommend a couple of things for you. First, if you think you'll continue your interest in data science and will be working on problems like this in the future, it is worthwhile to start familiarizing yourself with the field. There are a variety of introductory textbooks and courses you can choose from. If you have a strong mathematical background, I'd recommend Tibshirani's "The Elements of Statistical Learning", which is a solid classic in the field. If you're approaching this from a more application-oriented angle, I'd recommend something written specifically for the language you'd like to work in. If you plan on using R, there's "Machine Learning with R"; if you plan on using python, there's "Machine Learning in Python." There are of course a variety of free online resources as well.

To your question, you could certainly solve your problem with Support Vector Machines (SVMs) or Neural Networks, but I don't think that's the first place to start. To me, this looks like a classic multiple regression application situation (for an overview for doing this in R, see this or this textbook). If you really want to try something in machine learning, perhaps decision trees are a good place to start, as they'll allow you to easily assess the influence of variables on the final model fit (e.g., using Gini Impurity). I've found the rpart library in R to be quite nice to work with, but it can be a little slow for larger data. You could also use sklearn in python, if that's your preferred language.


What NN architecture to predict fantasy character names based on description? I would like to build a neural network to predict a fantasy character name given a description.

Like 'Scar-faced long haired elf warrior' -> 'Glorfindel'

I have a dataset of about 12,000 fantasy names and description from various fantasy works. I  want to be able to map the description to names. Names are not vocabulary words and I want to NN to be able to generate new names for new description.

I wanted to use something like Elmo to embed the description and the name which would then easily teach the NN to map one to another, but the problem I faced is how do I go back from an embedding vector to characters representing a word.

First off, I think that since the goal of your model will be to generate new names based on a description, your model should work at a character-level and not word-level.

You can think of the level at which your model is working as the building blocks you are providing for it (it needs to learn them during training). These building blocks are than used for generation of new constructs. So if you want to construct new words (names) than you need to teach the model to understand the connection between the individual characters and the input description. Your model can deal with the input at a word-level but its output needs to be at character-level.

You can read more about it at: Besides Word Embedding, why you need to know Character Embedding?


What are the performance measures in the neural networks field? I constructed a neural networks in R using neuralnet package.

I want to test that using cross-validation, that is a technique based on using 4/5 of the dataset to train the network and the fifth one as the test set.

I wonder about what measures I should use to measure the neural networks performance in terms of predictability.

Could you suggest what measures are commonly used in the field and explain me why?

Any hint and ideas about that will be appreciated.

Typical predictive performance measures used to compare accuracy of ANN models are:

RMSE - (root mean squared error) measures the distance between estimated and actual outcomes. Other metrics that measure the same concept are MSE, MAE or MPE.
R square - (R^2 or coefficient of determination) measures the reduction of variance when using the model.

When comparing two different ANN models for performance, metrics that take into account the complexity of the model may be used, such as AIC or BIC.


Alternatives to regression to decide weights in an expression I have a use case in which I am required to predict variable $y$ which depends on five variables $x_i$. Consider something like

$$ y=w_1 x_1+ w_2 x_2+ w_3 x_3+ w_4 x_4+ w_5 x_5.$$

This expression doesn't necessarily  need to be linear. I applied regression (linear, ridge, lasso, ARD) but I am not getting any good results for example, the original $y= 200$ but the predicted $y= 1$ or $2$.

Is there any other ML approach to train a model which takes as an input five variables and predict a new 6th variable?

First, consider normalising your data, inputs and outputs. If it does not do the desired behaviour, you can have different options.

A simple solution is to add some higher order polynomials to the current features for each input pattern and use the simple linear regression. The fact about that is this that you can not be sure what features you would need due to the fact that you can not see your data. Consequently, if you don't find the acceptable answers you want you can use the current features the higher order polynomials that you are going to add to train an MLP.


Time series clustering using dynamic time warping and agglomerative clustering I'm new to data science and I'm currently working on a project to classify electricity consumption profiles.
This consists of electricity meter readings taken from sites on a half-hourly interval over a year (17520 readings in total). I am using Python to analyse the data.
I have settled on the current process of z-normalising the data, applying dynamic time warping on the entire dataset (using DTAIDistance) and then using agglomerative clustering as suggested by answers to similar questions on the site. I have produced the dendrogram below using 'complete' linkage using scipy.

I have some questions regarding the process.

What is the "best"* metric to determine the quality of clustering
(i.e. where do I set the cut off on my dendrogram?) and is there any
iterative method to determine the "optimal" cut off? (* I am aware
that "best" is quite subjective for this type of clustering)
Following up on 1. - say if I want to produce an elbow plot from my
DTW distance matrix. How would I go about doing this in Python?
How can I find out if I have selected the correct linkage method? I
have discounted Ward linkage as it apparently relies on an Euclidean
distance matrix and have plotted dendrogram using other linkage
methods, but I am unsure as to assess their suitability for my
dataset.
I have read that z-normalising data is required for most
applications of DTW - when is z-normalising not required for
DTW?
Are any other clustering methods I should consider?

Visually, there are no clusters in your data.

If your data had clusters, there would be subtrees with a small MRCA, with a long line leading to it.

There are metrics to quantity this, but you are not there yet. Can you show the time series? (or better, send them to me)

You may be lucky, and have one of ten problems that are easy to fix. See this paper.

You ask " when is z-normalising not required for DTW?" The answer is never.
eamonn


Merging large CSV files in pandas I have two CSV files (each of the file size is in GBs) which I am trying to merge, but every time I do that, my computer hangs. Is there no way to merge them in chunks in pandas itself?

No, there is not. You will have to use an alternative tool like dask, drill, spark, or a good old fashioned relational database.


Combining neural networks with different variables My organisation provides consultation to other firms, in part by making use of neural networks trained with extensive datasets that we have collected over the years.

Whenever available, we would inquire for any similar data our clients may have so that we can tailor a new model based on a checkpoint of our own. This has worked well as the data we used for training is relatively generalised and often collected by organisations in general.

However, recently one of our clients provided data that is highly predictive of the same response variable in our own models, but differs wildly in variable types and size.

An example in order to simplify:

I trained my own network to predict height based on weight using 100,000 records. Oftentimes my clients provide an additional 10,000 records of their employees height & weight which allows me to tailor the network to their needs. A new client, however, has provided me with 10,000 records of their employees' gender, country of origin, and daily dairy consumption. All of which, when trained in a separate network, prove highly predictive of height as well.

How do I effectively combine these separate models into a single classifier or regressor?

This process is called fine tuning frequently in the neural network literature. If you already have the trained network, you simply run additional training steps on the new data.

You'll need to balance the weight of the prior training with that of the new observations to some degree, and that decision is parameterized by the number of free variables in the network, the learning rate of the new training steps, and the number of new training steps.

However, your particular case is very complicated by the presence of new variables. Maintaining the form of the prior network is difficult in this case. You may want to simply ensemble a newly trained network on the new data and variables with the old network to combine the predictions.The simplest ensemble (but maybe not the best) would be averaging the results of the two models.


Confusion-matrix clarification from Python confusion_matrix(y_test1, pred)
That is my codes

Confusion matrix, without normalization
True     [[724258    438]
value    [ 25396    302]]
          Predicted value

I understand that is how this is the confusion matrix. But I do not know the order of classification result.

0   [[724258    438]                1 [[724258    438]
1   [ 25396    302]]                0 [ 25396    302]]
       0         1                        1        0

Which order generally? I check the documentation, it does not tell me the specific result.

it's the first one

0   [[724258    438]
1   [ 25396    302]]
       0         1

According to the documentation

sklearn.metrics.confusion_matrix(y_true, y_pred, labels=None, sample_weight=None)

  labels : array, shape = [n_classes], optional List of labels to index
  the matrix. This may be used to reorder or select a subset of labels.
  If none is given, those that appear at least once in y_true or y_pred
  are used in sorted order.


Does the length of the text data affects the score of clustering I'm learning about clustering with Pythons Scikit-Learn lib.
I have a list of sentences (strings) . Im wondering, does the length of the string affects the silhouette_score.

For example, I have sentences from 2 words to 35 words, and i have tried number of clusters from 2 to 60, and the biggest silhouette_score that I get is around 7. Can that affect the silhouette_score?
Is it better to filter my data, so that I can select sentences that are much closer by number of words, for example, to set number of words from 20-25, or 5-10?

This is how my code looks like:

list_of_comments = data

#cv = TfidfVectorizer(analyzer = 'word', max_features = 6500, lowercase=True, preprocessor=None, tokenizer=None, stop_words = 'english')
cv = CountVectorizer(analyzer = 'word', max_features = 8000, lowercase=True, preprocessor=None, tokenizer=None, stop_words = 'english')

x = cv.fit_transform(list_of_comments)

my_list = []
list_of_clusters = []
for i in range(2,35):

    kmeans = KMeans(n_clusters = i, init = 'k-means++', random_state = 0)
    kmeans.fit(x)
    my_list.append(kmeans.inertia_)

    cluster_labels = kmeans.fit_predict(x)

    silhouette_avg = silhouette_score(x, cluster_labels)*100
    print(round(silhouette_avg,2))
    list_of_clusters.append(silhouette_avg)

Yes, the length is very likely to influence any kind of similarity score. In general:

String containing very few words will have their highest similarity scores with other short strings with which they share one or two words in common. However if the words they contain are not common at all, they will have a lot of zero similarity scores (possibly only zeros).
Long strings (i.e. with many words) usually have low similarity scores, simply because they are unlikely to have a high proportion of their words in common. However compared to short strings they rarely have no word in common so they are much less likely to have lots of zero similarity scores.

As a result strings of similar length may tend to cluster together, depending on the data.


how to update column in data frame based on condition How to update column IsLCap column in dataframe based on Lvalue column whether it is capitilized or not.

df.loc[df.Lvalue.istitle(), 'IsLCap'] = 1  # need to be corrected

getting an error -AttributeError: 'Series' object has no attribute 'istitle'

You access string functions with .str. This should work:

df.loc[df.Lvalue.str.istitle(), 'IsLCap'] = 1

Update
To check the length of the strings in the column you can use the string method .str.len:

df.loc[df.Lvalue.str.len() < 3, 'less_than_three'] = 1

Pandas Series string methods


What to do this type of data I have plots of feature vs target values as

and

And many more. I have used Tree based models and able to get high accuracy but not able to improve further. Any suggestion ? almost every feature is like this.

Since you have more than one target value for a specific value of the feature and because the Y values are have quite a large range I guess the nueral network finds it hard to converge. Try to normalize the target values so that it has zero mean and unit variance and then try to use any of the regression models like logistic regression, A CNN with a squared error to get a prediction. After that plot the ROC-AUC to determine the dominant regressor.


How to save a Numpy array output of an autoencoder as an image I have a 256*256*3 numpy array "SP" out of an autoencoder decoder layer which I want to save and open as an .jpg image. I used something like the following python code snippets:

img = Image.fromarray(SP, 'RGB')
img.save('my.jpg')
img.show()

However I have noticed the array "img" is 256*256 in dimension and the image is just a noise. What can be the right way to display the image? I have attached the array as a output.npy file: ---> https://ufile.io/410iu

That is right. The size you see is the Frame Size of the image i.e. height and width. It does not refer to the dimensionality of a color image array. To see it try this

rgb = np.zeros((255, 255, 3), dtype=np.uint8)
img = Image.fromarray(rgb, 'RGB')
r = img.getchannel("R")
g = img.getchannel("G")
b = img.getchannel("B")
print(np.array(r.getdata()))
print(np.array(g.getdata()))
print(np.array(b.getdata()))

where output is

[  0   1   2 ... 252 253 254]
[55 55 55 ... 55 55 55]
[  1   0 255 ...   5   4   3]

So you have 3 dimensions (or colors). And the point about noise is in dtype=np.uint8. Convert your array to this and it will work.

You could also simply try

from scipy.misc import imsave
imsave("file_name.jpg", SP)

It does the job.

Good Luck!


Predict upcoming coordinates Given sequences of points that define similar lines amongst each others, e.g:

Where the blue lines are similar to each other, given the red point sequence and assuming that it will follow the same behaviour as the blue ones, I want to be able to determine what are the next red points.

Therefore my question is three fold: i) Is it possible to use machine learning approaches to address this problem? ii) Should I use machine learning approaches to address this problem? iii) Which known machine learning approaches are most suitable to address this problem?

Thank you for your time.

An RNN might work for this :

Lines have a "sequence of points"
Network should "learn" that points tend to shift towards left (as Y increases) and should be able to apply similar shift for prediction of X for a given Y

An example : Prediction of points for a Sin Wave


Predicting time series data I have a dataset as following:

This is test case 1. My goal is to fill the missing years data. As the age sex and smoking is not changing so I have to predict the condition and percent data for year 0 to all the way 54. I found high correlation between condition and percent variable. This seems easy. But I am a bit confused now. Should I have to use multivariable regression? what would be the most best method to approach?

Best approach would be to perform data preparation first:

Remove features (columns) with no variance in it (you could use: sklearn feature_selection)
one-hot-encoding of categorical features
insert a lag column of -t steps

If you have more than one explanatory variable, the process is called multiple linear regression. Instead of using a regression model you could also use other learners like XGBoost or LSTMs


Tips on how to preprocess data and outliers for churn analysis I am doing analysis on telecom churn dataset. I have 4617 observations and 17 variables. I am using Python. I have the following questions,

1) When I skewness and kurtosis for the normality test, two variables are not normally distributed (value exceeds 1). Should I do log transformation for those two variables or else for the entire dataset?

2) When I check the outliers using IQR method, close to 700 observations are outliers. I do not want to remove the outliers. Should I apply log transformation as well to address the outliers? Is it the correct way?

3) I checked google, they say you can cap the outliers by taking percentile values. Is it a good practice to address the outliers?

4) My final aim is to apply all classification algorithms. Can I do scaling after log transformation? Can we need to do scaling after log transformation? Tree models are ok without outliers but want to do for other models. My question is can we build the model after log transformation or we need to do scaling?

Please advise.

1) when the variable follows the non normal distribution you can use different transformation to make it normal, if variable follows the skewed distribution(either positive or negative) you can use log transformation to make it normal.

2) and 3) Instead of removing the outlier always go through outlier capping method, it will help to improve the performance of the model.

4) In machine learning some algorithm need to scale the data before go to modeling. for ex. cluster Analysis, Principal component Analysis(PCA). we can use scaling when the some values of the variable is too high as compare to the values of the other variable. (1,2,3,56,900,100,34,22,9)
while using such type of algorithm always go to scaling instead of transformation.


Why Continous Variable Buckets Overfitting model I have a continuous (high cardinal discrete) variable 'numInteractionPoints' in my dataset during training model - I binned this feature in order to avoid overffing , first top bar chart is from training and second bar chart is from testing for this binned feature. Both datasets has same distribution as per following bar graph

this is how i created bin for this feature
i converted variable 'numInteractionPoints' into bins using simple bin method based on visual analysis
bins = [0, 6, 12, np.inf]
names = ['0-6', '6-12','12+']

train_data['numInteractionPointsRange'] = pd.cut(train_data['numInteractionPoints'], bins, labels=names)
train_data['numInteractionPointsRange']=train_data['numInteractionPointsRange'].astype('object')

in both cases when i include raw variable numInteractionPoints without bin and with above defined bucket the overall F1 score of model decrease on test data from 0.47 to 0.33 for a class 1. On validation dataset it is giving 0.47 while on test it gives 0.33 with or without bins

Any idea please what i am missing? This behavior is in both algo Xgboost and catboost, target class is highly imbalanced and i am controlling it Xgboost parameter scale_pos_weight. Why model overfitting due to defined buckets , as you can see above bar chart distribution of both dataset are same

When you use buckets you lose some information. Basically you assume the relationship between the variable and the target is flat within the interval. When this is probably not the case. This is point #3 on Frank Harell's list of reasons why you generally shouldn't categorise at all.
More specifically, 80% of your instances seems to be in one bucket. That mean your algo can't learn what happen inside this bucket. As a starting point you may use more buckets to check if that was the problem. More convoluted techniques would include splines to smooth the response (removing noise help not learning from it).  Honestly given that you work with a tree like algo (Xgboost) you should probably let it choose optimal splits instead of enforcing some splits trough variable categorisation. That is : don't use a priori variable categorisation at all, enforce the absence of overfitting trough other parameters (tree depht at least) and if you really need category use those found by the algo.
Edit: re-reading your question it seems that you are overfitting in both case... the solution might be more about tuning your xgboost parameters.


PhD program in statistics I am a first year PhD student in statistics. During this year I have analyzed the scopes of interest of my scientific advisor and found them unpromising. He is majored in mixtures with varying concentrations models for which I have not found any references to authoritative sources.

Now I want to change my PhD theme, however, there are no other scientific advisors in my university which majored in statistics. Therefore, I have 2 questions:

Is it possible to write at least 5 articles together with a PhD thesis without scientific adviser? If yes, what is a proper way to do this? Here I mean how to choose a theme, where ask for a help and so on.
Is it possible to find a remote adviser to consult with? If yes, how and where can I find him?

Also I have no much time for the search. I am interested in statistics, especially in machine learning. I would like my PhD thesis to be of practical value, not pure research one which is popular in my department. Also I have commercial experience in programming (C/C++, R, Python) if that can help.

Thanks in advance for any help!

Certain ingredients are needed to give you the best chance of a successful PhD. One of the important ones is that you and your supervisor have mutual interests.

A second important ingredient, in my opinion, is that you immerse yourself in that environment. It's important to develop a network of colleagues. It helps to spread ideas, start collaborations, get help when needed, and to explore unthought of opportunities.

From what you have said, I think you will be missing out on these two important ingredients if you continue in the same place or if you work remotely.

What is also important is what you to do after the PhD. PhD is required for academic position. But I think you will be in a weak position to get to the next step (fellowships, faculty positions, etc.) if you do what you proposed. In certain industrial positions it can be looked on favourably, not necessarily for the topic you pursued, but because it says something about your personally. Basically that you can get things gone, rise to a challenge, work independently, work as a team, communicate difficult topics and can bring creativity to solving a problems.

My advice would be to find a machine learning research group and apply for PhDs. If this is not possible why not consider following the topic of your supervisor and keep machine learning as a hobby? You will become and expert in statistic and so you will find manny concepts will translate between the various statistical disciplines. But only do this if you get along with her/him, and you can see yourself studying this topic.

Finally you could try a compromise? Are there applications for "mixing statistics" in machine learning? Can you find one? Is there an unexplored opportunity to do something new?

As a side note I find it ridiculous that PhD supervisor ask the student for topics. This always leads to problems because the student doesn't really have a clue about the research field. There is room for flexibility but often this hides supervisor laziness.


NLP - Identify Tagged Words Please pardon me as the title might not be very accurate

I am trying to create a model that learns the word representation and then is able to predict word representation in another piece of text. An example will make it more clear. Please see below for the example:

Model Training Text:

I lived in *Munich last summer. *Germany has a relaxing, slow summer lifestyle. One night, I got food poisoning and couldn't find !Tylenol to make the pain go away, they insisted I take !aspirin instead.

Model Predicts:

['Munich','Germany','Tylenol','aspirin']

Evaluation Text:

When I lived in Paris last year, France was experiencing a recession. The nightlife was too fun, I developed an addiction to Adderall and Ritalin.

Output:

['Paris','France','Adderall','Ritalin']

The question is that what sort of NLP technique will be helpful in such a case. I don't even know what are these kind of problems called. Can you please tell what are these problems called?

One approach I can think of is to train the RNN with Embedding Layer to predict the position of * and ! since * will be prefixed to the name of the country and ! will be prefixed to the name of the drug but my challenge is that how can I structure my data for such training. Is it a feasible approach?

Is there any resource/material I can refer to and draw the inspiration from?

I would very much appreciate any help or suggestions. Thanks a lot in advance.

I think the aspect of NLP you're looking for is Named Entity Recognition (NER). It's already built into Python libraries like NLTK and SpaCy. Some videos that might be of interest:

How to train a new 'DRUGS' entity type: https://prodi.gy/docs/video-new-entity-type
Practical example: https://towardsdatascience.com/named-entity-recognition-with-nltk-and-spacy-8c4a7d88e7da

The great thing for you is that out-of-the-box models are probably going to be extremely accurate in detecting place names (either LOC or GPE entity tags). They may even be able to detect drugs as a 'PRODUCT' tag. If you need to be more specific, the first video shows you how you would go about training a model for a new tag. For instance, in spaCy:

import spacy
nlp = spacy.load("en_core_web_lg")
doc = nlp("""When I lived in Paris last year, France was experiencing arecession. The nightlife was too fun, I developed an addiction to Adderall and Ritalin.""")
print([(e.text, e.label_) for e in doc.ents])

will print something like this:
[('Paris', 'GPE'), ('last year', 'DATE'), ('France', 'GPE'), ('Adderall', 'PRODUCT'), ('Ritalin', 'PRODUCT')]


Predict datetime variable with orange canvas Since Orange is able to read datetime variables, i tried to build a simple workflow to predict datetime variables via regression. Loading the data etc. is no problem, but when it comes to predict the data, the output of Predictions Widget (in this case with AdaBoost Regression) is something like:

1450396800.000;
1438905600.000;
1438732800.000;

whereas the original datetime is:

2015-12-22;
2015-08-07;
2015-08-05;

What does that mean? Did i make any mistakes? Is there any other way to predict datetime variables in orange?

The workflow does its job for any "common" continuous variable.

It appears those numbers are Unix timestamps. The numbers you quote correspond to the following human-readable dates in ISO format:

Epoch date  Human readable date (GMT)
1450396800  2015-12-18 00:00:00
1438905600  2015-08-07 00:00:00
1438732800  2015-08-05 00:00:00

You can use an online epoch converter to confirm.

Apparently, Predictions widget treats datetime variables as continuous variables (which is ok) but fails to present them nicely. You should probably submit a bug report about it.


to include first single word in bigram or not? in a text such as

  "The deal with Canada's Barrick Gold finalised in Toronto over the
  weekend"

When I try to break it into bigram model, I get this

"The deal"
"deal with"
"with Canada's"
"Canada's Barrick"
"Barrick Gold"
"Gold finalised"
"finalised in"
"in Toronto"
"Toronto over"
"over the"
"the weekend"

my question

shall I include the first word and the last word as single words

"* The"
"Weekend *"

What you describe is called padding and is indeed used frequently in language modeling. For instance if one represents the sequence "A B C" with trigrams:

# # A
# A B
A B C
B C #
C # #

The advantages of padding:

it makes every word/symbol appear the same number of times whether it appears in the middle of the sequence or not.
it marks the beginning and end of a sentence/text, so that the model can represent the probability to start/end with a particular word.


What's a good method for combining data of different deep learning models? Suppose I want to predict the probability of a person to buy something. I want to analyze the person image and I can use a convolutional neural network, but I also want to input in my predictive model how many times he bought something, where he goes more frequently and so on. What is a correct way to input in a deep learning model informations from different deep learning models?

Actually you can merge two different DL model into a big one.

I'm using Keras+Tensorflow for deep learning, and there's an example about models with multiple inputs(outputs). In this example, tweets are fed into a RNN, while the extra data are fed into a fully-connected layer along with the output of this RNN.


Performance metrics changing significantly based on batch size I am working on a binary classification problem where there is significant class imbalance (minority class makes up nearly 10%). The dataset has ~15,000 observations and I have split this in to a training, validation and test set (that are stratified).
Using PyTorch I build a neural network with 5 fully connected layers (using ReLU activation), CrossEntropyLoss and SGD optimiser. Below is parts of my code
The problem is that my training vs validation loss changes a lot based on batch size (passed in the DataLoader). If I use a batch size of 64, the loss functions look like

which is quite odd. But if I use an unconventionally large batch size of say 1000, it looks like:
This looks more familiar but I can't make sense of what is going wrong here. I am also seeing that the training set reaches a high recall fairly quickly (after ~4 epochs) while the validation set improves slowly. So there seems to be an issue of overfitting as well.
I don't really know where I am going wrong: my neural network architecture consists of 5 fully connected layers with appropriate input and output dimensions. I initialise the weights. The forward function applies ReLU to the inputs (I don't use Softmax because I only need to classify 0 or 1 so I thought I can simply use argmax, see the 'c' variable in the code above).
I have tried setting Shuffle to true in the training_loader but this produces highly fluctuating training loss values.

Batch size is very related to the learning rate, especially in non-adaptive optimizers like the vanilla SGD that you are using.
I would suggest two alternatives:

Tune (reduce) the learning rate. You can check the answers to this SO question for some heuristics on how to choose the value. Apart from that, I recommend the article Don't Decay the Learning Rate, Increase the Batch Size.

Use an adaptive optimizer, like Adam.


Naive Bayes / SVM classifiation - min. number of records (Python) I am doing text classification with Python. I have around 120 records with 2 columns:

text
class

I tokenize, stem and lematize the words, I also did some of my own text preprocessing. When I run the alghoritms using sklearn and divide it to training and test sets each time I run the script the Accuracy Score is so different each time for both alghoritms. Sometimes I get around 70%, sometimes 40%. Is it because of the number of records (120) or not necessarily? if it is about number of records how much of them should I have?

My suspicion is that you did not put random_state parameter on train_test_split. Don't worry it is quite common error, it just happen that your dataset is small and often with small dataset variability between result will be heavily amplified. Having small amount of data is OK, and sometimes we have to work with what we have, as long as you know how to handle it, just don't expect your model to generalize very well.

Your model relies on bag of words representation I presume. With small data what could go wrong is having a lot of Out of Vocabulary words (words in test-set does not available in training set), to handle this you can apply smoothing e.g. Add-one smoothing.


Effect of Skewness and data range in machine learning I have a feature for machine learning as follow that skew to the left, and only have number in certain number range (here 0-2000). Will skewness and range of number affect the learning? If yes what should I do?

Typically, folks would transform the variable. When it is strictly greater than zero, a log transform is usually sufficient. If zero is included, as in your case, one popular alternative is the box-cox transformation.


What is the best way to scale a numerical dataset I have a dataset with differents attributes which don't have the same range on their values which is a problem when we need to compute distance beetween objects. After some research i found that i can do the regularisation job with this formula : (value-min)/(max-min)  where min and max are respectively the minimum and maximum value in the domain of val attribute.
The question is that one, does it exist other ways ?

Thank you for your help.

There is pretty much mess in terminology in your question :).
Data Regularization is used for model selection, it is not about data processing. Here it is described in more friendly manner.

What you mean is Feature Scaling. It can be done in several ways including Rescaling, the method you described. You may also use Standardization (normalization) and Scaling to unit length.

These answers may be helpful:

Normalization vs Scaling
Normalization vs Standardization


How to treat Compass data in random forest regression I'm working on a project where two of the features are entryHeading and exitHeading. Both state the direction (N, NE, E, SE, S, SW, W) of a vehicle at multiple points. My question is how would i go about pre-processing this? My first thought would be to circularize it like I would a 24 hour period but I'm not sure I should go about it in the same way. The data will eventually be used to train a Random forest regression

One option is to map the ordinal categories to numerical degrees. This would allow the machine learning model to learn the relative relationships. The machine learning model might interpolate values which may or may not be useful.

One common mapping is:

n  ->   0
ne ->  45
e  ->  90
se -> 135
s  -> 180
sw -> 215
w  -> 270

The book "Statistical Analysis of Circular Data" by Fisher goes into greater detail about more options.


Using Neural Networks To Predict Sets I'm building a neural network for data analysis, however I'm stuck on how many output neurons I need and what they should represent. The neural network tries to predict peoples choices for certain objects. There are 75 possible objects, but here's the catch, they choose 6 of them at a time. These six can (theoretically, though this won't happen realistically) be any combination of the 75 objects, in rare cases there might even be duplicates of a certain object in this set of six.

I naturally considered creating an output neuron for each possible set, but since that would lead to 75^6th output neurons (or more if we consider duplicates) I imagine that that would have disadvantages for the learning speed. Another option I considered is just taking the six highest ratest items as a set, but I'm unsure if this would work since the choice of the second, third, etc. objects depend on which earlier objects were chosen. I wonder if there are any better, faster or more accurate ways of doing this?

Edit, some more information about the trainingdata:
The neural network will be used to analyse the ingame item choices in the computergame called league of legends. There players can choose up to six items each game. These six items are most likely based on some factors that change each game such as the ingame characters they and their enemies play, but also things such as what the other 5 items are they have. However, the reason for this may be universal to all games. For instance, the (likely) biggest reason why items depend on eachother is the multiplicativity of item stats, if you already have an item that makes your attacks hit harder then getting an item that allows you to attack more often is going to do more for you than getting an item that makes your attacks hit even harder. There are also some links between item choices though, while probably more limited, there is for instance an item that returns damage to the enemy when he attacks you, while it's not detrimental it is probably not a great idea to combine that with an item that lowers the amount of attacks that an enemy can do to you.

The 75^6 option is not only bad for speed, but it is a very difficult representation to train, because the NN doesn't "understand" that any of the output categories are related. You would need an immense amount of data to train such a network, because ideally you need at least a few examples in any category that you expect the network to predict. Unless you had literally billions of examples to train from, the chances are certain combinations will never occur in your training set, thus could never be predicted with any confidence.

Therefore I would probably use 75 outputs, one for each object representing the probability that it would be chosen. This is easy to create training data for, if you have training examples with the 6 favoured objects - just a 1 for the objects chosen and 0 for all others as a 75-wide label.

For prediction, select the 6 objects with the highest probabilities. If these choices are part of a recommender system (i.e. may be presented to same person as being predicted for), then you can select items randomly using the outputs as weights. You may even find that this weighted Monte Carlo selection works well for predicting bulk user behaviour as well (e.g. for predictions fed into stock purchases). In addition, this stochastic approach can be made to predict duplicates (but not accurately, except perhaps averaged over many predictions).

A sigmoid transfer function on the output layer is good for representing non-exclusive probability. The logloss objective function can be used to generate the error values and train the network.

If you want to accurately predict duplicate choices out of the 6 items chosen, then you will need plenty of examples where duplicates happened and have some way to represent that in the output layer. For example, you could have double the number of output neurons, with two assigned to each object. The first probability would then be probability of selecting the item once, and the second probability would be for selecting it twice.

The question has since been updated, and it appears there are strong relationships between items making the choice of a set of items potentially very recipe-like. That may reduce the effectiveness of the ideas outlined above in this answer.

However, using 75 outputs may still work better than other approaches, and is maybe the simplest setup, so I suggest still giving it a try, even if just to establish a benchmark for other ideas. This will work best when decisions are driven heavily by the feature data available, and when in practice there are lots of valid choices for combining items so there is a strong element of player preference. It will work less well if there is a large element of game mastery and logic in player decisions in order to combine items.


About different structures of neural network https://www.mathworks.com/help/deeplearning/ref/fitnet.html is the tutorial that I am following to understand fitting data to a function. I have few doubts regarding structure and terminologies which are the following:

1. Model & number of hidden layers

By hidden layer we mean the layer that is inbetween the input and output. If number of layers = 1 with 10 hidden neurons (as shown in second figure) then is it essentially a neural network which is termed as an MLP. Is my understanding correct? In general,

if the number of hidden layers = 0, we call the NN as a perceptron.
If the number of hidden layers >=1 but less than 3, the NN becomes an MLP. Is the picture in the link that of an MLP since it contains 1 hidden layer of 10 neurons?
if the number of hidden layers >3, the NN is called as deep NN aka deep learning

Is that correct?

2. Linear vs nonlinear mapping function

The resulting model eventually learn to map the input to output data.

Do we call the machine learning model as linear or nonlinear ? Or is
this term associated to the mapping function ?
Which layer's mapping function determines this? Based on which layer's activation function do we say that the mapping function or the model is linear or nonlinear? For ex, In this picture, the last layer is the output layer and the activation function looks like an identity/linear. But the hidden layer has sigmoid activation function which is nonlinear. Therefore, is this model a nonlinear function?

I will answer your questions one by one:

By hidden layer we mean the layer that is inbetween the input and output. If number of layers = 1 with 10 hidden neurons (as shown in second figure) then is it essentially a neural network which is termed as an MLP. Is my understanding correct?

The fundamental building block of a Neural Network is the perceptron. It's modeled on biological neurons: a unit that receives multiple inputs, weights them, and outputs a signal transformed. In its simplest form (i.e. a perceptron with sigmoid activation) it's practically identical to a logistic regression.
When you compose many perceptrons on more than one layer you will form an MLP (Multi Layer Perceptron). The term "MLP" is used as a synonim of Neural Networks - in their basic feed forward architectures.
Neural Networks with no hidden layer cannot even be considered as Neural Networks.
The ones with just one hidden layer are called shallow Neural Networks, no one really uses them, since they are not as powerful as Deep Neural Networks.
When you have more than one hidden layer, then you talk about Deep Neural Networks. They are the real big thing right now, the most powerful ML models available right now.

Do we call the machine learning model as linear or nonlinear ? Or is this term associated to the mapping function ?

ML models can be either linear or non-linear, the choice is yours. Different models will map input and output in different ways (linear or other non-linear ways).
I think it's important to stress that Machine Learning is not a set of models, but an approach to data. You could use a linear regression (i.e. a linear model) with a ML approach, or an SVM or a Deep Neural Network (i.e. non-linear models) to solve the exact same problem.

Which layer's mapping function determines this?

If you refer to Deep Neural Networks, I believe all layers altogether are responsible of the non-linear mapping between inputs and outputs. Non-linearities can be learned only thanks ot a combination of depth and non-linear activation functions. The deeper a Network is, the more complex will be the non-linear patterns that it will able to learn. It's like if you need "all of them at once" to do that, in my opinion.