how can I visualize the bag-of-words in a scatterplot I have a bag-of-words consisting of about 60000 features. Each feature represents a diminsion. I want to represent this bag-of-words in a reduced 2D space. How do I do it?

I have see an example here, which looks more like what I want but not really the same. In the example they have 2 transform and I only have one. Therefore as suggested I do not want to use pipeline. Below is my code which taking for ever and does not show any error message:

#myList contents about 800000 words
bag_of_words = vec.fit_transform(myList)
X = bag_of_words.todense()
pca = PCA(n_components=2).fit(X)
data2D = pca.transform(X)
plt.scatter(data2D[:,0], data2D[:,1])
plt.show()

I have not found any better option and right now it looks like I am doing something wrong.

What is the best way to visualize a bag-of-words in a scatterplot?

The bag_of_words looks like this:

(0, 548)    3
(0, 4000)   6
(0, 15346)  1
(0, 23299)  1
(0, 22931)  2
(0, 32817)  1
(0, 51733)  1
(0, 38308)  6
(0, 14784)  1
(0, 146873) 1
 ....

I think you should have a look at t-SNE which is a visualization technique based on dimensionality reduction. Wikipedia excerpt:

t-distributed stochastic neighbor embedding (t-SNE) is a machine learning algorithm for dimensionality reduction developed by Geoffrey Hinton and Laurens van der Maaten. It is a nonlinear dimensionality reduction technique that is particularly well-suited for embedding high-dimensional data into a space of two or three dimensions, which can then be visualized in a scatter plot. Specifically, it models each high-dimensional object by a two- or three-dimensional point in such a way that similar objects are modeled by nearby points and dissimilar objects are modeled by distant points.

This paper describes a python implementation of the technique on various word2vec datasets.


selecting sample from population using genetic algorithm I have a set of 31390 individuals with an associated weight (kg) and I want to select a number of them such that the maximum weight is 3000kgs and that the weight distribution around several variables describing the sample are close to some targets.

I was approaching the problem by considering each individual as a gene of a chromosome and then multiplying the chromosome by the weight of the sample. In this case 1 is for being included in the sample while 0 is being excluded from the sample.

I followed the example used in https://www.r-bloggers.com/genetic-algorithms-a-simple-r-example/ although my constraint is different as I want to minimize the difference between the weight distribution I get from the selected sample and my target.

Taking 1000 iterations and populations of 50 and 100 I don't get satisfactory solutions. I have read that people take populations that are (# genes)^2 but I will run out of memory if I take a population of that size.

I think crossover could help but it seems as if the genalg library only uses mutation.

Any suggestions on how to solve the problem? Thanks. I have posted the code below

Weight.Limit<-30*100

evalFunc <- function(x) {
  current_weight <- as.numeric(x %*% ind.data$weight)
  sample.dist<-x %*% as.matrix(ind.data[,4:ncol(ind.data)])/current_weight
  current_solution_dist <-  sum(sqrt(abs(constraints.data-sample.dist)))

  if (current_weight> Weight.Limit)
    return(0) else return(current_solution_dist)
}

GAmodel <- rbga.bin(size = 31390, popSize = PopSize, iters = iter, mutationChance = MutationChance, elitism = T, evalFunc = evalFunc)

cat(summary(GAmodel))

There's nothing super special about a population size of L^2. Larger populations are often better, but don't worry about trying to hit some threshold needed for the algorithm to work.

There are a couple of potential issues here that I see. I'm making some assumptions that when you say "individuals", you mean people. Which means an average person is going to contribute something on the order of 50-100 kg toward your limit, which puts you at like 50 people in the sample in order to stay under the 3000kg limit.

With your encoding scheme, that's going to mean only very sparse vectors are feasible solutions. You can deal with that, but it likely means a standard GA pulled from a library won't work as well as one that included some prior knowledge. For example, when initializing your population, you might want to bias things pretty heavily in favor of setting bits to 0 rather than 1.

In any case, you have constraints, so you'll want to include some type of constraint handling scheme. A simple thing you can do in this case is to perform a simple local search on every individual you generate. For example, suppose I have an individual i selected from my population. If the total weight of the sample encoded by i is less than 3000kg, iterate over every bit of i (in random order) and see if you can flip a 0 bit to a 1 bit without exceeding your weight capacity. Similarly, if you generate a sample that weighs more than 3000kg, try to find 1 bits you can flip to remove weight until you're under the limit. When you're done, you have a feasible individual and you insert it into the child population.

You can incorporate more sophisticated measures or search schemes as well. You could have your local search operator consider those other variables. You can replace the simple hill-climbing operator with a more sophisticated method like tabu search or simulated annealing. There's a spectrum of algorithms with all sorts of different ways of combining the ideas of the global search of a GA with more targeted local search operators, and there's no general "right answer". You mostly just need to experiment a bit to see what works.


Rule of thumb for good number of features when dealing with grouped data I have a classification problem on clinical data where I have multiple samples for each patient. So the samples related to the same patient are somehow dependent from each other.

I know that is not possible to know a priori the optimal number of features to use, but there are some rule of thumb that works in many cases.

My question is: are those rules valid also in my case? In particular, I should relate the number of features to the number of instances or to the number of groups?

Thanks

This a really hard question to answer. I recommend you do some reading to get a feeling on what can be done and particularly, what can be done for your particular task.

This paper is a must. But if you prefer a more practical approach have a look at these two interesting sources:

a) ML Mastery which also provides additional further readings

b) Kaggle

Good luck!


How do you calculate sample difference in terms of sensor signals? A paper I read called Preprocessing Techniques for Context Recognition from  Accelerometer Data refers to sample difference as the delta value between signals in a pairwise arrangement of samples that allows a basic comparison between the intensity of user activity.

How would you do the pairwise arrangement? Would it require you to have different files of data representing different classes?

For example, I have a CSV:

 1495573445.162, 0, 0.021973, 0.012283, -0.995468, 1
 1495573445.172, 0, 0.021072, 0.013779, -0.994308, 1
 1495573445.182, 0, 0.020157, 0.015717, -0.995575, 1
 1495573445.192, 0, 0.017883, 0.012756, -0.993927, 1

where the second, third, and fourth columns are the axes of accelerometer data.

I have several files named for one gesture and several others for another gesture and would like to use this sample difference statistic to help classify the data.

Also as a secondary question, this was listed as a preprocessing technique but it sounds like it's more of a feature. Could I get clarification on that as well?


How many dimensions should matrix be reduced to with PCA? I want to reduce the dimensions of my dataset to decrease the complexity without losing too much information.

But I am not sure how many dimensions I should reduce to, so that I don't lose too much information.

Do we have any direct attribute/function to use for such task?

Thanks in advance.

Being PCA the same as SVD, I'll explain the approach with the last one.

SVD decomposes a matrix $A$ in three others, so that $A = U * \Sigma * V^t$, where $\Sigma$ is diagonal consisting of $A$'s singular values in descending order.

The sum of all the singular values of $A$ is known as energy of matrix $A$. You want to retain most of the matrix's energy when reducing dimensions, so what you have to do is calculate how many top singular values you need to retain to preserve the most of the energy.

There are two ways of achieving this:

'Elbow method': plot the singular values and see from which on the energy isn't adding much to the total energy anymore.
calculate how many of the top singular values you need to preserve most of the energy. Usually you want to preserve $\simeq 90\%$ of the energy. So you have to find $m / \frac{\sum_{i=0}^{m} sv_i}{\sum_{i=0}^{n} sv_i} \simeq0.9$.

I recommend you expand my answer with the links below. I especially recommend the mentioned book.

SVD on Wikipedia: https://en.wikipedia.org/wiki/Singular_value_decomposition
Leskovec's book Mining of Massive Datasets, the chapter on Dimensionality reduction: http://infolab.stanford.edu/~ullman/mmds/ch11.pdf


Parameter estimation: reduce time I have a two-class prediction model; it has n configurable (numeric) parameters. The model can work pretty well if you tune those parameters properly, but the specific values for those parameters are hard to find. I used grid search for that (providing, say, m values for each parameter). This yields m ^ n times to learn, and it is very time-consuming even when run in parallel on a machine with 24 cores.

I tried fixing all parameters but one and changing this only one parameter (which yields m × n times), but it's not obvious for me what to do with the results I got. This is a sample plot of precision (triangles) and recall (dots) for negative (red) and positive (blue) samples:

Simply taking the "winner" values for each parameter obtained this way and combining them doesn't lead to best (or even good) prediction results. I thought about building regression on parameter sets with precision/recall as dependent variable, but I don't think that regression with more than 5 independent variables will be much faster than grid search scenario.

What would you propose to find good parameter values, but with reasonable estimation time?

If an exhaustive nonlinear scan is too expensive and a linear scan doesn't yield the best results then I suggest you try a stochastic nonlinear search i.e. a random search for hyperparameter optimization.

Scikit learn has a user friendly description in its user guide.

Here is a paper on random search for hyperparameter optimization.


What are CRF (Conditional Random Field) Looking for language modeling, I have been finding CRF in a lot of places which is but looking online for the same isn't actually helping me a lot. I referred Edwin Chen's blog and Ravish Chawala's Medium article but rather than solving my problem, raises more question.

Could you please refer some interesting blog posts, Github repo's or research papers that are going to further help me with this?

Stanford CoreNLP is a very good implementation of CRF (In Natural Language Processing domain).

https://nlp.stanford.edu/software/CRF-NER.html  . CRF specific implementation is : https://github.com/stanfordnlp/CoreNLP/blob/master/src/edu/stanford/nlp/ie/crf/CRFClassifier.java

Few other resources :

https://www.analyticsvidhya.com/blog/2018/08/nlp-guide-conditional-random-fields-text-classification/

https://github.com/lancifollia/crf

https://towardsdatascience.com/conditional-random-field-tutorial-in-pytorch-ca0d04499463


Do xgboost and random forests in general handle multiple splits of the same numeric feature in a single branch? For example, let's say that the age (say $x$) of a person for $12< x< 25$ can be used to predict computer usage to a high degree of certainty. In a decision tree, this could be represented by a split of $x<25$ followed by a split of $x>12$.

Does xgboost, or other decision tree learning algorithms in general, handle multiple splits within the same branch?

Yes, it can handle multiple splits withing the same branch.
A decision tree model can use the same feature as many times as optimally needed.
You can see an example in the sklearn doc.
The petal-length feature is used multiple times in the same branch.


correct value of output_dim or units parameter of dense I have started learning deep learning and using keras library. But I am confused as to how to take a proper estimate of the value to use for units parameter of the dense method. I came across this tip that we can take it as the average of the number of input nodes and output nodes but everywhere it says that it comes from experience. I want to know if there are things to look out for to estimate it wisely or any other things I need to know. Thanks for the help.

classifier.add(Dense(units=6, kernel_initializer="uniform", activation="relu", input_dim=11))

There is no known way to determine a good network structure evaluating the number of inputs or outputs. It relies on the number of training examples, batch size, number of epochs, basically, in every significant parameter of the network.

Moreover, a high number of units can introduce problems like overfitting and exploding gradient problems. On the other side, a lower number of units can cause a model to have high bias and low accuracy values. Once again, it depends on the size of data used for training.

Sadly it is trying some different values that give you the best adjustments. You may choose the combination that gives you the lowest loss and validation loss values, as well as the best accuracy for your dataset.

You could do some proportion on your number of units values, something like:

# Build the model
model = Sequential()
model.add(Dense(num_classes * 8, input_shape=(shape_value,),  activation = 'relu' ))
model.add(Dropout(0.5))

model.add(Dense(num_classes * 4, activation = 'relu'))
model.add(Dropout(0.2))

model.add(Dense(num_classes * 2, activation = 'relu'))
model.add(Dropout(0.2))

#Output layer
model.add(Dense(num_classes, activation = 'softmax'))

The model above shows an example of a categorisation AI system. The num_classes are the number of different categories the system has to choose. For instance, in the iris dataset from Keras, we have:

Iris Setosa
Iris Versicolour
Iris Virginica

num_classes = 3

However, this could lead to worse results than with other random values. We need to tune the parameters to the training dataset by trial and error.


Score for Ranked plots using scatter plot widget I'm using orange 3 and when I Score plots using the scatter plot widget how do I view the numerical score associated with the ordered list of plots. Did that feature get removed with the upgrade from 2.7 to 3?

Thanks,

Hm, what was the meaning of this score?

See. This is why it was removed.

(As I recall, it was the average probability assigned to the correct class by the k-nearest neighbour classifier on the projection -- or something similar. The number is useful for ranking projections, but doesn't have any meaningfyl interpretation or absolute scale, like >0.70 is good, >0.90 is excellent. Hence there's no point in showing it.)


Are neural networks able to deal with non-normalised inputs? All the techniques/models that I have learnt so far for deep learning start with some sort of normalization to the features, for example gaussian method, minmax scaling, robust scaling, batch normalization, instance normalization.

Are there any techniques to run neural networks without normalization so that the network can see (in absolute values) the magnitude of the value and respond according to that instead of normalized values? Will there be exploding/vanishing gradient issues if I don't normalize my data?

For example, if I am training a custom LSTM network for multivariate time series data, the input dimension for a feature vector $x$ is all the values from $t-n$ to $t$, where $n$ is the number of time steps and the output vector is the value at $t+1$. Is there any need for normalization in this case?

Normalization helps to eliminate scale factors that might exist between variables in your data. Take, for example, the classic problem of predicting home prices. If you represent the square footage of your home in square millimeters, a large change in this value will have a relatively small effect on home price, implying a small gradient on this variable. If you represent that value in square kilometers, a small numerical change will have a large impact on price, implying a large gradient. Normalization isn't necessarily required, but can help to balance the problem by making all variables have "equal weight" in your model. If you were to include both the square millimeter and square kilometer variables in your training data, the neural network would likely spend a lot of effort optimizing on the square kilometer variable, since it is numerically more important. You can still do training with un-normalized data, but it will likely take longer, and possibly have worse output if your important variables are numerically smallest.


Meaning of curved line shape distribution in t-SNE plot I have blood test results with 20 features from 170 patients, and trying to predict a categorical disease outcome.
The input features are all continuous values except sex (0:male, 1:female)
There are no missing values (used imputation for the few missing values)

When I plotted a 2D t-SNE plot, the results is as the attached image.(Figure 1)
A single curved line shape.
I expected the plot to be more scattered, but even if I increase the perplexity value from 13 to 20 or 30 it still remains as a line shape.
(the initial value of 13 is from a recommendation that sqrt # of samples is a good starting point)

Reducing the perplexity value to 6 makes the datapoints form small chunks (Figure 2), and I am aware this is because small perplexity values make the datapoints conserve proximity with only a few neighbors so this results in small groups.

In a related question this curved line shape appeared in a time-series dataset. I undetstood that time-series data have datapoints that change in small increments across most of the features, so the t-SNE results form a line.

So this means in my dataset, patients that have similar values, the "direction" of similarity is not random due to human physiology. For instance if patient A & B are similar and B has a slightly higher RBC value, the WBC, hematocrit are also higher and BIL values are lower. Hence, each individual patient may look like data from a time-series dataset.

Does this explanation make sense? Or is could there be a different explanation of this line-shaped distribution of a t-SNE plot?

From my own experience, if you have curved lines in your t-SNE plots, it usually implies your original data are scattered similarly in lines as well. One thing you can confirm whether this hypothesis is correct is that you can do principle component analysis (PCA) on your original data and then plot the first two dimensions. Usually, this plot will also be in curved lines.

Your explanation is plausible. However, you suggest you plot your first two PCA components to see what is really going on in your data.


Predicting with categorical data I have a dataset which contains various columns: numerical and categorical.

Dataset here:

I was able to process the categorical data using .astype('category') and cat.codes features in Pandas dataframe as explained here in Approach #2.

def process_categorical(self, dataset):
        """

        :param dataset:
        :return:
        """
        # Label Encoding.
        for categorical_feature in LABEL_ENCODED_FEATURES:
            categorical_feature = _to_string(categorical_feature)
            dataset[categorical_feature] = dataset[categorical_feature].astype('category')
            dataset[categorical_feature + '_cat'] = dataset[categorical_feature].cat.codes
        # Drop previous values.
        dataset = dataset.drop(LABEL_ENCODED_FEATURES, axis=1)
        # Rename back categories. {'to_user_cat': 'to_user' ... }
        dataset.rename(columns={c + '_cat': c for c in LABEL_ENCODED_FEATURES}, inplace=True)
        return dataset

I'm able to train my model and test data.
When I generate my training dataset where categorical features were encoded. (Using .cat.codes), this numerical data is different than my predictions. Example:

dataset['user_agent_numerical'].value_counts()

10    1002
11     850
1      288
8       18
7       17
2       16
6       14
3        5
0        4
9        2
5        2
4        2

Which corresponds to: dataset.groupby(['user_agent']).size()

user_agent
24gt2wreg24h                4
Avaya one-X Deskphone     288
Deskphone                  16
FreePBX 1.8                 5
HiDude UA v3.81             2
SimpleSIP V4.3              2
Twilio Gateway             14
caller                     17
eyeBeam release 3006o      18
friendly-scanner            2
pplsip                   1002
sipcli/v1.8               850

When I want to do a prediction and I pass the original record which looks like this: (Header just for reference)

ruri,ruri_user,ruri_domain,from_user,from_domain,from_tag,to_user,contact_user,callid,content_type,user_agent,source_ip,source_port,destination_port,contact_ip,contact_port
sip:789011972592277524@13.57.9.131,789011972592277524,13.57.9.131,78901113579131,13.57.9.131,1823821775,789011972592277524,78901113579131,1365624720-309058623-1808658022,application/sdp,pplsip,163.172.120.42,60993,5060,212.129.10.158,60993

I need to do pre-processing, My prediction data contains pplsip as user-agent hence it ends up with 0, instead of 10.

How can I pass data to my predictions that needs to be transformed to categorical values. If I tried to convert it to categorical I end up with a different class value.

User-agent is just an example, but could be IP address or called number which corresponds to very large finite set hence a dictionary does not scale. I think this is a common problem, but not sure how can I solve it. I tried using dummies Approach#3 and I end up generating additional columns which doesn't match my prediction dataset. Issue reported here.

Complete code here.

Yep this is a common problem. What I would do is use SKLearns label encoder. With a similar API to SKLearn models, it can be fit to your category - meaning that any new data passed through the encoder object is encoded in the same fashion. For example

# Import the encoder
from sklearn.preprocessing import LabelEncoder

# Fit it to your training set + transform
encoder = LabelEncoder()
train[categorical_feature] = encoder.fit_transform(train[categorical_feature])

# Use it on the test set to ensure the same transformation
test[categorical_feature] = encoder.transform(test[categorical_feature])

Huge caveat # 1

This assumes every possible category is present in your training set. If you think this isn't the case, fit the encoder to a dataset that does have them all.

Huge caveat # 2

Depending on the model you're using, categoricals still aren't 'processed' when they've been label-encoded , you probably still need to one hot encode (SKLearn has a package for that too).


Your second change (calculating the average error) is the correct method. Imagine this: if there are billion examples in your training set, even if you make (very) small error on each, the sum total is going to add up to a large number.
So the cost function in case of least squared is mean squared error, not just sum.


How should I format input and output for text generation with LSTMs I'm attempting to generate a response to an input line of text using an LSTM. I've considered various forms of input, including one-hot encoding each character in the line and passing each input line as a vector of one-hot encoded vectors. I've also considered using a dictionary and one-hot encoding each word in the sentence based on its alphabetical position.

However, I'm not sure about any of this, as I am new to natural language processing in machine learning. What would be the best way to format my input (and my output) for this problem?

You can use word embedding, to encode words as vectors of real numbers. Then all calculations, such as comparison of words (to find similarity), are performed in that high-dimensional space.

  "What would be the best way to format my input (and my output) for
  this problem?"

I cannot tell which is the best approach (depends on your problem) but this one is very common. It's also very easy to implement in Keras, using the Embedding layer.


What is a 1D Convolutional Layer in Deep Learning? I have a good general understanding of the role and mechanism of convolutional layers in Deep Learning for image processing in case of 2D or 3D implementations - they "simply" try to catch 2D patterns in images (in 3 channels in case of 3D).

But recently I bumped into 1D convolutional layers in the context of Natural Language Processing, which is a kind of surprise for me, because in my understanding the 2D convolution is especially used to catch 2D patterns that are impossible to reveal in 1D (vector) form of image pixels. What is the logic behind 1D convolution?

In short, there is nothing special about number of dimensions for convolution. Any dimensionality of convolution could be considered, if it fit a problem.

The number of dimensions is a property of the problem being solved. For example, 1D for audio signals, 2D for images, 3D for movies . . .

Ignoring number of dimensions briefly, the following can be considered strengths of a convolutional neural network (CNN), compared to fully-connected models, when dealing with certain types of data:

The use of shared weights for each location that the convolution processes significantly reduces the number of parameters that need to be learned, compared to the same data processed through fully-connected network.
Shared weights is a form of regularisation.
The structure of a convolutional model makes strong assumptions about local relationships in the data, which when true make it a good fit to the problem.

3.1 Local patterns provide good predictive data (and/or can be usefully combined into more complex predictive patterns in higher layers)

3.2 The types of pattern found in the data can be found in multiple places. Finding the same pattern in a different set of data points is meaningful.

These properties of CNNs are independent of the number of dimensions. One-dimensional CNNs work with patterns in one dimension, and tend to be useful in signal analysis over fixed length signals. They work well for analysis of audio signals, for instance. Also for some natural language processing - although recurrent neural networks, which allow for different sequence lengths, may be a better fit there, especially ones with memory gate arrangements such as LSTM or GRU. Still a CNN can be easier to manage, and you could simply pad the input to be fixed length.


NLP: what are the advantages of using a subword tokenizer as opposed to the standard word tokenizer? I'm looking at this Tensorflow colab tutorial about language translation with Transformers, https://www.tensorflow.org/tutorials/text/transformer, and they tokenize the words with a subword text tokenizer. I have never seen a subword tokenizer before and don't know why or when it should be used as opposed to a word tokenizer.
The tutorial says The tokenizer encodes the string by breaking it into subwords if the word is not in its dictionary.
To get an idea of what the results can look like, the work Transformer gets broken down into index-subword pairs.
7915 ----> T
1248 ----> ran
7946 ----> s
7194 ----> former

Does anybody know what the advantages of breaking down words into subwords is and when somebody should use a subword tokenizer instead of the more standard word tokenizer? Is the subword tokenizer used because the translation is from Portuguese to English?
*The version of Tensorflow is 2.3 and this subword tokenizer belongs to tfds.deprecated.text

Subword tokenization is the norm nowadays in NLP models because:

It mostly avoids the out-of-vocabulary (OOV) word problem. Word vocabularies cannot handle words that are not in the training data. This is a problem for morphologically-rich languages, proper nouns, etc. Subword vocabularies allow representing these words.  By having subword tokens (and ensuring the individual characters are part of the subword vocabulary), makes it possible to encode words that were not even in the training data. There's still the problem with characters not present in the training data, but that's tolerable in most of the cases.

It gives manageable vocabulary sizes. Current neural networks need a pre-defined closed discrete token vocabulary. The vocabulary size that a neural network can handle is far smaller than the number of different words (surface forms) in most normal languages, especially morphologically-rich ones (and especially agglutinative ones).

Mitigates data sparsity. In a word-based vocabulary, low-frequency words may appear very few times in the training data. This is especially troublesome for agglutinative languages, where a surface form may be the result of concatenating multiple affixes. Using subword tokenization allows token reusing, and increases the frequency of their appearance.

Neural networks perform very well with them. In all sorts of tasks, they excel: neural machine translation, NER, etc, you name it, the state of the art models are subword-based: BERT, GPT-3, Electra,...


How do I get Multiple CSV files (csv file names will be column names) from a folder to a pandas dataframe? I think the title is enough for what is my problem here. I have 100 tickers in a folder. I got all csv files to one list. but I wanna see the tickers name which these are csv file names.

what should I do for getting all tickers like that. I'm doing it manually but I need a simple way to do that.
import pandas as pd
import os

#BİST100-2010.01.01-2020.07.01/ is my folder.
log_total = []
for file in os.listdir('BİST100-2010.01.01-2020.07.01/'):
    log_total.append(pd.read_csv('BİST100-2010.01.01-2020.07.01/'+file+''))

by the way after I do that I got this.

How do you get to the second picture from that code? I wil give you rough set of steps to follow along with the main function names. you can search through the pandas documentation for more details.

initialise some empty lists:  ticker_names=[] dataframes=[]
Loop over the files and read them using df = pd.read_csv(file_name)
Take just the Close column of each single ticker's dataframe: df = df["Close"]
Keep track of the tickers' names in a list: ticker_names.append(ticker)
Bring them all together into one dataframe: final_df = pd.concat(dataframes, left_index=True, right_index=True). We uses the indices because you have date-time index values, and we want to keep them all.
Then give each column the correct name: final_df.columns = ticker_names

You can play around with each of these steps, and I would suggest looking at the **pd.merge** documentation.


How to build a symptom checker and medical diagnose chat bot I want to build a chatbot which can diagnose an illness depending upon the symptoms which are given to it. I want to show you an example scenario of how it should work;

Application - A   ,   User - U

  A) Hello, What is your symptom?

  U) Headache

  A) How long do you have this symptom? ( System will list possible
  choices to the user)

  U) 3 Months

  A) Do you also have photo-affinity? ( System will provide simple "Yes-No-I don't" know choices)

  U) Yes

  .

  .

  A) According to your symptoms, you have "migraine".

There are several applications which work in that manner like "Ada Health", "YourMD" and so on.

I want you to show me a way, give me a kick starter where to start. Are there any resources helpful about this topic?

Can you describe briefly, how one would build such a system in terms of machine learning and data science?

After my questions, I want you to show you my way of thinking about how this system can be built. Please give your comments about it as well

Let's say we have a database and there are symptoms - disease wives. Here is the example;

Once the user enters a symptom to the system, my algorithm is going
to find all of the matched diseases from the database, e.g., "headache" is the entered symptom by the user and the matched diseases with this initial symptom are "migraine", "flu", "cold", "tumor", etc...
After that, all of the symptoms which are connected with these new
diseases are added to the question queue, e.g., for "flu" match, new questions about "flu" disease will be asked to the user like "Do you have also sneeze?"
While the user continues to answer questions, the positive symptoms which user has will be recorded. At the end of the question-answer phase, all of the symptoms which are recorded are going to be used in our machine learning model as the input features.

Does that way of thinking hit the right notes? Thanks in advance for all of the answers.

We built the symptom checker with AI API endlessmedical.
We provide free to download for any use source code of entire symptom checker here:
https://endlessmedical.com/chatbot-like-symptom-checker-free-source-code-download/
Have fun!


Damerau-Levenshtein Edit Distance in Python I found some python codes on Damerau Levensthein edit distance through google, but when i look at their comments, many said that the algorithms were incorrect. I'm confused.

Can someone share a correct python code on Damerau Levensthein Distance?

Thank you.

Which one is correct? (Sorry, i don't know how to properly edit the typing with my handphone, my only online device for the time being)

https://www.guyrutenberg.com/2008/12/15/damerau-levenshtein-distance-in-python/

The first one,

"""Compute the Damerau-Levenshtein distance between two given strings (s1 and s2)"""

    def damerau_levenshtein_distance(s1, s2):

        d = {}
        lenstr1 = len(s1)
        lenstr2 = len(s2)
        for i in xrange(-1,lenstr1+1):
            d[(i,-1)] = i+1
        for j in xrange(-1,lenstr2+1):
            d[(-1,j)] = j+1

        for i in xrange(lenstr1):
            for j in xrange(lenstr2):
                if s1[i] == s2[j]:
                    cost = 0
                else:
                    cost = 1
                d[(i,j)] = min(
                               d[(i-1,j)] + 1, # deletion
                               d[(i,j-1)] + 1, # insertion
                               d[(i-1,j-1)] + cost, # substitution
                              )
                if i and j and s1[i]==s2[j-1] and s1[i-1] == s2[j]:
                    d[(i,j)] = min (d[(i,j)], d[i-2,j-2] + cost) # transposition

        return d[lenstr1-1,lenstr2-1]


does smaller training set always lead to better training accuracy? Does anyone know any existing research or have observed some experimental results in deep learning about the following:

For a fixed data set, if you subset the dataset from 50% to 90% as training set, using the same number of epochs, will the training error be bigger as the training set becomes bigger?

would be great if anyone could give some concrete experimental results or paper about this?
For example, this one https://arxiv.org/abs/1901.04169v1
and this one
https://www.researchgate.net/publication/333671691_Resampling-based_Assessment_of_Robustness_to_Distribution_Shift_for_Deep_Neural_Networks

but both papers have not drawn any conclusions on this issue.

I don't think there a connection between size of the training set and model's performance during training. A dataset can be very noisy and unpredictable, independently from its size.

To my knowldge, it is not possible to derive any general rule about this.


How to correctly set a target for a time series based model? I need help determining the best way to go about creating the target variable for a machine learning model that is trading a financial instrument (stocks, foreign currencies, crpyto, etc).

Below is some sample data to help with answering the question.

Time: When the data was recorded
Price: Is the current price of the instrument and is the price I would be buying at.
price_good: Price level that I need to sell at to make a profit
price_bad: Price level that I need to sell at to minimize losses
good_id: Returns the ID of the first row where price >= price_good
bad_id: Returns the ID of the first row where price <= price_bad
target: If good_id < bad_id, target = 1. Else target = 0

My ideal scenario is when price_good occurs before price_bad. The issue I'm running into is how to correctly set the target field. I have 2 ways that I can solve this -

Option #1 - This option uses all of the data and has no restrictions on creating the target field but has a slight mismatch on how it would work in production.

id         time       price    price_good   price_bad    good_id   bad_id   target
1          01-01-19   100      110          90           4         nan      1
2          01-02-19   105      115          95           4         nan      1
3          01-03-19   109      120          99           4         nan      1
4          01-04-19   121      131          111          nan       5        0
5          01-05-19   110      120          97           nan       nan      nan

Option #2 - This option doesn't use all of the data and doesn't allow a target to be set until the initial row reaches it's target which is how it would work in production.

id         time       price    price_good   price_bad    good_id   bad_id   target
1          01-01-19   100      110          90           4         nan      1
2          01-02-19   105      115          95           nan       nan      nan
3          01-03-19   109      120          99           nan       nan      nan
4          01-04-19   121      131          111          nan       5        0
5          01-05-19   110      120          97           nan       nan      nan

"In production" means how it would work after the model is built. For example,
If my model predicted that id:1 would hit target_good, I wouldn't be able to make any other trades until that trade reaches price_good or price_bad which aligns with option #2. However, I'm losing the information about all of the other trades that I could of made from option #1 if I had more capital to invest.

If I go with option #1, the model will sometimes "overfit" because it has data that is closer together in time and uses mostly the same information.

If I go with option #2, it is a more accurate representation of how it would work in production but I lose a ton of data points. For example, what if I started at id:2 instead of id:1. I would end up with different data points that the model is built on. I could test starting at multiple places to find the most optimal starting place but that would take a ton of iterations and resources to test.

Any tips on how to model this correctly? Thanks!

You've got an interesting, if over analysed question. I would just say your question itself limits your potential profit. You want to give your computer more choice in this matter. the whole price_good, price_bad approach is price_bad. The computer should decide EVERYTHING in this game and you will want to build in a consideration of previous stock history, share earnings, maybe even public sentiment from twitter, etc.

This really seems like the optimal scenario to implement some variant of q-learning. You'll need a lot of data and the stock market is notorious for being difficult for the layman to derive any meaning from.

What is Q-learning?

Q learning is a deep learning method with allows a computer to find the ideal set of actions to maximize some final score. The system is composed of 3 parts: a deep learning network that predicts the target, a q table that contains the state at a given time and a possible action, and the change in the final score.

Read more here: https://towardsdatascience.com/simple-reinforcement-learning-q-learning-fcddc4b6fe56

Really, read that before continuing :)

Step 1:

You create the q-table (a table of many possible actions). In this case your state will be a vector of the number of shares you hold, the liquid funds you have for buying, and n number of consecutive prices for a stock (You'll also need the next price of the stock for a later step). You will then make a choice: how much of the stock do you want to sell? this number can be any real integer. a negative number indicates selling, a positive number indicates buying and a 0 indicates holding.

EDIT: I want to explain a bit more what's going on with the "n number of consecutive prices for a stock". This is where your "timeseries" bit comes into play. It is a vector that looks at a frame of prices at any given time, but it not necessarily the latest frame. It's just a way for you to train on what would have happened in the past if you were making that trade. For example if I have a vector of prices that looks like all_prices below, the frames would look like:

 all_prices = [12, 13, 14, 13, 14, 15, 16, 17, 18]
 frame1 = [12, 13, 14, 13]
 frame2 = [13, 14, 13, 14]
 frame3 = [14, 13, 14, 15]
 frame4 = [13, 14, 15, 16]

For each state you'll have to add rows to the table of every possible action (everything between sell all and buy with as much cash I have). Then for each row you'll calculate the reward for that action. Now, the reward is calculated based on the next price of the stock. It can be negative (bad) or positive (good).

So here's what that table would look like for a single stock:

 *n_shares* | *cash* | *price_at_t=1* | *price_at_t=2* | .... | *last_reward* | *action*
 50         | 1000   | 13.41          | 12.45          | .... | 0             | -10
 50         | 1000   | 13.41          | 12.45          | .... | 0             | -5
 50         | 1000   | 13.41          | 12.45          | .... | 0             | 0
 50         | 1000   | 13.41          | 12.45          | .... | 0             | 5
 50         | 1000   | 13.41          | 12.45          | .... | 0             | 10

Step 2:

You'll train some deep learning network on each row (state) to predict the reward. yay!

Step 3:

When you actually want to make a decision on your data, you create a table of all possible states and actions and have your nueral network predict the reward (now it's unknown). You can select the action which maximizes long term reward.

Some caveats

Now the above really doesn't look at all that much data. You can add in soooo much more and potentially create a better model. I've never done this, but I don't have particularly high hopes for something like this. I'm definitely open to reconsider though. Perhaps if you look at this with each timepoint being the close price, and each day you get one change to sell, buy, or hold, it might be feasible. Otherwise, you'll need an API that allows you to trade (super cool, but I haven't heard of something like this for consumers).

Are there better solutions?
Yes, this is just a simple reinforcement learning solution. We can do better, but this seemed like a simple starting point to me.

Am I liable if you loose all your moneeeys?
Nope.

Is the above normal q-learning?
Not really. I took some shortcuts and glossed over some aspects of it. I do hope that it wasn't too complex and that it helped you understand how to better approach the problem.

Will you become a billionaire after this?
....maybe. I hope so. If it starts working, tell me and I'll invest with you.


K-Nearest Neighbours algorithm explanation needed I need some explanation for K-Nearest Neighbors algorithm.

Why is the training process needed in KNN algorithm? In regression
models the training process means to find optimum parameters for a
selected function. Does the training in KNN mean that the similarity
distances are calculated between training datapoints? Why is it good
for a later prediction phase?
Are similarity distances also calculated between training and
testing datapoints? If so then why is it necessary at all that
before that the distances between training datapoints were
calculated?
If the following code is executed then what does it exactly do? Does
it mean that distances between training datapoints are calculated?

neigh = KNeighborsClassifier(n_neighbors = k).fit(x_train, y_train)
If the following code is executed then what does it exactly do? The
manual says that class labels for each data sample are returned. But
how is it done? Does it mean that distances between training and
testing datapoints are first calculated? If not then how can we find
neighbors for "unseen datapoints" at all? We need to know distances.

yhat = neigh.predict(x_test)

K Nearest Neighbors is a Classification Algorithm. Just as with every classification algorithm is important that the algorithm doesn't "remember" the answers and that the answer it gets can be generalized to all the population, not just learned from the database.

1. Why is the training process needed in KNN algorithm? In regression model the training process means to find optimum parameters for a selected function. Does the training in KNN mean that the similarity distances are calculated between TRAINING datapoints? Why is it good for a later prediction phase?

The traininig process is needed in KNN because (as I stated before), we want to avoid it from learning-the-database, the famous "overfitting".

As you stated, in a regression the training is for finding the optimum parameters, here too, but the idea of "parameters" slightly changes: Here the parameters are the selected points (individuals) and of course, the $k$, separating in train/test allows the model to be more general when the test data is unknown.

Yes, it means the KNN calculates distance only in the training set. When the optimum parameters are found in the training set, the results in the test set are better, more general.

Are similarity distances also calculated between TRAINING and TESTING datapoints? If so then why is it necessary at all that before that the distances between TRAINING datapoints were calculated?

The new datapoints (the ones from the TEST dataset) do not participate in a recalculation of KNN, the way the new points decide if they are blue or red is determined by the points which were calculated before (TRAIN dataset), by measuring which color the k nearest neighbors (of the new point) have.

For the new green point, with $k=3$, two were blue and one red, the new color is likely to be blue (and is assigned blue).

3. If the following code is executed then what does it exactly do? Does it mean that distances between TRAINING datapoints are calculated?

neigh = KNeighborsClassifier(n_neighbors = k).fit(x_train, y_train)

No, a more thoughful explanation would be that they are "memorized" in certain position and certain y_value. The calculation of distances is done when fitting TEST data.

4. If the following code is executed then what does it exactly do? The manual says that class labels for each data sample are returned. But how is it done? Does it mean that distances between TRAINING and TESTING datapoints are first calculated? If not then how can we find neighbours for "unseen datapoints" at all? We need to know distances.

yhat = neigh.predict(x_test)

As was explained in another point, the former code (fit) did not calculate distances, the code memorized the points. In the predict code, distances between x_train and x_test are computed, and for every member of x_test, the k nearest neighbors (from x_train) are found, and they vote to retrieve a class for this new datapoint (in x_test).

In the image, when $k=3$, they vote 2-1 to say that the green point (new one) is blue.


Why do histogram bars vanish when we keep the bins value high in matplotlib? Also, the histogram bar widths are different on certain values of bin.
How to keep the bar widths uniform?
I have tried using the rwidth but that dos not solve my problem.

Data:

test    age
17 - Alpha OH PROGESTERONE - HORMONE ASSAYS 23
17 - Alpha OH PROGESTERONE - HORMONE ASSAYS 26
17 ALPHA HYDROXY PROGESTERONE   18
17 ALPHA HYDROXY PROGESTERONE   21
17 ALPHA HYDROXY PROGESTERONE   25
17 ALPHA HYDROXY PROGESTERONE   27

Code

axes = plt.gca()
axes.set_xlim(0, 100)
axes.set_ylim(0, 100)
plt.hist(df.loc[0:1,'age':'age'].to_numpy(),100, rwidth=0.9)

axes = plt.gca()
axes.set_xlim(0, 100)
axes.set_ylim(0, 100)
plt.hist(df.loc[2:5,'age':'age'].to_numpy(),10, rwidth=0.9)

axes = plt.gca()
axes.set_xlim(0, 100)
axes.set_ylim(0, 100)
plt.hist(df.loc[6:14,'age':'age'].to_numpy(),100, rwidth=0.9)

The height of a bar shows the number of measurements within that range.

The bars get shorter when you increase the bin count, because the more bins you have in total, the less measurements fall into an individual bin.


SPARK 1.5.1: Convert multi-labeled data into binary vector I am using SPARK 1.5.1, and I have DataFrame that looks like follow:

labelsCol, featureCol
(Label1, Label2, Label 32), FeatureVector
(Label1, Label10, Label16, Label30, Label48), FeatureVector
...
(Label1, label 95), FeatureVector

The first column is the list of labels for that sample, and in total I have 100 label.

I would like to build a binary classifier for each label, so I want to transform the labels list column into a binary vector.

The binary vector will have a length of 100 and the value will be 0 or 1 depends on the existence of the label for sample.

Is there any strait forward solution for this?

Spark only recently implemented CountVectorizer, which will take the labels (as strings) and encode them as your 100-dimensional vector (assuming all 100 labels show up somewhere in your dataset).  Once you have those vectors, it should be a simple step to threshold them to make them 0/1 instead of a frequency.


How does the seed value work in Weka for clustering? I have recently been learning about the various clustering methods, and I decided to apply Furthest Point clustering in Weka with varying seed values.

The seed value dictates the initial choice of points in the data set. It can't be totally random, since repeatedly doing the clustering with, say, seed = 3 gives the same result.

How does it select data points based on the seed?

Pseudo-random number generators are initialized with a seed.

There is little real randomness available. Most just looks random to us, because the pattern is too complex. Read up some examples on pseudo random number generation such ad Mersenne Twisters.


How do I produce generative neural networks? To my understanding, a learning method is used to train a neural network on a dataset by training it's outputs.

I'm aware that the same neural network can be run backwards to generate data.

I'd like to learn how to reverse a neural network's algorithm to generate.

Any ideas?

See this video.

Feed-forward neural networks are discriminative models, i.e. they model $P(y|x)$.

Something like restricted Boltzmann machines look a lot like neural networks, and share some of the proprieties, and are generative models, i.e. they model $P(y,x)$ --- which, of course, you can always turn into a $P(y|x)$ using Bayes' rule, but the opposite is not true.

Libraries like Theano have support for RBMs.

EDIT: since the writing of my reply, generative adversarial networks have been invented. (Actually, they were invented in 2014, but only in 2016 were they taken to their full potential.) In these networks, you feed noise to the neural network and it generates data samples. This is achieved by co-training it against a regular discriminative neural network. Both networks are trained at the same time: the discriminative one must classify each sample as being either true or synthetically created by the other neural network. And the generative neural network must fool this one, so they get better and better, and, in the end, you have both a discriminative and a generative model!


Multilingual data handling for text summarisation I have a dataset having text documents as each entry , the documents are of various ethical languages which are not specified. How should be my strategy if I want to summarise the content of the dataset.

For eg. Doc1 : "Ruby is a dog lover.Ruby loves her dogs so much. Ruby has a pet whose name is a dog"

Now summarising it is easy as "Ruby loves dog.Ruby has a dog"

But How should be the approach , when the dataset also has documents in french, chinese, hindi etc...

Some of the problems are: there are cases of code switch, and I don't know what are the languages in the dataset before hand.

What I am guessing at this is to may be change all the dataset to english language and then do the tasks? In case if converting to english is a good choice then which library would be better?

There are two basic approaches to summarization: abstractive and extractive.

Extractive summarization is simpler and consists of selecting informative pieces/sentences from text. It is commonly performed using algorithms like TextRank (a variation of PageRank). An example implementation can be found in gensim. There are other approaches, for example check out sumy library.

Abstractive summarization on the other hand consists of summarizing documents with shorter sentences which do not have to come from the input text (an abstract may paraphrase some fragments of text). It is commonly performed using Deep Learning Seq2Seq methods.

Which approach is better for start? Extractive wins by far. First of all, you don't have to train any model, as these methods rely on information found solely in summarized text. Second, deep learning models, even if they exist and can be easily found *, are trained on specific texts, so they might not generalize well across application domains.

  But How should be the approach , when the dataset also has documents
  in french, chinese, hindi etc...

If you have good tokenizers and stemmers for these languages you should be fine with TextRank, as it only relies on word counts (I'm not sure about Chinese though). BTW language detection is by far easier than summarization - just count number of stopwords of different languages that document contains.

*They're not, for example Google published a paper sometime ago on the topic and didn't even disclose model, since the training dataset was proprietary. I was trying to find something for abstractive summarization sometime ago but I couldn't find anything that had good pretraining and worked out of the box.


Is there a rule of thumb when designing neural network in deep reinforcement learning? In deep learning, we can assess model's performance with loss function value and improve model's performance with K-fold cross-validation and so on. But how can we design and tune neural network used in deep reinforcement learning? We can assess reinforcement learning algorithm's performance itself with rewards and so on, but how can we be sure that neural network used in reinforcement learning algorithm is good or bad ?

The process of reinforcement learning already implies that you have a base model to work from, that's what you're reinforcing. So, presumably, that underlying model is already good, otherwise you wouldn't be using it, right?

The whole point of reinforcement learning is to introduce your (functioning) model to new information and/or changing conditions. Reinforcement learning isn't going to take a bad model and turn it into a good one. If your base model is not performing to your satisfaction, then you should go back to a model design/selection phase; reinforcement learning is not going to help you in that scenario.


Probabilities in KERAS I'm struggle with a problem OF class probabilities (binary, 0 and 1). Dont know why but after 100 epoches the probabilities became 0 or 1 (like the class to predict).
Maybe something is not working with the code or I missing something?

Train data is 20.000 rows (more or less), test data is 2000 rows. Each row is a match where 0 is lost and 1 is won.

y_train_binary = keras.utils.to_categorical(Y, 2)
y_test_binary = keras.utils.to_categorical(t_Y, 2)
model = Sequential()
model.add(Dense(40, input_dim=45, activation='relu'))
model.add(Dropout(0.05))
model.add(BatchNormalization())
model.add(Dense(30, activation='relu'))
model.add(Dropout(0.05))
model.add(BatchNormalization())
model.add(Dense(20, activation='relu'))
model.add(Dropout(0.05))
model.add(BatchNormalization())
model.add(Dense(10, activation='relu'))
model.add(Dropout(0.05))
model.add(BatchNormalization())
model.add(Dense(2, activation='softmax'))
keras.optimizers.Adam(lr=0.5, beta_1=0.9, beta_2=0.999, epsilon=0.3)
model.compile(loss='categorical_crossentropy', optimizer='Adam', metrics=['accuracy'])
model.fit(X, y_train_binary, epochs=300, validation_split=0.2, batch_size=10, verbose=0)
prediction_classes = model.predict_proba(t_X)
numpy.savetxt("C:/Users/Megaport/Desktop/foo.csv", prediction_classes, delimiter=",")

I've tried so many parameters (learning rate, epochs, batch size, epsilon, add layer, less layer, different value of dropout) but the problem is the same: probs are not working.

the probabilities of the class are 1 or 0 and not a value between 1 and 0. This happen after 100 epoches.

data test with prediction after 100 epoches is like this:

**RESULT**   VALUE_A     VALUE_B     VALUE_C    **PRED_0    PRED_1**
   0           4          5          3           1          0
   0           7          4          5           0          1
   1           6          7          6           0          1
   1           2          3          4           0          1

What I'm looking for:

**RESULT**   VALUE_A     VALUE_B     VALUE_C    **PRED_0    PRED_1**
   0           4          5          3           0.65          0.35
   0           7          4          5           0.25          0.75
   1           6          7          6           0.20          0.80
   1           2          3          4           0.30          0.70

Plus, someone could please give me some advice looking at accuracy and loss graph?

https://i.stack.imgur.com/vTnz4.png
https://i.stack.imgur.com/Zz5cv.png

Try predicting log-probabilities that would give you bigger seperation (i.e. values between -inf and 0) I Would clip these values (so that you dont take log from 0 which would be -inf but something like if 0 than log(0.0001)
For example:

y_pred = estimator.predict_proba(X.drop(columns=self.drop_columns))
log_prob = lambda x: np.log(np.clip(x, 0.001, 0.999)).reshape(-1, 1)
y_pred = np.hstack([log_prob(y_pred[:, 0]), log_prob(y_pred[:, 1])])


Neural Network Learning Rate vs Q-Learning Learning Rate I'm just getting into machine learning--mostly Reinforcement Learning--using a neural network trained on Q-values. However, in looking at the hyper-parameters, there are two that seem redundant: the learning rate for the neural network, $\eta$, and the learning rate for Q-learning, $\alpha$. They both seem to change the rate at which the neural net takes new conclusions over old ones.

So are these two parameters redundant? Do I need to worry about even having $\alpha$ as anything other than 1 if I'm already tuning $\eta$, or do they have ultimately different effects?

There is usually only one learning rate active when using a neural network as function approximator in reinforcement learning. The different names $\eta$ and $\alpha$ are just different conventions for the same basic concept.

When you use a function approximator, other than a linear one, in reinforcement learning, then typically you would not use TD error based update like this:

$$\mathbf{w} \leftarrow \mathbf{w} + \alpha[R+\gamma\hat{q}(S', A',\mathbf{w}) - \hat{q}(S, A,\mathbf{w})]\nabla \hat{q}(S, A,\mathbf{w})$$

But you would train your estimator in a supervised learning manner on sampled TD target (which would use $\eta$ param in a neural network):

$$\mathbf{x} = \phi(S, A), y = R+\gamma\hat{q}(S', A',\mathbf{w})$$

You can actually do either if your library supports it - it is certainly possible to calculate $\nabla \hat{q}(S, A,\mathbf{w})$ for a neural network for instance, instead of using an explicit training loss function. However, the two approaches are equivalent ways of expressing the same thing, there is no reason to use both.

There are other parameters used in reinforcement learning that may affect rates of convergence and other properties of learning agents. For example with differential semi-gradient TD learning - which might be an algorithm you would look at for a continuous task - Sutton and Barto present $\beta$ as a separate learning rate for the average reward, distinct from the learning rate of the estimator.

  So are these two parameters redundant? Do I need to worry about even having $\alpha$ as anything other than 1 if I'm already tuning $eta$, or do they have ultimately different effects?

They are essentially the same parameter with a different name.

If you are trying to pass an error value like $R+\gamma\hat{q}(S', A',\mathbf{w}) - \hat{q}(S, A,\mathbf{w})$ (whether multiplied by $\alpha$ or not) into the neural network as a target, then you have got the wrong idea. Instead you want your neural network to learn the target $R+\gamma\hat{q}(S', A',\mathbf{w})$. That's because the subtraction of current prediction and multiplication by a learning rate is built into the neural network training.


Which Spark MLlib regression algorithm is suitable for numeric predictions based on non-numeric features? I am working on Spark MLlib and have a project where I have to make predictions for numeric data based on non-numeric features. I am a bit confused about which regression algorithm to use from Spark MLlib library primarily due to being new at this. The algorithms present in Spark MLlib library are:

-linear models (SVMs, logistic regression, linear regression)
-naive Bayes
-decision trees
-ensembles of trees (Random Forests and Gradient-Boosted Trees)
-isotonic regression

Can anyone provide me some guidance as to which algorithm will be suitable for predictions for numeric data based on non-numeric features?

I would suggest -ensembles of trees (Random Forests and Gradient-Boosted Trees).

Here is a nice reference for handling such data with decision trees.


How to handle a feature vector that could be variable length? I would like to train a machine learning model with several features as input as X[] and with one output as Y. For example Every sample has a Data frame like this: X[0], X[1], X[2], X[3], X[4], Y
Let's say One sample the followings Data is only one value: X[0], X[1], X[2], X[4], Y This is normal machine training problem.
But now, if I would like to set X[3] multiple values for example sample 1 Data is:
X[0] | X[1] | X[2] |         X[3]         | X[4] | Y
 10  |  5   |   6  | [10, 20, 30, 40, 50] |  7   | 90

Data in sample 2 is:
X[0] | X[1] | X[2] |         X[3]         | X[4] | Y
 11  |  7   |   5  | [20, 30, 40, 50, 60] |  3   | 80

Is this possible to follow the normal machine training process and got a model which could calculate a sample with other example with Data like:
X[0]   | X[1] |  X[2]  |         X[3]         | X[4] | Y
 10.5  |  6   |   5.5  | [15, 25, 35, 45, 55] |  5   | ???

If the length for each X[3] is not long, it is possible to divide the X[3] into multiple new features, but if the length of X[3] is very long (len > 1000) with different distribution, making binary is also lead to too many new features. Is there any way to treat the X[3] directly without adding new features?
Really appreciate for your help.

There are many options.
The most common method to handle variable length vectors is to pad the vectors. Add zeros as placeholders to the shorter length vectors until all the vectors are the same length. Then it can be modeled similar to any other feature vector.
Another option is to take the norm of the vector. That would yield a single scalar that could be used in the machine learning model.
Ultimately, the way to handle a feature vector that could be variable lengths depends on what the feature represents.


Entropy loss from collapsing/merging two categories Suppose I am counting occurrences in a sequence. For a classical example, let's say I'm counting how many of each kind of car comes down a highway.

After keeping tally for a while, I see there are thousands of models. But only a handful show up frequently, whereas there are many that show up once or only a few times (iow the histogram resembles exponential decay). When thinking about the statistics of this situation, it hardly seems to matter that I saw this one obscure car this one time as opposed to another obscure car - it doesn't seem like it conveys much information either way. So what if I collapse all the rare models into a single category like "others", to make the data easier to store. How much information will I lose?

I've gotten as far as reducing the problem to a smaller one and finding an upper bound.

Collapsing 3 categories A, B and C into one category D is the same as first collapsing categories A and B into category E, then collapsing E and C into F. F will be exactly the same as D. So the final information loss is not path-dependent, and it is sufficient for us to solve information loss from collapsing 2 categories. The result should easily generalize to n categories.
For 2 categories, we can re-encode the sequence so that each occurrence of categories A and B is instead recorded as C. However, for each occurrence of C, an additional bit is recorded that shows whether that C came about from A or B. This re-encoding incurs zero information loss. Erasing these bits would effectively collapse A and B into C. Therefore the average information loss from collapsing categories A and B is (1 bit) * ((number of occurrences of A) + (number of occurrences of B)).

Is my logic above correct? Is my upper bound accurate? What is the lower bound/exact solution?

I ended up coming up with my own solution. I'm not sure if it's correct so I won't mark it as an answer.

Let's call the original distribution P. Suppose we will be collapsing A and B into a new category X. The entropy of P will be: $S_P = - \sum P_i\ln P_i$ where $i$ is A, B, C, ... Notably, $P_A > 0, P_B > 0, P_X = 0$ .

After collapsing, we get a new distribution $Q$. Now $Q_A=Q_B=0$ and $Q_X =Q_A+Q_B$. This will have entropy $S_Q$.

The entropy we lost by collapsing is $S_P-S_Q = P_A \ln P_A + P_B \ln P_B - Q_X \ln Q_X$. All other terms, such as $P_C\ln P_C$ for category $C$ which is unaffected, are present in both distributions and cancel out.


Advantages of CNN vs. LSTM for sequence data like text or log-files When do you tend to use CNN rather than LSTM (or the other way round) in classification or generation tasks of sequential data like text or log-data? What are the reasons for the decision and what does it depend on? Are there any papers or statistics that confirm this?
I'm thinking of data like Linux log entries or short sentence of length of less than 20 words/tokens.
Personally i would almost always use LSTM but I'm curious if CNN wouldn't be better in some cases, if its possible to implement them in a meaningful way. On short sentence there isn't much buffer to use CNN if i'm not mistaken.

The WaveNet paper is a good place to start for a discussion of Causal CNNs vs LSTMs for synthesis and classification.  In that paper they actually train the network to do both at the same time, for example.
For short tokenized sentences such as you describe, Transformers are probably the state-of-the-art though.  In fact, they have been used recently in place of CNNs even for image classification tasks.  So, you don't necessarily have to have an encoder portion at all.  And, I believe the general thought is that the Transformer is encapsulating the function of both RNNs and CNNs and is outperforming them both in benchmarked tasks.
So, any of these networks could probably do the job you describe, but I would tend to using a Transformer as my first go.  It is built for this kind of task and is quicker and easier to train than a RNN.  I do suspect though that you would get good results with a LSTM on such short sequences.  They can be difficult to train properly on very long data.


correlation between categorical(ordinal) and discrete(continuous) value I am doing my bi variate analysis  but right now looking to see the correlation between  my atributes

some are categorical 5 levels and others amount of money.

how can I see the correlation between them ?

do I have to create class for my money amount?

You can juse bin them to numerical bins [1 - 5] as long as you are sure you're doing this to ordinal variables and not nominal ones. Bivariate analysis should be easier for you.

If you still want to see how to get correlation of categorical variables vs continuous , i suggest you read more about Chi-square test and Analysis of variance ( ANOVA )


Advantages of AUC vs standard accuracy I was starting to look into area under curve(AUC) and am a little confused about its usefulness. When first explained to me, AUC seemed to be a great measure of performance but in my research I've found that some claim its advantage is mostly marginal in that it is best for catching 'lucky' models with high standard accuracy measurements and low AUC.

So should I avoid relying on AUC for validating models or would a combination be best? Thanks for all your help.

Really great question, and one that I find that most people don't really understand on an intuitive level. AUC is in fact often preferred over accuracy for binary classification for a number of different reasons. First though, let's talk about exactly what AUC is. Honestly, for being one of the most widely used efficacy metrics, it's surprisingly obtuse to figure out exactly how AUC works.

AUC stands for Area Under the Curve, which curve you ask? Well, that would be the ROC curve. ROC stands for Receiver Operating Characteristic, which is actually slightly non-intuitive. The implicit goal of AUC is to deal with situations where you have a very skewed sample distribution, and don't want to overfit to a single class.

A great example is in spam detection. Generally, spam datasets are STRONGLY biased towards ham, or not-spam. If your data set is 90% ham, you can get a pretty damn good accuracy by just saying that every single email is ham, which is obviously something that indicates a non-ideal classifier. Let's start with a couple of metrics that are a little more useful for us, specifically the true positive rate (TPR) and the false positive rate (FPR):

Now in this graph, TPR is specifically the ratio of true positive to all positives, and FPR is the ratio of false positives to all negatives. (Keep in mind, this is only for binary classification.) On a graph like this, it should be pretty straightforward to figure out that a prediction of all 0's or all 1's will result in the points of (0,0) and (1,1) respectively. If you draw a line through these lines you get something like this:

Which looks basically like a diagonal line (it is), and by some easy geometry, you can see that the AUC of such a model would be 0.5 (height and base are both 1). Similarly, if you predict a random assortment of 0's and 1's, let's say 90% 1's, you could get the point (0.9, 0.9), which again falls along that diagonal line.

Now comes the interesting part. What if we weren't only predicting 0's and 1's? What if instead, we wanted to say that, theoretically we were going to set a cutoff, above which every result was a 1, and below which every result were a 0. This would mean that at the extremes you get the original situation where you have all 0's and all 1's (at a cutoff of 0 and 1 respectively), but also a series of intermediate states that fall within the 1x1 graph that contains your ROC. In practice you get something like this:

So basically, what you're actually getting when you do an AUC over accuracy is something that will strongly discourage people going for models that are representative, but not discriminative, as this will only actually select for models that achieve false positive and true positive rates that are significantly above random chance, which is not guaranteed for accuracy.


Loss function for multivariate regression where relationship between outputs matters I am attempting to build a sequential model with Keras (Tensorflow backend) that has multiple outputs. My targets are proportions of a whole so each observation is an array like [0.5, 0.25, 0.15, 0.1]. The sum of each observation's array equals 1.

When I train the model, minimizing mean squared error, I get OK results. But looking over the validation results, it's clear that mean squared error may not be the best loss function for my problem. It is important for me that the sequences within y_true and y_pred are highly correlated.

As an example, let's say I have one observation [10%, 20%, 30%, 40%]. And each of the following are potential predictions for this observation:

[20%, 30%, 20%, 30%]
[20%, 10%, 40%, 30%]
[00%, 30%, 20%, 50%]
[00%, 10%, 40%, 50%]

import numpy as np
from sklearn.metrics import mean_squared_error

y_true   = np.array([[0.1, 0.2, 0.3, 0.4]])
y_pred_0 = np.array([[0.2, 0.3, 0.2, 0.3]])
y_pred_1 = np.array([[0.2, 0.1, 0.4 ,0.3]])
y_pred_2 = np.array([[0.0, 0.3, 0.2, 0.5]])
y_pred_3 = np.array([[0.0, 0.1, 0.4, 0.5]])

preds = [y_pred_0, y_pred_1, y_pred_2, y_pred_3]

for i, pred in enumerate(preds):
    corr = np.corrcoef(y_true, pred)[0,1]
    mse = mean_squared_error(y_true, pred)
    print(f'y_pred_{i}: corr={corr:.2f}, mse={mse:.2f}')

Gives us:

y_pred_0: corr=0.45, mse=0.01
y_pred_1: corr=0.60, mse=0.01
y_pred_2: corr=0.87, mse=0.01
y_pred_3: corr=0.98, mse=0.01

So while each of these four predictions has the same error, the fourth is most preferable to me because the sequence within the predicted array is most correlated with the observation's sequence.

I've found others who have used a modified correlation coefficient function as a loss function. But if I optimize on correlation coefficient, an optimal prediction for the example observation might be [0%, 1%, 2%, 3%] since the two sequences are perfectly correlated. This doesn't work since the mean error is so large.

So I need to optimize both for high correlation and a small error. I'm unable to find anywhere where this kind of problem has been solved. Is there a way I can optimize for both of these objectives? Particularly in the Keras framework?

MSE uses the euclidean distance function. You could simply use the Pearson correlation coefficient as similarity measure, so define a new distance function that decreases when the coefficient increases ($exp(-\rho)$ for instance).

Another possibility is to use the Wasserstein distance (or Wasserstein metric, or earth mover's distance). See the definitions on Wikipedia, especially this one which is not too complicated.
This metric is designed to compute a distance between distributions (continuous) or histograms (binned / discrete). It does not relate to correlation, but rather to the shape of the vector when plotted as an histogram. This is not what you asked for but it might be what you're actually looking for.


Validity of cross-validation for model performance estimation When applying cross-validation for estimating the performance of a predictive model, the reported performance is usually the average performance over all the validation folds. As during this procedure, several models are created, one model has to be chosen as the model which is actually used for prediction on real-world samples (e.g. in a product).

I am curious whether it is really valid to report the validation performance as estimated performance of the final (selected) model (as the performance was derived using all the other models which were created during the validation procedure, but are not considered when using the final model for predictions).

I would expect that the deviation of the selected model's performance might vary drastically from the average performance of all models (depending on several factors such as the used algorithm and validation scheme).

Why is cross-validation used to estimate the performance of a predictive model despite the given fact (e.g. in many peer-reviewed scientific publications)? Wouldn't it always be better to conduct an additional performance evaluation with the selected model on an independent test set and report the resulting performance alongside the validation performance?

Cross-validation is used to estimate the performance of a certain type of model on a specific dataset.

  one model has to be chosen as the model which is actually used for prediction on real-world samples (e.g. in a product).

Selecting one of the models obtained during cross-validation is not appropriate, and proceeding in this way would indeed cause the problem that you mention. The correct methodology is to train the final model on the full training data after cross-validation (i.e. independently from the models trained during CV). This way the performance obtained through CV is representative of the expected performance of the final model.


Introducing weights into spectral clustering Suppose I have a data set with points $x_i$ and a dissimilarity measure $d_{ij}$ between each pair, as well as a weight $w_{ij}$ that qualifies the quality of this dissimilarity. I have two problems:

The first one is how to introduce weights when performing spectral clustering. Like in the Shi-Malik, Ng-Jordan-Weiss versions, where only the distances are used.
The second one is that my dissimilarity measure does not exists sometimes. In these cases, it has zero weight. How to introduce these cases in the algorithm? I suppose the zero weight solves the problem if I know how to introduce them.

An idea I had is to perform the similarity transformation as:
$$
s_{ij} = w_{ij}  e^{-\frac{d^2_{ij}}{\sigma^2}}
$$
but I feel it does not make sense since the weigth should reflect the importance of the measure, not the measure itself.

The data is not Euclidean, so I can use only this dissimilarity measure. Any reference is appreciated.

What happens in Spectral Clustering is simply finding some blocks in the data according to the property for which your matrix is defined. If it's similarity you get the similar data points.

The first one is how to introduce weights when performing spectral clustering. Like in the Shi-Malik, Ng-Jordan-Weiss versions, where only the distances are used.

Do not get confused. All what you need is to introduce your similarity measure which can be weighted. Just insert all the weights into a classical spectral clustering version (Shi-Malik or Ng or whatever) and there you go!

The second one is that my dissimilarity measure does not exists sometimes. In these cases, it has zero weight. How to introduce these cases in the algorithm? I suppose the zero weight solves the problem if I know how to introduce them.

I think this should not be a big issue. Just put zero and try your algorithm with a mock data and see what happens. If didn't work, try to fill missing values by some statistical indicators e.g. mean of the data and see the result.

Please write a comment about how it worked.

Good luck :)


Updating Google News Word2vec Word Embedding? Is it possible to update the Google News Word Embedding with a custom text dataset (text data pertaining to a particular domain) ?

Google News Word2Vec - Word Embedding clearly helps us to come with a robust set of word vectors but it unfortunately cannot be used for most business case.
For example:

embeddings.most_similar('python')

[('pythons', 0.6688377857208252),
 ('Burmese_python', 0.6680365204811096),
 ('snake', 0.6606293320655823),
 ('crocodile', 0.6591362953186035),
 ('boa_constrictor', 0.6443518996238708),
 ('alligator', 0.6421656608581543),
 ('reptile', 0.6387744545936584),
 ('albino_python', 0.6158879995346069),
 ('croc', 0.6083582639694214),
 ('lizard', 0.601341724395752)]

This output is clearly not what we want. We could create a custom word2vec model using gensim library for this business case but it would not be exhaustive (vocabulary will be comparatively less). What is best practice in such cases ? Is is possible to update the weights of a pretrained Word Embedding model so that the word embedding also learns from domain text data?

Transfer-learning is one possible approach :

Design and implement a neural net to match Google Word2Vec's design
(In terms of number of layers, activation functions and etc.,).
Pre-initialize weights with these vectors
Retrain with domain-specific corpus

This is an implementation that can be used as base and modified for step #1


Meaning of latent features? I am learning about matrix factorization for recommender systems and I am seeing the term latent features occurring too frequently but I am unable to understand what it means. I know what a feature is but I don't understand the idea of latent features. Could please explain it? Or at least point me to a paper/place where I can read about it?

At the expense of over-simplication, latent features are 'hidden' features to distinguish them from observed features. Latent features are computed from observed features using matrix factorization. An example would be text document analysis. 'words' extracted from the documents are features. If you factorize the data of words you can find 'topics', where 'topic' is a group of words with semantic relevance. Low-rank matrix factorization maps several rows (observed features) to a smaller set of rows (latent features).
To elaborate, the document could have observed features (words) like [sail-boat, schooner, yatch, steamer, cruiser] which would 'factorize' to latent feature (topic) like 'ship' and 'boat'.

[sail-boat, schooner, yatch, steamer, cruiser, ...] -> [ship, boat]

The underlying idea is that latent features are semantically relevant 'aggregates' of observered features. When you have large-scale, high-dimensional, and noisy observered features, it makes sense to build your classifier on latent features.

This is a of course a simplified description to elucidate the concept. You can read the details on Latent Dirichlet Allocation (LDA) or probabilistic Latent Semantic Analysis (pLSA) models for an accurate description.


Bidirectional GRU: validation loss stuck on plateau diverges from well performing training loss tl;dr: What's the interpretation of the validation loss decreasing faster than training loss at first but then get stuck on a plateau earlier and stop decreasing?

The accuracy behaviour is similar.

Background:

The task is multi-class document classification with a high number of labels (L = 48) and a highly imbalanced dataset. It's a NLP task, using only word-embeddings as features.

The model implemented is a Recurrent Neural Network based on Bidirectional GRU layer. The full implementation is as follows:

inp = Input(shape=(MAX_LEN,))
x = Embedding(num_words, embed_size, weights=[embedding_matrix])(inp)
x = SpatialDropout1D(0.2)(x)
x = Bidirectional(GRU(80, return_sequences=True, dropout=0.5, recurrent_dropout=0.5))(x)
avg_pool = GlobalAveragePooling1D()(x)
max_pool = GlobalMaxPooling1D()(x)
conc = concatenate([avg_pool, max_pool])

dense = Dense(num_classes)(conc)
bn = BatchNormalization()(dense)
outp = Activation("softmax")(bn)

adam = Adam(lr=LEARNING_RATE)

model = Model(inputs=inp, outputs=outp)
model.compile(loss=focal_loss,
              optimizer=adam)

Relevant insights:

Highly regularized: SpatialDropot, Dropout, Recurrent Dropout, Batch normalization...
Imbalanced sensible loss function: Focal loss
Adaptative optimizer: Adam

Training setup:

Epochs: 300
Learning rate: $1e-4$
Batch size: 32

Your training and test sets are different. If your test set has less errors than your training set then it means that the data contained in the test set is easier for the algorithm than the data in the train set. For example in the training set you have Shakespeare and in the test set you only have text with short sentences, clear structure and low vocabulary.

The effect that comes after the first phase of learning is called overfitting. The algorithm finds some logic in the data that does not really exist. For example, in the text above word "logic" is always preceded by word "some". But this rule does not really exist and you can still freely say "there is logic" without word "some". You can see this effect in how people think. Imagine you are in a vary dark forest. Your brain will see animals even if there is just a tree or the wind blew some leaves. It is better to see patterns when they do not exist than not see patterns when they exist (or the tiger will eat you). And you can see it in the training. Your model "sees" stuff that does not exist but at the same time still improves pattern recognition that really matters. This goes till epoch 150 in your data where the model starts "seeing" more stuff that does not exist and starts to lose the previously learned patterns. Non realistic patterns, that are only specific to the train set, start to overwhelm the good patterns. This is considered normal for neural networks. Almost all neural networks should stop learning before the training error becomes zero.


Building Recommendation engine with Python Which all are the equivalent or advanced libraries in Python for building recommendation systems like Mahout for  Collaborative Filtering and Content Based Filtering ?

Also is there a way to integrate Mahout with Python?

I don't know of a well documented python option for collaborative filtering, but one option is to use Spark's MLib. There is a wrapper that allows you to run it in python (pyspark) and it is straightforward to get it running on your local machine.


pandas data frame doesn't show any thing ,when view as data frame in pycharm import pandas as pd;
dataSet = pd.read_csv("winequality-red.csv");
dataSet.describe(include = 'all');

When view data set as data frame ,it show empty table.But when printing dataSet,I get following result.

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1599 entries, 0 to 1598
Data columns (total 12 columns):
fixed acidity           1599 non-null float64
volatile acidity        1599 non-null float64
citric acid             1599 non-null float64
residual sugar          1599 non-null float64
chlorides               1599 non-null float64
free sulfur dioxide     1599 non-null float64
total sulfur dioxide    1599 non-null float64
density                 1599 non-null float64
pH                      1599 non-null float64
sulphates               1599 non-null float64
alcohol                 1599 non-null float64
quality                 1599 non-null int64
dtypes: float64(11), int64(1)
memory usage: 150.0 KB
Backend TkAgg is interactive backend. Turning interactive mode on.
<class 'pandas.core.frame.DataFrame'>

Welcome to the site! Instead of describe, try print(dataSet.head()) and that should show you some of the data in your dataframe.


What do you do with one hot encoding items that are a non-match for all classes in a confusion matrix? I have trained a model for one-hot binary prediction for many classes, and am now applying it to the testing set of samples. However, a lot of the predictions for samples are 0 for every class. I'm not sure what to do with these results, as I need to make a confusion matrix (nxn for the number of classes) but I don't know where these predicted-no-class results should go. Do I just discard them? I would imagine that this would create a faulty image of the error rate of the model.

It depends on the design of your task, there are two options:

The task is regular multiclass classification, i.e. every instance must belong to exactly one class. In this case it would be a mistake to one-hot-encode the class, it can simply be encoded as an int (for example with LabelEncoder). The model will always predict exactly one class for one instance so the case of zero class is impossible.
The task is multi-label clasification, i.e. every instance can belong to zero, one or multiple classes. In this case an instance can be predicted as belonging to no class at all, this is normal. In this setting the confusion matrix should not be done with a $n \times n$ matrix across classes, because the classes are independent (btw it's not only about the case of zero class, the case of multiple classes would also be impossible to represent this way). Instead there should be one binary confusion matrix for every independent class.


How to handle overfitting in the following classification case The confusion matrix is as below :-

[[  0   0   5   1   0   0]

[  0   0  19  14   0   0]

[  0   0 217 151   0   0]

[  0   0  84 282   0   0]

[  0   0   6 111   0   0]

[  0   0   0  10   0   0]]

np.random.seed(0)
classifier = Sequential()
classifier.add(Dense(300,input_dim=11))
classifier.add(Dropout(0.5))
classifier.add(keras.layers.LeakyReLU(alpha=0.86))
classifier.add(Dense(250,activation='tanh',kernel_regularizer=keras.regularizers.l2(0.12)))
classifier.add(Dropout(0.5))
classifier.add(Dense(100,activation='relu'))
classifier.add(Dropout(0.4))
classifier.add(Dense(6,activation='softmax'))

classifier.compile(optimizer='adam',loss='categorical_crossentropy',metrics=['accuracy'])

Thanks for the help

I recommend including some more information in your post - you'll be able to get more help that way. What is the size of your dataset? How many epochs are you training for? What is the accuracy and loss value after training?

Nonethless, I recommend taking a look at the early stopping callback.

It will stop training when your model stops improving according to a given metric. I've found this useful when using loss as the metric. You should also definitely monitor the loss and accuracy of your model during each epoch and stop training when they stop improving. Tensorboard, another callback, can help with this.

You might also want to try reducing the number of nodes you have per layer, as a model with more parameters has a greater capacity for memorization, reducing your model's capability to generalize. 300 nodes is a lot to have in the first layer of your model for data with just 11 features, but to be honest everything really all depends on the data.


Explainable AI, how did the computer classified Text? My question is not about explaining the model or the algorithm
like which neurons were triggered and what are the parameters of perceptrons.
I will explain further
The problem
I have medical reports I want to analyse and extract diagnoses from it.
I built deep neural network model and extracted diagnoses with good accuracy 80%
That is good.
Now When patient look at the results and say: hmmmm, your results says I have Corona, how did the computer know?
at this moment there is no way to answer that.
The predictive model gives us the diagnoses not how it was diagnosed.
Requirements
Is there any mechanism or technique that allows us to find how did the computer made this decision?
If patient was classified with diabities, the model shoud identify words that led to the identification.
Results : Diabities

Explanation :
               the word "Thirst"
               the word "urinating frequently"
               "weight loss"
               "Fatigue"
               "Insulin"

I have been googling with not much luck and I don't where to look and what is the topic name?
Any idea how to do that or at least guidance of where I can find the answer?

There are libraries like ELI5 or LIME, which can provide explanations for text classification, here is a link to an example:
https://eli5.readthedocs.io/en/latest/tutorials/black-box-text-classifiers.html#textexplainer
import eli5
from eli5.lime import TextExplainer

te = TextExplainer(random_state=42)
te.fit(doc, pipe.predict_proba)
te.show_prediction(target_names=twenty_train.target_names)


How to prepare training data for deep learning models I am working on a project which involves the application of deep learning models. I have collected training data. In collected images, I have more than one object in interest. I am not very clear how to label these images. Should I crop and then label or just feed as it is? I would appreciate any kind of help. In the attached image, I have marked two objects.

You can create a mask image with the background having value 0 and give all pixels the value of its class.

I am not sure what approach you are taking, but you can for example use Convolutional Neural Networks. They will learn from the mask image what features contribute to a pixel class and can therefore, at inference time, output a per pixel class probability.


How to calculate probability of non independence using bayes theorem? i looked into one of the post about  naive bayes calulation of naive part

Predit the class label for instance (A=1,B=2,C=2) using naive Bayes classifcation.

Let C1 be class 1 and C2 be class 2.

For C1, by the assumption of Naive Bayesian Classifier, we have
P(A=1,B=2,C=2∣C1)=P(A=1∣C1)⋅P(B=2∣C1)⋅P(C=2∣C1)

Take P(A=1∣C1) as an example. There are 4 training records of C1, among which there are 2 records with A=1. Therefore, P(A=1∣C1)=24. Similarly, you can calculate P(B=2∣C1) and P(C=2∣C1).

It is similar to calculate P(A=1,B=2,C=2∣C2).

My query is how can we calculate the NON Naive part of this ?

P(A=1,B=2,C=2∣C1)= P(A=1∣C1)⋅P(B=2∣C1)⋅P(C=2∣C1)  -> Here the events are considered to be independent of each other

What if the events  are dependent ? How should we calcualte in that case ?

I think in practice you would make the assumption of independence in any case. Considering all the dependencies can get quite complicated very quickly and you'd need many observations. For example, in your example A=1, B=2 and C=2 was never observed,but you can't really tell, if that is because P(A=1 , B=2, C=2 | C1) is small and there is too little data, or if P(A=1 , B=2, C=2 | C1)=0. In many practical applications the approximation with independence introduces less error than not having enough observations for all possible events.

There is hybrid approaches, though, where you would model some dependence, e.g. tree-augmented naive bayes. These approaches usually perform better, but are computationally more complicated as well.


What is the state-of-the art ANN architecture for MNIST? What is actually the best neural network architecture for the classic MNIST digit classifying task? I couldn't find any that would claim to be the winner...

Actually, what I'm going to discuss is not an architecture and is like a module used in networks. It performs really well on character based data-sets like MNIST although there is other functionality for them. There is a paper called Spatial Transformer Networks written by Max Jaderberg et al. It tries to introduce an alternative for pooling layers. What it does is tries to find the canonical shape of its input by reducing transformations, like translation and rotation, or even diminishing the distortion of the inputs. It introduces a module which helps convolutional networks to really be spatial invariant. The work is amazing and the paper is easy to be read. Based on experience, using these modules may reduce the number of neurons and layers, because networks employing them won't try to learn extra stuff of the inputs, like applied transformations to the inputs. The reason is that the net will try to learn the canonical shape of inputs.

(a) shows the arbitrary input to the network (b) shows what spatial transformer has done and finally (c) is the output of the spatial transformer which can be used through other layers of the networks.

One of the significant achievements of this module is that it tries to enhance distorted inputs.

Its performance can be seen here.