Creating new classifications Is it possible for an algorithm to predict a new class that has never been before in training? For example, in my training data I have:

I need a flight to Boston --> flight query

What is two plus two --> math query

Who is the president of the United States --> Donald Trump

How can I make a model that can predict a new class for:

How are you today --> ??? (Should be something like conversation query)

I am almost looking for a way to combine supervised learning and unsupervised learning. Using supervised learning for a model to predict new classes (unsupervised learning)

Do you mean a class label that the algorithm has never seen before? Then no, it is not possible. If you dont have labelled data for all your samples you can run a LDA, to get some topics and then assign labels based on the topics obtained. Even this approach wouldnt be really great. Since your problem is classification, I would recommend restricting your domain and using only samples for which the label is known. For better classification accuracy you can look into CNN.


Efficient dimensionality reduction for large dataset I have a dataset with ~1M rows and ~500K sparse features. I want to reduce the dimensionality to somewhere in the order of 1K-5K dense features.

sklearn.decomposition.PCA doesn't work on sparse data, and I've tried using sklearn.decomposition.TruncatedSVD but get a memory error pretty quickly. What are my options for efficient dimensionality reduction on this scale?

Have you heard of Uniform Manifold Approximation and Projection (UMAP)?

  UMAP (Uniform Manifold Approximation and Projection) is a novel
  manifold learning technique for non-linear dimension reduction. UMAP is
  constructed from a theoretical framework based in Riemannian geometry
  and algebraic topology. The result is a practical scalable algorithm
  that applies to real world data. The UMAP algorithm is competitive
  with t-SNE for visualization quality, and arguably preserves more of
  the global structure with superior run time performance. Furthermore,
  UMAP as described has no computational restrictions on embedding
  dimension, making it viable as a general purpose dimension reduction
  technique for machine learning.

Check their code and original paper for list of pros and cons, it is super easy to use.

Quick Facts: UMAP can handle large datasets and is faster than  t-SNE and also supports fitting to sparse matrix data, and contrary to t-SNE, a general purpose dimension reduction technique, meaning that not only it can be used for visualisation but also for reducing the feature space for feeding into other machine learning models.

Concrete Examples: I have benchmarked the method and compared it against some other dimensionality reduction techniques benchmark notebook, if interested to have a quick look and a jump start.


Missing value in continuous variable: Indicator variable vs. Indicator value Most data has missing values, and as far as I'm aware, these are the options:

Imputation (mean, hot-deck, etc.)
Indicator variable. A categorical variable that tells what type the primary variable is. For the missing value case, this is binary. Something still has to be imputed, though.
Indicator value. If the model is powerful enough, it can learn to associate a specific imputed value to certain types of predictions.

In my case, a missing value reveals important information, and thus is Missing Not At Random. From what I've read, most imputation methods don't cover this scenario. Thus, I've opted for the indicator value approach.

My question is: Is there any point adding an additional indicator variable, since I'm already using an indicator value? Am I completely misguided and should I be looking into some other approach?

Example:

| Primary variable (-50 to 50)          | Indicator     |
|-------------------------------------  |-----------    |
| 20.5                                  | 0             |
| -14.2                                 | 0             |
|  0.1                                  | 0             |
| 500 (out of the usual distribution)   | 1             |

I can provide more information of my problem, if it's required to answer the question.

As you realise, you are introducing some form of redundancy by using both indicator values and variables; however, it might be a good starting point. Many models such as neural networks and boosted regression will be able to pick up on this, and will likely end up making use of one of the two primarily. This is because, in your particular case, the variable will be 100% correlated with the imputed indicator value.

Using both will give your model as much help as possible to learn about the missing values and their relation to the rest of the data.

My advice would be to as many combinations for which you have time. So try:

indicator value and indicator variable
indicator value
imputation via one or more methods

You can then compare the results of each of these and (assuming you have enough data to make things statistically trustworthy), you will be able to assess, which method is indeed the best in your case.

You could try thinking about how to encode perhaps a little more information into either the value or variable, in order to distinguish them from one another. E.g. you could vary the value (don't always make it 500, as per your example), depending on a metric e.g. a function of how long it has been since the previous missing value.


Equivalent of numeric encoding when rows can contain multiple values If we have a column like:

    Name
0  Alice
1    Bob
2   Dave

then, after numeric encoding, it becomes:

    Name
0  0
1  1
2  2

What if, however, we have a column like this:

              Names
0        Alice, Bob
1  Alice, Bob, Dave
2              Dave

One way to encode it would be like this:

   Alice  Bob  Dave
0      1    1     0
1      1    1     1
2      0    0     1

However, this creates a lot of extra columns. Is there a way to encode such a column in a way that doesn't end up with loads of extra columns, just like using numeric encoding instead of one-hot-encoding prevents too many columns from appearing in a column like the first one I showed?

If you're using Python, here's some code to reproduce my DataFrame:

import pandas as pd
df = pd.DataFrame({'Names': ['Alice, Bob', 'Alice, Bob, Dave', 'Dave']})

Edit: the ultimate purpose of this is to then pass this through a tree-based classifier.

The binary encoding seems to be the most natural, capturing exactly the present relationships and not introducing nonexistent relationships.  The combinations of names then are thought of as vertices of a hypercube; naturally, if you collapse this down to one dimension, you will be introducing nonexistent relationships.  Indeed, this is what happens with label encoding: Bob isn't really any closer to Alice than Dave is, but your label encoding makes that appear to be the case.

I've usually been happy enough with just the binary encoding, so I don't have much to back up the following, but some ideas:

For your case, you could still project the hypercube to one dimension; there are any number of ways to do it, and every one will add some additional information that shouldn't be there.  You could experiment and try to find one that seems most suitable for your purposes.

That should sound reminiscent of word embedding, and suggests another method: use a neural network to generate "entity embeddings", a projection from your hypercube to some smaller-dimensional space.  In this way, you uncover combinations of persons that are useful for your problem (maybe you end up with a feature that's "at least one of Alice and Dave", etc.).  You could also do something more transparent, and do some feature engineering by hand to pick out interesting combinations and binning to keep the number of features manageable.  For this you could consider target encoding each combination.  (Indeed, that's a very easy solution by itself, though target encoding can easily lead to overfitting if you have very many people or some rare combinations of people.)

Finally, you might be able to split a row containing (Alice, Dave) into two, one with Alice and the other with Dave.  You can now label-encode (or whatever else), and will need either a model that understands multiple rows per sample point, or a postprocessing method to deal with combining rows.


Layman's description of PDF and CDF Can anyone please explain what a PDF and CDF are in simple words.

(Please don't define it from wiki.)

First let's look at how these are related. The cumulative distribution function (CDF) is the cumulative sum of the probability density function (PDF).
You take the sum of all the probabilities of the previous values.

For example, if we roll a dice, then there is a 1/6th chance of getting any value. Thus the PDF is

Then the cumulative distribution function at any point is the sum of all the points before it. So the CDF of rolling a 2, is 2/6th, a 3 would be 3/6th. The graph looks like

Mathematically we define the CDF as $F$ as
$F(x) = Pr(X < x)$.
For our dice example
$F(1) = P(X=1) = 1/6$
$F(2) = P(X=1) + P(X=2) = 2/6$
$F(3) = P(X=1) + P(X=2) + P(X=3) = 1/6 + 1/6 + 1/6 = 3/6$
$F(4) = 4/6$
$F(5) = 5/6$
$F(6) = 1$.
So you can see how we get the plot above.


How to deal with errors of defining data types in pandas' read_csv ()? I have a table with 118,000 rows and 80 columns. I would like to select 8 columns from the table. I am reading the file using the pandas function  pd.read_csv command as:

df = pd.read_csv(filename, header=None, sep='|',
                 usecols=[1,3,4,5,37,40,51,76])

I would like to change the data type of each column inside of read_csv using dtype={'5': np.float, '37': np.float, ....}, but this does not work.

There is a message that column 5 has mixed types. The command print(df.dtypes) shows all columns of the type object. When I examine the column 5, I cannot see any problems. I have to change the data type for each column separately using pd.to_numeric.

My question is: Is there a way of setting data types inside read_csv and what is the problem in this case?

If you see the warning that your column has mixed types, but you only see numbers there, it could be that missing values are causing the problem.

In Pandas 1.0.0, a new function has been introduced to try to solve that problem. Namely, the Dataframe.convert_dtypes (docs).

You can use it like this:

df = pd.read_csv(filename, header=None, sep='|', usecols=[1,3,4,5,37,40,51,76])
df = df.convert_dtypes()

then check the type of the columns

print(df.dtypes)


class_weight on sklearn's DecisionTreeClassifier Can class_weight='balanced' on scikit-learn's DecisionTreeClassifier be interpreted as having identical duplicate data points for the minority classes?

I know that doesn't work that way, class_weight works as a misclassification cost. But I want to understand if it would give the same results as oversampling the minority classes.

From sklearn's documentation,

  The “balanced” mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as n_samples / (n_classes * np.bincount(y))

It puts bigger misclassification weights on minority classes than majority classes. This method has nothing to do with resampling; it modifies the misclassification cost matrix instead.

Changing the misclassification cost of each class is a different approach from resampling approaches. In my opinion, it won't give exactly the same result as oversampling the minority class. Having said that, these two approaches are both helpful to dealing with imbalanced (or unbalanced) data classification


computational time and precision trade off I know that there are a number of predictive models (generized linear ones, trees, neural network, support vector machines, knn, Naive Bayes, ...) that have been proposed to perform various analytical tasks. Now I am striving to find appropriate references about their performance when the data becames "Big". In other words, how is their performance when the data becames really big. Does the training time increase more than linear? Is there any comparative benchmark between computational time and precision when the data becames high (for the various predictive models).

A good reference can be found there http://scikit-learn.org/stable/modules/


Parallel and distributed computing What is(are) the difference(s) between parallel and distributed computing? When it comes to scalability and efficiency, it is very common to see solutions dealing with computations in clusters of machines, and sometimes it is referred to as a parallel processing, or as distributed processing.

In a certain way, the computation seems to be always parallel, since there are things running concurrently. But is the distributed computation simply related to the use of more than one machine, or are there any further specificities that distinguishes these two kinds of processing? Wouldn't it be redundant to say, for example, that a computation is parallel AND distributed?

Simply set, 'parallel' means running concurrently on distinct resources (CPUs), while 'distributed' means running across distinct computers, involving issues related to networks.

Parallel computing using for instance OpenMP is not distributed, while parallel computing with Message Passing is often distributed.

Being in a 'distributed but not parallel' setting would mean under-using resources so it is seldom encountered but it is conceptually possible.


Ways to create simulated numerical data based on small sample (using Python)? For an internship, I'm being asked to simulate the electrical consumption for virtual appliances (e.g. fridges, freezers).

The company currently has a bunch of recorded second by second data from several different appliances. The data displays strong patterns. It has only two variables - time and (W) consumed. I will need to simulate a large amount of data to train the algorithm. What are the best ways to do this in Python? I have seen some modules that have functionality in that vein but they don't seem suitable for my data format.

You can find what you need in

numpy.random

For example, generating a 100x100 matrix with normal data:

data = numpy.random.normal(mean, sigma, shape=(100, 100))

see here

[EDIT]: If you want more advanced way of generating data similar to an existing dataset, you can look into GANs.


What makes a problem good for an evolutionary strategy vs a genetic algorithm vs particle swarm optimization? I understand that evolutionary strategies (ES), genetic algorithms (GA), and particle swarm optimization (PSO) are all algorithms used to solve optimization types of problems, but what might make an optimization problem better to choose one of those techniques over the other?

For example, I read in an article that genetic algorithms are typically preferred for combinatorial problems, but other than that one sentence or so, I haven't been able to find any source that explains when you might choose one of those techniques over the other.

Is there a general approach one might use when trying to decide which technique to use, or is it more of a guess and check type of situation? It would be great if someone could help me understand when each of the three cases might be preferred.

Both ES and PSO are algorithms that have canonically been described as inherently real-valued optimization algorithms. You can (and lots of people have) formulate combinatorial variants of those algorithms, but the variants that people learn first include operators that assume floating point numbers as parameters. I suspect that's part of the answer you're looking for. If you were taught that ES was for real-valued problems, you naturally tend to think of something else as being better at combinatorial problems.

A short answer that's likely accurate is "one really isn't better than the other". We have No Free Lunch results that show this to be true over a sufficiently broad class of problems, but even in practice, GAs, ES, PSO, these are sort of fuzzy concepts that bleed into one another without sharp boundaries. I could take a GA, modify it to work on real-valued problems, and if I keep tweaking it, end up with an ES, with no clear defining boundary of when it stopped being a GA and started being an ES. And at any point along that journey, it may or may not be performing better on a specific problem. The same is true if I start from ES or PSO and make combinatorial versions. Those versions will just be better sometimes on some problems.

The deeper answer is really that we just don't know why that is. There aren't good theoretical models of why one algorithm outperforms another on a specific problem. I've not been involved in this research field for a few years now, so I'm going to be unaware of some developments I'm sure, but generally, I think this is still an open question. You can sometimes take a specific problem and try to analyze what one particularly successful algorithm is doing, but often the answer is less than completely satisfying (e.g., it turns out that my particular mutation operator is biased to jump just the right distance to escape a local optimum on this specific problem, or whatever).

I'm also aware of work done maybe 6-8 years ago to attempt to build up a dataset of problem/algorithm performance data that one could feed to a machine learning algorithm to hope for insights, but I don't think that ever reached a point where a general question like "why is this better for combinatorial problems" could really be answered.


Neural Network backpropagation taking forever I am constructing a neural network using Sigmoid neurons. My current network is  3 layers of 39 nodes each, using an input of 39 input variables across 10,000 observations.

I am calculating loss by summing the loss in each observation. That being said, I have to numerically calculate $39^2*3*10,000$ weight deltas in the network for every iteration of the code using backpropagation, starting from the output neuron and working backwards. This takes a long time.

Is there a more efficient way to approach backpropagation?

To make this question answerable, I have to assume that you have implemented back propagation correctly. That is, in terms of the underlying maths, the steps are in the right order, there is not any non-necessary repetition, and the values are calculated using correct formulae.

You can check whether the gradient values are correct by implementing gradient checking for weights, which in general you do by taking a random test case (any network architecture, random weights, inputs and target values), run your back propagation code to get your gradients, then for each weight in turn run the network forward, measure error, then adjust the weight by a very small delta, measure error again and use the difference in error to get an estimate for the gradient. If the estimate and your back propagation agree - approximately - for all weights, then your back propagation code is likely correct. If you are going to implement just one unit test for a NN implementation, this is the one to do.

To address the question directly: "Is there a more efficient way to approach backpropagation?", the answer is complicated.

The basic answer is "no". You absolutely have to do all the calculations involved in back propagation, there are no shortcuts.

However, most neural network frameworks will perform much better than your implementation, where you give an example of a network that has layers [39,39,39,1] training on 10,000 examples and taking 28 minutes to process a single epoch.

As you don't link your implementation, it is not possible to figure out precisely what the difference is. However, ignoring the possibility of bugs, the following things will probably apply:

Avoid calculating results using loops in an interpreted language. If you are looping through neuron or weight indices to multiple some values together and store back in an array in a high-level language, then this can be very inefficient. Most frameworks take advantage of fast loops and parallelisation that is possible if you frame the neural network processes (both forward and back propagation) in terms of matrix and vector manipulations. This is sometimes called vectorizaton, and most frameworks split this up so that the vector and matrix operations are black box routines implemented with strong attention to optimisation in a low-level language (like C), whilst passing the results around and deciding what to do with them can be done in higher-level language like Python or Matlab.
Mini-batches. You have to process forward and back propagation for each example seen by the network. But you don't have to apply weight updates for each example. It turns out that applying an update only once every so many examples is not only more efficient in terms of calculations, but also has benefits for convergence. Frameworks that support large amounts of parallel processing via GPU can also improve performance greatly from working in batches.

It is unlikely you will find the time to explore optimisation to the extent that the better resourced public frameworks already have. However, you can get quite far by taking advantage of vectorisation options built into your language - for instance using matrix operations in Matlab, or Numpy features in Python. This might be worthwhile so you can continue to work on and understand low-level detail in neural network models. However, at some point, if you want to work on practical problems instead of your own NN library, you will find yourself naturally switching to a third-party library because the optimisation problems and many useful additions have been solved for you already.


Pandas replace column values by condition with averages based on a value in another column I have a dataframe with people's CV data.
Among others, there's a column with years of experience, and a column with age.
Some people stated their age and experience in a way that experience > age. For example age=30 and experience=40, which is a problem.
I am trying to perform a following task:
If experience > age, replace the value of experience with an average experience of people of the same age. For example if age=30 and experience=40, replace experience with an average experience of all 30-year olds.
The problem is that I don't know how to obtain the age of a specific person in the slice df[df['experience'] > df['age']].
AFAIK it could be done by something like this:
mask = (df['experience'] > df['age'])

df['experience_cleaned'] = np.where(mask, df[df['age'] == age]['experience'].mean() ,df['experience'])

I would appreciate if you show me how it can be done, because I could use the technique to also preprocess more data in a similar way.

Get a dictionary with average age where experience < age

mapping = df[df['age']>df['experience']].groupby('age').mean().apply(list).to_dict()['experience']

Use this dictionary to replace values where experience > age

df.loc[df['age']<df['experience'],'experience'] = df[df['age']<df['experience']]['age'].map(mapping)


Feature Selection on Aggregated Targetdata I have a question about feature selection on a dataset where the target variable is aggregated by the sum of different data points. I want to predict the number of sales depending on a variety of features like:

week
price per unit
store country
store city
2-3 other categorical meta-data
other features

I am aware that this data should be interpreted as time series but because of the lack of available historical data, no model can compete with the naive approach.

The problem I am facing is, that the target variable is the sum of grouped features like month, price, country and city. If I add or remove one of the grouped features I will get some identical data points and the dataset will be inconsistent, so I have to perform a grouped aggregation (sum) to get consistency back. This operation will change the target variable and the number of data points. I have no idea how to validate regression models trained on different subsets of the features because the underlying dataset is not equal. I know there are other feature selection techniques than wrapper methods like filter or embedded which provides workarounds but I would like to know if there are techniques to resolve this issue.

Example:

    week  price_per_unit store_country  sales
0      1             3.0            C1     30
1      1             3.0            C2     32
2      1             4.0            C1     23
3      2             3.5            C1     19
4      2             3.5            C2     27
5      2             6.5            C1     35
6      3             2.0            C1     17
7      3             3.0            C1     15
8      3             4.0            C2      7
9      4             2.0            C1     19
10     4             5.0            C1     41
11     4             5.0            C2     21

After dropping the column store_country:

    week  price_per_unit  sales
0      1             3.0     30
1      1             4.0     23
2      1             3.0     32
3      2             6.5     35
4      2             3.5     19
5      2             3.5     27
6      3             2.0     17
7      3             3.0     15
8      3             4.0      7
9      4             5.0     41
10     4             2.0     19
11     4             5.0     21

Now there are duplicate data points and the sale column is wrong because I need the sum so after aggrgation I have:

   week  price_per_unit  sales
0     1             3.0     62
1     1             4.0     23
2     2             3.5     46
3     2             6.5     35
4     3             2.0     17
5     3             3.0     15
6     3             4.0      7
7     4             2.0     19
8     4             5.0     62

Let's assume I want to perform forward or backward selection with linear regression. In every step a column will be added or removed. So the number of rows depends on the feature which is selected. I can't think of a metric to compare these regression models.

It seems to me that there's a bit of confusion between these two steps:

Designing the problem and preparing the data accordingly
Applying ML methods to the data (feature selection, etc.)

The two steps must be distinct: any aggregation must take place during step 1 so that at the end of step 1 the dataset is fixed.

In your example in step 1 you decided to predict the total volume of sale across  stores given the week and price per unit. Note that you could also choose other options:

average volume of sale by store as target
keep one row for each store: depending on the goal, sometimes it's ok to have inconsistencies in the data, regression can deal with these.
you could decide that an instance for a store spans over the N past weeks and provide the sales for these weeks, the target being the week after that (that would probably help btw).
...

And of course you can do this as many times as needed for studying different problems, each time obtaining a different version of the data which represents a particular problem. It's only once you have formally defined your problem and formatted your data for it that the ML part starts. For instance stepwise regression (or any other technique) cannot work if the data is modified/aggregated during the steps.


What kind of data visualization should I use? I'm going to program a customized phone keyboard where some letters are larger than others, depending on how often I misstype them. For example, if I often pressed "w" instead of "e", I'd make the "e" button take up some of the space of the "w" button:

 (screenshot from here)

In order to figure out how often I make specific typos, I'll need to collect data and store it in some kind of graphic organizer to help me visualize it. Right now, I'm thinking of something like a cluster map. Does that sound like a good plan, or do you have any other suggestions?

Here's an example cluster map (from Microsoft PowerBI). Instead of pictures, they would say things like "KI", to show how often I type "K" instead of "I", and the circle would be larger or smaller depending on how often I make this mistake:

Thanks!

First, I think you'll need to measure when you've made a typing mistake. For example, you might log each key press and then in an analysis after, look at when you press the backspace key. If you press it only once, you might consider the key you pressed to be incorrect and the one you type after to be the correct key.

This supplies you with a truth value. It would be difficult to measure anything if you don't know what would ideally happen.

In terms of visualizing this, I would opt for a confusion matrix. There are some nice visuals provided by Seaborn, but it might look like what's in this SO answer. As you can see, each letter has a high value for itself, and maybe a couple mistakes for other letters. Looking at this plot, you might say "F" is often typed when "E" is desired. The y-axis would be the letter you intended to type, the x-axis might be the letter you actually typed. This could help you see which letters are frequently mistyped. Additionally, it would be intuitive to compute ratios off of this.

If you're not interested in which keys are mistyped as other keys, you could easily do a bar chart of key frequencies. Or a bar chart where each x-tick is a letter with proportion typed (in)correctly.


Text understanding and mapping I am looking at a problem where I have two documents, both are textual documents. The first one is a long (few pages long) textual document, while the second one contains about 10 short texts, each being a one liner. The problem is to automatically understand the 1st document and "map" key ideas from here to one or more short texts in the second document. I am not being able to get a good grip on the problem as to how to proceed. I tried the following approach:

first collected as much similar documents as possible which are relevant; I have obtained 136 such documents, each few pages long. Then build a document level embedding using each sentence as a document (doc2vec). Then for each short sentence in the second document, inferred an embedding based on the document model I built. Then tried to find the most similar sentences for each of these short texts from the 2nd document. However the results are not good. Only about 30% match is what I am getting and the  corresponding sentences are not much related to the target sentence.

I am wondering if this is the right approach or are there any other approach available which I am not aware of.

Your advice would be greatly appreciated. It may be the case that my explanation above is not adequate. Please let me know and I will try to improve it.

Your input would be greatly appreciated.

Thanks

Instead of using embeddings and doc2vec, maybe you should start by an easier and more straightforward information retrieval approach. Start by building a bag-of-words representation of your long document. Given its vocabulary $V$ of size $|V|$, you represent it as well as the short texts for the second document in a space $\mathbb{R}^{|V|}$. This will give a vector for the first document and several vectors, one for each of the lines, for the second document.

Proceed by calculating the similarity (euclidean distance) of the vectors of the lines of the second document with the vector of the first document.

There are many ways in this line of though that would enable you to use word embeddings if you want, but I would suggest starting with the simplest approach first.


Text Mapping - Medicine Names We have a problem where we have a standardized database of Medicine names.  On the other hand, there is a subset of medicine names which could have spelling mistakes, different structure or hypens, missing words etc.  There is also some metadata available, like manufacturer name, unit size etc.

Human can easily map those two database with each other.  We have used some string comparison and created some probabilistic scoring and in some cases it serves the purpose.

But lot of times we are running into lot of nuanced issues and conditions are keep getting piled up.  Is there any idea if any machine learning type of algorithm can help?  I have basic understanding of all major algorithms but yet I am drawing blank for this problem.  Simple example is mapping Epilex 300mg tab with Epilex 300 Tablet.  I can give more examples if needed.

I think there is no need to use ML for this problem. We can solve this by a simple lookup table/map table. The thing is we have to update the lookup table whenever we encountered a new category.

To apply ML to this problem. You should have the data of a set of preprocessed, processed responses like [Epilex 300mg(preprocessed), Epilex 300(processed)]. If you have this data, you can apply the decision tree(without pruning) model to predict the processed text, given an unprocessed text. Remember you will get predictions with only the categories within the trained categories. What you can do is, if a new category has occurred you can manually add that response to train data. And train your data again. If next time that category is encountered it will predict correctly. Like this you can improve your model prediction power. After some time you can able to predict every response correctly. The screen shorts of sample code is attached here.

To have actual code consider this Link.


How can I run SVM on 500k rows with 81 columns? I have about 500k rows of data. I'm new to data science and I'm trying to train a model utilizing support vector machines as part of my analysis. On my little macbook pro, it seems endless. Right now I'm using an RBF kernel and am not seeing an end to this computation.
I do however have access to 100+ compute nodes with 16 cores a piece. My problem is, I don't know how to utilize this to my extent as I am lacking knowledge on how I should approach this SVM. Right now I am using scipy.
SciPy Code
def makeModelandPrediction(trainData, trainLabel, testData):
    model = svm.SVC(
        C=1.0,
        cache_size=200,
        class_weight=None,
        coef0=0.0,
        decision_function_shape=None,
        degree=3,
        gamma='auto',
        kernel='rbf',
        max_iter=-1,
        probability=False,
        random_state=None,
        shrinking=True,
        tol=0.001,
        verbose=False,
        )
    model.fit(trainData, trainLabel)
    prediction = model.predict(testData)
    return prediction

I've already preprocessed the data and done a 70/30 train/test split on the data. Can someone point a beginner in the right direction?

If you want to stick to SVC(sklearn) , you can use Ensembling
we have many ways of ensembling
good guide is here : http://mlwave.com/kaggle-ensembling-guide/
for your problem you can train multiple SVC models on different subset of your data,and as quick solution you can combine them whit BaggingClassifier like this :

clf = BaggingClassifier(SVC(C=1.0,
        cache_size=200,
        class_weight=None,
        coef0=0.0,
        decision_function_shape=None,
        degree=3,
        gamma='auto',
        kernel='rbf',
        max_iter=-1,
        probability=False,
        random_state=None,
        shrinking=True,
        tol=0.001,
        verbose=False,
        ))

If you want to use each record is used only once for training in the BaggingClassifier, set the bootstrap parameter to False.
now you can parallel each of this models in on core of your cluster.


What criteria use in order to select the best internal validation for clustering? I am doing homework about how to evaluate a clustering algorithm both hierarchical and partitional.
For doing this I have a dataset that I can plot as you can see:

The clustering algorithms that I am using are K-Means, Gaussian mixture, and Agglomerative.
The problem is that I don't know which criteria use for selecting an internal validation for K-MEANS, for example, I read that silhouette is better for a convex cluster, but I am not able to see in the image if any of groups are or aren't convex.
I am read some papers like:

http://datamining.rutgers.edu/publication/internalmeasures.pdf
https://arxiv.org/pdf/1905.05667.pdf

but the problem is that I am not found any criteria to select A or B method for internal evaluation with facts, they're only a test about all vs all.
Where can found or what is found the criteria for choosing one or the other method for evaluation criteria?

There is no catch-all metric that can be used for evaluation (internal or otherwise) of the clustering achieved. This is why machine learning is also art. There are no hard limits, many things depend on application, domain, and data themselves.
TL;DR
The purpose of the homework is to familiarise yourself with the problem of clustering, but also with the fact that there is no definite best method nor evaluation metric as panacea for all cases.
As such, you can try various criteria (see below references) on your data and algorithm results and ponder on their effectiveness for your problem at hand.
An Impossibility Theorem for Clustering

Although the study of clustering is centered around an intuitively
compelling goal,it  has been very difficult to develop a  unified
framework for reasoning about it at a technical level, and profoundly
diverse approaches to clustering abound in the research community.
Here we suggest a formal perspective on the difficulty in  finding
such a unification, in the form of an impossibility theorem: for a set
of three simple properties, we show that there is no clustering
function satisfying all three. Relaxations of these properties expose
some of the interesting (and unavoidable) trade-offs at work in
well-studied clustering techniques such as single-linkage,
sum-of-pairs, k-means, and k-median.

Wikipedia has a nice summary of internal evaluation metrics:

Therefore, the internal evaluation measures are best suited to get
some insight into situations where one algorithm performs better than
another, but this shall not imply that one algorithm produces more
valid results than another. Validity as measured by such an index
depends on the claim that this kind of structure exists in the data
set. An algorithm designed for some kind of models has no chance if
the data set contains a radically different set of models, or if the
evaluation measures a radically different criterion. For example,
k-means clustering can only find convex clusters, and many evaluation
indexes assume convex clusters. On a data set with non-convex clusters
neither the use of k-means, nor of an evaluation criterion that
assumes convexity, is sound.
More than a dozen of internal evaluation measures exist, usually based
on the intuition that items in the same cluster should be more similar
than items in different clusters. For example, the following methods
can be used to assess the quality of clustering algorithms based on
internal criterion:
Davies–Bouldin index
The Davies–Bouldin index can be calculated by the following formula:
$$DB={\frac {1}{n}}\sum _{i=1}^{n}\max _{j\neq i}\left({\frac {\sigma _{i}+\sigma _{j}}{d(c_{i},c_{j})}}\right)$$
where $n$ is the number of clusters, $c_{x}$ is the centroid of cluster $x$, $\sigma _{x}$ is the average distance of all elements in
cluster $x$ to centroid $c_{x}$, and $d(c_{i},c_{j})$ is the distance
between centroids $c_{i}$ and $c_{j}$. Since algorithms that produce
clusters with low intra-cluster distances (high intra-cluster
similarity) and high inter-cluster distances (low inter-cluster
similarity) will have a low Davies–Bouldin index, the clustering
algorithm that produces a collection of clusters with the smallest
Davies–Bouldin index is considered the best algorithm based on this
criterion.
Dunn index
The Dunn index aims to identify dense and well-separated clusters. It is defined as the ratio between the minimal inter-cluster distance
to maximal intra-cluster distance. For each cluster partition, the
Dunn index can be calculated by the following formula:
$$D={\frac {\min _{1\leq i<j\leq n}d(i,j)}{\max _{1\leq k\leq n}d^{\prime }(k)}}\,,$$
where $d(i,j)$ represents the distance between clusters $i$ and $j$, and $d'(k)$ measures the intra-cluster distance of cluster $k$.
The inter-cluster distance $d(i,j)$ between two clusters may be any
number of distance measures, such as the distance between the
centroids of the clusters. Similarly, the intra-cluster distance
$d'(k)$ may be measured in a variety ways, such as the maximal
distance between any pair of elements in cluster $k$. Since internal
criterion seek clusters with high intra-cluster similarity and low
inter-cluster similarity, algorithms that produce clusters with high
Dunn index are more desirable.
Silhouette coefficient
The silhouette coefficient contrasts the average distance to elements in the same cluster with the average distance to elements in
other clusters. Objects with a high silhouette value are considered
well clustered, objects with a low value may be outliers. This index
works well with k-means clustering, and is also used to determine the
optimal number of clusters.

Furthermore:
An Evaluation of Criteria for Measuring the Quality of Clusters

An important problem in clustering is how to decide what is the best
set of clusters for a given data set, in terms of both the number of
clusters and the member-ship of those clusters.  In this  paper we
develop four criteria for  measuring  the quality  of different  sets
of clusters. These criteria are designed so that different criteria
prefer cluster  sets that generalise at different levels  of
granularity.  We  evaluate  the  suitability  of  these  criteria  for
non-hierarchical  clustering  of  the  results returned by a search
engine. We also compare the  number of clusters  chosen by  these
criteria with the  number of clusters  chosen by a  group of human
subjects.  Our  results  demonstrate  that  our  criteria  match  the
variability  exhibited  by  human  subjects,  indicating there is no
single perfect criterion. Instead, it is  necessary to select the
correct criterion to match a human subject's generalisation needs.

Evaluation Metrics for Unsupervised Learning Algorithms

Determining the quality of the results obtained by clustering
techniques is a key issue in unsupervised machine learning. Many
authors have discussed the desirable features of good clustering
algorithms. However, Jon Kleinberg established an impossibility
theorem for clustering. As a consequence, a wealth of studies have
proposed techniques to evaluate the quality of clustering results
depending on the characteristics of the clustering problem and the
algorithmic technique employed to cluster data.

Understanding of Internal Clustering Validation Measures

Clustering validation has long been recognized as one of the vital
issues essential to the success of clustering applications. In
general, clustering validation can be categorized into two classes,
external clustering validation and internal clustering validation. In
this paper, we focus on internal clustering validation and present a
detailed study of 11 widely used internal clustering validation
measures for crisp clustering. From five conventional aspects of
clustering, we investigate their validation properties. Experiment
results show that _ is the only internal validation measure
which performs well in all five aspects, while other measures have
certain limitations in different application scenarios.


Books about the "Science" in Data Science? What are the books about the science and mathematics behind data science? It feels like so many "data science" books are programming tutorials and don't touch things like data generating processes and statistical inference. I can already code, what I am weak on is the math/stats/theory behind what I am doing.

If I am ready to burn $1000 on books (so around 10 books... sigh), what could I buy?

Examples: Agresti's Categorical Data Analysis, Linear Mixed Models for Longitudinal Data, etc... etc...

Introductory:

Machine Learning: The Art and Science of Algorithms that Make Sense of Data (Flach)
Learning From Data (Abu-Mostafa et al.)
Introduction to Statistical Learning (James et al.)

Digging deeper:

Elements of Statistical Learning (Hastie et al.)
Pattern Recognition and Machine Learning (Bishop)

Some special interest examples:

Convex Optimization (Boyd)
Bayesian Reasoning and Machine Learning (Barber)
Probabilistic Graphical Models (Koller)
Neural Networks for Pattern Recognition (Bishop)

Broader reference works on machine learning (not really what you asked for, but for completeness):

Machine Learning: A Probabilistic Perspective (Murphy)
Artificial Intelligence: A Modern Approach (Russell & Norvig)

Bonus paper:

Statistical Modeling: The Two Cultures (Breiman)


What does updated alpha mean in LDA model? I'm trying to understand LDA model by reading through implementations of the algorithm.
Many implementations update alpha during training iterations with codes like:
psi_sum1 = 0.0e0;
psi_sum2 = 0.0e0;

for (int d = 0; d < doc_size; d++)
{
    psi_sum1 += digamma(float(n_dk[d][k]) + alpha_old[k]) - digamma(alpha[k]);
    psi_sum2 += digamma(float(doc_words[d]) + alpha_sum) - digamma(alpha_sum);
}

alpha[k] = alpha_old[k] * psi_sum1 / psi_sum2;

e.g. for 10 topics, the init alpha could be [1, 1, ..., 1] (length = 10),
after training, the alpha is updated to [0.29, 0.30, ..., 0.28] (length = 10).
What does this updated alpha mean?
Can I use the updated alpha to determine whether the topic number is appropriate?

The alpha is a hyperparameter which controls the mixture of topics for any given document. Turn it down and the documents will likely have less of a mixture of topics. Turn it up and the documents will likely have more of a mixture of topics.

Can I use the updated alpha to determine whether the topic number is
appropriate?

Tuning it depends upon your problem statement which could be solved with help of domain knowledge.


Looking for language and framework for data munging/wrangling I have an extensive series of projects that require importing data from many different types of different sources (websites, web apis, sensors, legacy text files, etc). Is there a good framework (preferably open source) that has been built bottom up to tackle this type of problem? Preferably one leveraging an appropriate language (python, most likely) and already having an extensive plugin database.

If you are interested in a very high-level (enterprise architecture) framework, I suggest you to take a look at the MIKE2.0 Methodology. Being an information management framework, MIKE2.0 has, certainly, much wider coverage than the domain of your interest, but it is a solid, interesting and open (licensed under the Creative Commons Attribution License) framework. A better fit for your focus is the Extract, transform, load (ETL) framework, which is extremely popular in contexts of Business Intelligence and Data Warehousing. On a more practical note, you might want to check my answer on Quora on open source master data management (MDM) solutions. Pay attention to the Talend solutions (disclaimer: I am not affiliated with this or any company), which cover a wide spectrum of MDM, ETL and data integration domains as open source and commercial offerings.


In a residuals vs fitted plot, how do I interpret a homoscedastic variance that is not randomly distributed above/below the line? I'm learning linear regression, and I ran a step function for linear regression and checked out the residuals vs fitted plot for the final equation. The residual looks homoscedastic but it's not randomly distributed above and below the line. I'm not sure how to interpret that. The residuals vs leverage plot doesn't indicate I have any points with a high cook's distance. My R-squared value is pretty awful...here's the summary output:

Residual standard error: 2.337 on 58588 degrees of freedom

Multiple R-squared:  0.3692,    Adjusted R-squared:  0.3671

F-statistic: 174.1 on 197 and 58588 DF,  p-value: < 2.2e-16

But I don't quite understand what this residuals vs fitted plot means. And what my next step in developing a better model would be.

This suggests that there's a nonlinearity in your data that isn't reflected in your fitted model.  You haven't told us anything about your predictor variables, but for example if you've fit a model of the form $y=ax+b$, then perhaps you could try adding a quadratic term $y=ax^{2}+bx+c$.


Dummy vectors and performance measurement for vector search Face Recognition I have about thousands of person face (from celebrity dataset LFW), which each person represented by 512 x 1 vector. I stored it on vector DB to build face searching system using embedded feature (MTCNN for face detection and arcface for embedding model). Someone suggest me to add many vectors as "dummy faces" to the database with unknown class (the number of the vectors is larger than the personal class).
It's still unclear for me why I need to add many unknown faces as "UNKNOWN" class and put it together with thousands of vector from each person. As far as I know, its pretty easy to check the performance by get the similarity score with only from known vectors (the vectors from each person) without the unknown one, for example let said if i put k = 3 or k = 5, i will take the minimum distance as the result and get the class of the vector (ID or label).

No, creating dummy unknowns is not the best way to do it.
A better approach can be, if a new face comes in, we calculate distance between vector of the new face and all of the vectors of known faces already present with us. And to identify the correct face, the minimum distance is considered. But this minimum distance should also be below a threshold value. If the minimum distance is above the threshold value, then it is considered as an unknown face. The threshold value is set manually.
Just to give you the idea, here is pseudo example -
Let's say you have 5 registered (known) faces. Their vectors are [1] [2] [3] [4] [5]. Assume that your model represents each face by a vector of shape 1 x 1.
You set the threshold value as 0.4
Your distance function is defined as abs(vector1 - vector2)
Scenario 1 -
A new face comes in, and your model generates a vector [2.2]
So you calculate the distance between new face and known faces as - [1.2] [0.2] [1.2] [2.2] [3.2]
Your minimum distance is [0.2] which is less than your threshold 0.4. Hence, this new face is identified as "[2] 2nd face" face.
Scenario 2 -
A new face comes in, and your model generates a vector [7.5]
So you calculate the distance between new face and known faces as - [6.5] [5.5] [4.5] [3.5] [2.5]
Your minimum distance is [2.5] which is more than your threshold 0.4. Hence, this new face is identified as an "unknown" face.
Lastly,
As you mentioned, you can use KNN based approach. Although it would produce very accurate results, it might not scale well. As number of faces in your database grows, KNN approach will slow down.


How can we utilize the TensorBoard I tried the sample code form the link below.
Get started with TensorBoard | TensorFlow
It pretty much worked fine; just had to make a couple tweaks to get it working on my machine. I was a little surprised to find that it ran just like normal Python code. I thought TensorFlow pops up an interactive GUI, called TensorBoard, and I thought you can interact with this visual interface. When I run the code from the link above, all I have is a command line interface. There is no TensorBoard, no graphics, no GUI, nothing but Python code. Is there a special command that need to be executed to make the TensorBoard work? Is there some additional step that I need to run? I'd appreciate any/all guidance on this.

Tensorboard is an interactive visualization tool that reads log files and helps you track experiment metrics.

For it to work with Keras's model.fit() method you need to add Tensorboard callback to model.fit() method.

log_dir="logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = tf.keras.callbacks.TensorBoard(
                                      log_dir=log_dir,
                                      histogram_freq=1)
model.fit(x=x_train,
          y=y_train,
          epochs=5,
          validation_data=(x_test, y_test),
          callbacks=[tensorboard_callback])

When you have your own training loop you can define a summary writer and use it to write a summary like this:

current_time = datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
train_log_dir = 'logs/gradient_tape/' + current_time + '/train'
test_log_dir = 'logs/gradient_tape/' + current_time + '/test'
train_summary_writer = tf.summary.create_file_writer(train_log_dir)
test_summary_writer = tf.summary.create_file_writer(test_log_dir)

for epoch in range(EPOCHS):
for (x_train, y_train) in train_dataset:
        train_step(model, optimizer, x_train, y_train)
      with train_summary_writer.as_default():
        tf.summary.scalar('loss', train_loss.result(), step=epoch)
        tf.summary.scalar('accuracy', train_accuracy.result(), step=epoch)

In any case, you will need to provide a log directory to run Tensorboard. (It doesn't pop up by itself when you train your network. You will have to run it separately. You can run it while training to monitor metrics at the end of each loop).

tensorboard --logdir yourmodel/logs/dir

If you want to make it work with Jupyter notebook you will have to load the Extention and then run magic command %tensorboard --logdir yourmodel/logs/dir

The code is taken from Tensorboard's getting started link. I hope it helps you understand how it works. Otherwise, you can visit these links. 1, 2


Data-generating probability distribution, probability distribution of a dataset, in ML In Goodfellow I, Bengio Y, Courville A. Deep learning. MIT press; 2016 Nov 10.
http://thuvien.thanglong.edu.vn:8081/dspace/bitstream/DHTL_123456789/4227/1/10.4-1.pdf

p. 102 (for example), it is said that with Unsupervised Learning, one usually wants to ''learn the entire probability distribution that generated a dataset'', $p(\vec{x})$.

My question is that I would like to have a better interpretation/understanding of the concept, i.e., can we say that $p(\vec{x})$, for a given an example (represented as a vector) $\vec{x}$, in the dataset, means the ''probability to find this example à priori'' in the wild... or something like that? E.g. if it is an image of a cat of breed X, e.g., $\vec{x_i}$ = ''vectorized image of this cat of breed X'' does $p(\vec{x_i})$ means the probability (we estimate given our limited dataset of course) to obtain this image of this cat of breed X, if we draw a sample from this dataset - and even if we want to generalize - the prob to obtain this cat if we sample the TEST SET. ?

There is a similar question here, but the answer is far from answering anything about the question:
What does it mean for the training data to be generated by a probability distribution over datasets

It's more of a theoretical distribution, that a concrete one.
The main idea is this:
we consider all data to have an underlying distribution which generates the data. Through the procedure of creating a dataset we effectively sampled some instances from it. Now I like to think of this distribution as a theoretical notion of all possible data of this type that could ever exist.
Let me give you an example:
suppose we have the cats vs dogs dataset. This dataset contains $25000$ images of cats and dogs. Now we can consider the cat images as samples from a larger population. But what would this population include? All cat images on the web? All cat images in existence? or all cat images that could ever conceivably exist? Let's refer to this as this population as $C$. This population follows a certain distribution (not every image is an image of a cat); this distribution essentially tells us what makes a cat, a cat. Now if I were to take a picture of a cat tomorrow (let's call this $c$), I would have effectively taken a sample out of this dataset (i.e. $c \sim C$).
Where does this come into play in Machine Learning?
Well, generative models essentially try to learn this distribution, and they try to do this through its samples (i.e. our dataset). They look at its samples and try to generalize to identify what distribution spawned them. Essentially they try to answer the question what makes $c$ a sample of $C$?
Furthermore, even discriminative models make several assumptions about the data (e.g. that the samples are independent and identically distributed, that the training and test set follow the same underlying distribution)
More formally

The training and test data are generated by a probability distribution over datasets called the data-generating process. We typically make a set of assumptions known collectively as the i.i.d. assumptions. These assumptions are that the examples in each dataset are independent from each other, and that the training set and test set are identically distributed, drawn from the same probability distribution as each other. This assumption enables us to describe the data-generating process with a probability distribution over a single example. The same distribution is then used to generate every train example and every test example. We call that shared underlying distribution the data-generating distribution, denoted $p_{data}$. This probabilistic framework and the i.i.d. assumptions enables us to mathematically study the relationship between training error and test error.
- I. Goodfellow et al. "The Deep Learning Book" sec 5.2

I'd suggest reading chapter 5 from this book because the authors explain a lot of well known ML concepts (bias, variance, overfitting, underfitting etc.) through the scope of this data-generating distribution.

Edit after comment suggestion:
The question is essentially,

how does the data-generating distribution fit into the training process of a neural network?

The answer isn't so obvious, mainly because neural network classifiers are discriminatve models (i.e. they don't try to identify the data-generating distribution; rather they try to find out what features separate the classes among one another). Also I'd like to add that, as stated previously, the data-generating distribution is a theoretical concept, not a concrete one employed during training.
There is a way, through, we can tie this into the whole training procedure. Initially, consider that NNs try to minimize the cross-entropy loss between its predictions $\hat y$ and the actual labels $y$:
$$
Loss(y, \hat y) = - \sum_i y_i \, log \, \hat y_i
$$
Now let's think of $y$ and $\hat y$ not as tensors but as probability distributions. The first represents the probability that a sample would belong to class $y$, while the second represents the probability with which the network thinks a sample belongs to that class.
We can take this one step further and compute the KL divergence between $y$ and $\hat y$. This metric essentially tells us the difference between two distributions (higher values mean distributions are more different).
$$
KL \left( y \|\| \hat y \right) = \sum_i y_i \, log \, \frac{y_i}{\hat y_i}
$$
Note that minimizing cross-entropy is equivalent to minimizing the KL divergence between these two distributions. If the two distributions are identical their KL divergence has a value on $0$; this value increases the more they differ.
Minimizing the KL divergence between two distributions is the same as minimizing the JS divergence between them. This is a metric derived from KL, which can be used as a distance function between distributions (i.e. *how close $y$ is to $\hat y$).
So if you think of it this way, Neural Networks are trained to minimize the distance between the actual data-generating distribution $y$ and their perception of the data-generating distribution $\hat y$.
In order for this to be achievable, some assumptions must be held:

The samples we have must be representative of the distribution (i.e. $y_i^{train} \sim y$).
The test samples we'll use to evaluate our network on must follow the same distribution (i.e. $y_i^{test} \sim y$).
The network must have sufficient capacity to learn this distribution.
The right optimization strategy needs to be followed to minimize the distance.
etc.


Would averaging two vectors in word embeddings make sense? I'm currently using the GloVe embedding matrix which is pre-trained on a large corpus. For my purpose it works fine, however, there are a few words which it does not know (for example, the word 'eSignature'). This spoils my results a bit. I do not have the time or data to retrain on a different (more domain-specific) corpus, so I wondered if I could add vectors based on existing vectors. By E(word) I denote the embedding of a word. Would the following work?

E(eSignature) = 1/2 * ( E(electronic) + E(signature) )

If not, what are other ideas that I could use to add just a few words in a word embedding?

Averaging embeddings vectors could make sense if your aim is to represent a sentence or document with a unique vector. For words out of vocabulary it make more sense to just use a random initialisation and allow training of the embedding parameters during the training of the model. In this way the model will learn the representation for the out-of-vocabulary words by itself.

Alternatively, you could use external resources like WordNet [1] to extract a set of synonyms and other words closely related to a specific term, and then leverage the vectors of those close words (averaging them might have sense but it's always a matter of testing and see what happens, as far as I know there are no grounded rules established yet).

[1] https://wordnet.princeton.edu


Why do convolutional networks work so well for images? Convolutional artificial neural networks work well in particular with images. Why?

The convolution neural networks take into consideration that an image already has a two-dimensional structure. This is a domain knowledge provided by humans and not something the neural network can pick up.

For example, consider if I would give you the raw pixels shuffled in a flattened array (or any other signal for that matter). Would you be able to reconstruct its 2D structure? Probably this would be almost impossible (unless someone builds an algorithm to solve jigsaw puzzles at the pixel level!)

Applying convolution with various strides we are looking for patterns in the patches of the image and not on the individual pixels. Using different strides we are explicitly expressing the amount of resolution that we are investigating the patterns of the image.

So given that these convolution filters are trainable we are able to have an architecture that works well with images. In fact, it would work well with any signal that would have a 2D structure.


Difference between tf.keras.backend.one_hot and keras.utils.to_categorical I'm working on a classification project and need to do one hot encoding on my data set. I'm just wondering what is the difference between tf.keras.backend.one_hot and keras.utils.to_categorical, and is one of them preferred over the other?

The difference is the input and output.
The utils.to_categorical function takes a vector as input and returns a matrix with one hot encoded rows.
The  backend.one_hot can be called as a transformation step inbetween a compute sequence of tensors.
It takes nD tensor as input and returns n+1 D one hot encoded tensor.


How does QUEST compare to other decision tree algorithms? SPSS Modeler has an implementation of QUEST, along with C&RT, C5.0 and CHAID. QUEST is relatively rarely covered in textbooks - what are its pros and cons compared to other decision tree algorithms? How does it make splits? Why is it (apparently) not in as widespread use as C&RT or C5.0?

QUEST stands for Quick, Unbiased and Efficient Statistical Tree.

It uses ANOVA F and contingency table Chi Square tests to select variables for splitting. Variables with multiple classes are merged into two super-classes to get binary splits which are determined using QDA (Quadratic Discriminant analysis). The tree can be pruned using the CART algorithm. It can be used for both classification and regression tasks.

Quest first transforms categorical (symbolic) variables into continuous variables by assigning discriminant coordinates to categories of the predictor. Then it applies quadratic discriminant analysis (QDA) to determine the split point. Notice that QDA usually produces two cut-off points—choose the one that is closer to the sample mean of the first superclass.

An advantage of the QUEST tree algorithm is that it is not biased in split-variable selection, unlike CART which is biased towards selecting split-variables which allow more splits, and those which have more missing values.

Not sure why it is not used as widely as CART or C5.0. It might be due to greater coverage of CART/C5.0 in literature than others.

References:

Quest reference manual: http://www.stat.wisc.edu/~loh/treeprogs/guide/guideman.pdf
http://www.stat.wisc.edu/~loh/quest.html


Untrained Network Giving 80% Accuracy I have a two class classification problem and my neural network prior to training predicts with an accuracy of 80%. After training i have an accuracy of 75%. Can you tell me how this is possible?

If your class imbalance is roughly 4:1, then a network that always predicts the more frequent class would have an accuracy of 80%. (That's why accuracy alone is an insufficient metric: for binary classification, you should use ROC and PRC curves. There are also more holistic scalar metrics such as MCC)
Let's say your positive samples are the frequent class. Without training, a random network might always output positive (80% accuracy). In this case, you'd only have true positives and false positives. After training, the network might start to also predict negatives; then you'll start getting true and false positives in addition to true and false negatives, which could yield a lower accuracy.


Does adding new features that contain information derivable from current features help performance? So say you have some data that consists of some values:

1.3, 0.9, 1.1

You introduce a new feature which is the average of these values: 3.3

In this example lets say that you know the average of these features is a good indicator to classify data with. My question is that, you're not actually adding anything to the model by creating this 4th feature, all the data is already in the features so does having this average help the model training identify the relationship more easily?

It depends what model will you train and depend on the new feature. However, your answer is that yes. it is possible to help the performance of that model. For example, suppose the new feature is a non-linear combination of other features (multiple of the other feature) and your model is a linear classifier. When you add the new feature might help to classify better the data depends on the context.


Retain similarity distances when using an autoencoder for dimensionality reduction I am trying to reduce the dimensionality of topic vectors (300, 1) to a two dimensional space. This has been done with various methods (e.g. t-SNE and autoencoders). A published example of reducing topic vectors to 2-dim space can be found here.

To train a test autoencoder, I took the top 10k words from Google's word vector model and tried to encode/decode them.
My autoencoder (300, 100, 2) with tanh activation functions (due to the nature of the vector elements of the word2vec model) seems to learn the vectors quickly (the loss function quickly goes negative).

However, when I compare the closest words based on the 2-dim reduction the results from the autoencoder (cosine and euclidean distance of the word presentations) I notice that they don't match the similar words provided from the original w2v model (output of most_similar).

Most similar words based on the autoencoded representation (2-dim, calculated via cosine distance). Results look random and it seems that the original distances didn't get preserved.

>>> extract_from_embed('car')  # extract method for my own embedding
[(football', 0.91979183998553593), (u'\xa9', 0.9515135906035519), (u'Thank', 0.96150527440098321), (u'innings', 0.96893565858480013), (u'thank', 0.9699300787802696), (u'balls', 0.97004978663463826), (u'Admission', 0.97050270191042776), (u'thanked', 0.97098601610349322), (u'Announces', 0.97186348679591361), (u'drills', 0.97214077185079129)]

Word2Vec most similar words based on 300, 1 vectors

>>> word_vectors.most_similar(positive=['car'])  # std gensim method
[(u'vehicle', 0.7821096777915955),
 (u'cars', 0.7423831224441528),
 (u'SUV', 0.7160963416099548),
 (u'truck', 0.6735789775848389),
 (u'Car', 0.667760968208313),
 ...
 (u'automobile', 0.5838367938995361)]

Is there a way to "preserve" the relative distances between vectors when I used an autoencoder? I understand that the absolute distances differ between vectors from a 2 and 300 dim space. But can I preserve the relative distances between the vectors? Is this even possible?

No it is not possible to preserve relative distances when reducing dimensions for arbitrary data. This is not due to a property of auto-encoders compared to e.g. PCA or T-SNE. It is due to geometry.

You can see this relatively easily by considering a reduction of dimensions from 3 to 2, and examining a tetrahedron where all four corner points are 1 unit apart. There is no two-dimensional shape that can place four points mutually equidistant (unless perhaps you consider non-Euclidean spaces). It should be clear that relative distances near those corner vertices would also be affected, and thus this special shape demonstrates a general property of dimension reduction.


Where is the rotated angle actually located in fitEllipse method? Main task: I fit the ellipse using the fitEllipse() method and then I'd like to count the rotation angle between the horizontal axis and the major axis of the generated ellipse. I'm going to do this using the 3rd returned argument from the fitEllipse() method - $\theta$ (rotation angle).
Main issue: I can't find exact information about which axes this angle is located between.
Other: If I'm right the length of minor axis and the length of major axis in ellipse it's the same lengths as two sides in a rotated rectangle.
Sources:

From documentation here (section nb. 9) it seems that this angle is between horizontal axis and the first side how it's written in CvBox2D section. So, it means that the angle can be between the horizontal axis and minor or major axis.
But:
In this article (section 3) the first example is good but in the 2nd example the angle should be between horizontal axis and height instead of width
(referring to my aforementioned first point).
In this article it shows that the angle is between vertical axis and one side of rectangle.

So, where is the rotated angle located in fitEllipse() method?
Any hints how it works are welcome.

I've implemented the following simple code:
import cv2
import numpy as np

nr_im = 9876
font = cv2.FONT_HERSHEY_SIMPLEX
fontScale = 1
colorText = (0, 0, 255)
thickness = 2

img = cv2.imread('testing/' + str(nr_im) + '.jpg')
original = img.copy()
blured_img = cv2.GaussianBlur(img,(17,17),5)

image = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
lower = np.array([0, 0, 140], dtype="uint8")
upper = np.array([0, 0, 255], dtype="uint8")
mask = cv2.inRange(image, lower, upper)

# Morphological Closing: Get rid of the noise inside the object
mask = cv2.morphologyEx(mask, cv2.MORPH_CLOSE, cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (25, 25)))

# Find contours
cnts, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)

print(len(cnts))

cntsElps = []
for num_cnt, cnt in enumerate(cnts):
    genElipse = cv2.fitEllipse(cnt)
    cntsElps.append(genElipse)
    cv2.ellipse(original,genElipse,(0,255,0),2)
    cv2.putText(original, str(num_cnt+1), (int(genElipse[0][0]),int(genElipse[0][1])), font, fontScale, colorText, thickness, cv2.LINE_AA)
    print("Ellipse nb: " + str(num_cnt+1) + " has angle: " + str(genElipse[2]) + "\n")

cv2.imwrite('testing/' + str(nr_im) + '_' + 'trash2' + '.png', original)

And I used this image as example:

I've got the following image result:

And the rotation angle for each ellipse was:

Ellipse nb: 1 has angle: 55.63788986206055
Ellipse nb: 2 has angle: 108.58539581298828
Ellipse nb: 3 has angle: 170.23861694335938
Ellipse nb: 4 has angle: 73.59089660644531

So, my conclusion is that an angle between horizontal axis and major side of rectangle(=major ellipse axis) is the rotation angle in fitEllipse() method.


Python - Appending a new Dataframe column that is a function of two separate numerical columns I have a dataset that gives me the geographic coordinates of residential properties.

My goal is to append a new column to this dataframe that shows the properties' distances to various prominent locations in the city. I plan on doing this by finding the difference in the latitude and longitude to give X and Y values, and using the pythagorean theorem to find the straight-line distance between the two locations.

How do I structure the code so that the new column is a pythagorean function of the differences between latitude and longitude?

Thank you.

Use the distance matrix to know the distance from one point to all other corresponding points.

Here is an example:

def dist_PQ(P,Q):
    PP=P*P
    QQ=Q*Q
    PSquared_Sum = np.sum(PP,axis=1,keepdims=True)
    QSquared_Sum = np.sum(QQ,axis=1,keepdims=True)

    squared_distance = PSquared_Sum + QSquared_Sum.T - 2*np.dot(P,Q.T)

    return(squared_distance)

Use the above function to find distances between two sets of points say (P and Q)
Say P has 10 points the values are as numpy array [[x11,y11],[x12,y12],[x13,y13]...] and Q has 3 points as [[x21,y21],[x22,y22],[x23,y23]].
pass P and Q to the dist_PR function to get the output as (10,3) matrix which gives the distances from all points in array P to all points of Q.

Now we can use the matrix of distance in our further calculation.
Suppose we want to find the place which is nearest from P1 (a point in P), we can simply find the minimum distance value in row number 1 of that distance matrix.


Approach fpr extracting/cropping features images using deeplearning and no annotations Let's say I want to have a bunch of images of hats from videos. How would I priniciple build something that would learn to recognize, and crop or bound box hats? I heard you need a dataset with bounding boxes manually drawn for training, but it seems there would be a way for a nn to identify that on their own?

I'm trying to understand the possibility of scraping video for different items. I.e. give it images of 1000 hats, and then it will crop out images of hats from a bunch of video files.

I am thinking this could be an interesting thing to work on, but would need some advicein terms of how to arrpoach it.

Also, the next logical thing is then to put hats on people in movies somehow, but that would be phase 2.

Thanks

So all you have is a set of images containing only cropped hats?
One idea is to leverage synthetic data to learn an object detector.
It would be better if you had crops of natural images with hats, not just hats alone (as it will lessen the domain shift).

Basically, the following: (1) take your hat images and generate "ground truth" images by randomly pasting a random number of hats (in random poses) into a random background image (from anywhere, ideally similar to the backgrounds of images that you plan to run on at test time). (2) Train an object detector model (e.g. Mask-RCNN, Faster R-CNN, or YOLO9000) using your synthetic dataset (since you know the "real" hat positions in the images, since you made them). (3) given a video, decompose it into a sequence of image frames, and run your trained detector frame-by-frame. (4) use the output of the detector to obtain the hat positions and crop them out.

A simple extension would be to generate synthetic videos for training, where your hats move around and you keep track of them; another one might be to try to make the synthetic images more realistic, by say attaching random people or at least heads underneath the hats.

The main challenge is the domain adaptation between synthetic and real images. Any of the many modern techniques for handling domain shift would likely be helpful. For instance, using GANs (also here).

It may interest you to know that this simple synthetic methodology has proven useful in practice. For instance, Tremblay et al at NVIDIA recently used it to also train an object detector via domain randomization. Other applications include robotics.

A much more trivial, but easier to implement solution is to train a hat classifier $C$, which takes in an image or image patch, and output whether the patch contains a hat or not. Using your cropped hat dataset as the true positives, and getting true negatives by randomly cropping patches from any random dataset of images, you can train $C$. Given a video, you then simply use a sliding window approach, where for every image patch at every frame, you run $C$ on it. You can then threshold the output of $C$ and/or crop out areas with high values from the classifier.


k-means with one cluster K-means may give different results, because the initial choice of centroids is random.

However, if I were to choose k=1, will the algorithm always provide the same answer equal to the "barycentre" of my data?

Yes. The centroid will converge to the center of all your data and this will occur in a single iteration. This is due to all the data points belonging to a single centroid, thus it will be centered according to all these instances immediately.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans

# Generate data
class1 = np.random.randn(1000, 2)

class2 = np.random.randn(1000, 2)
class2[:,0] = 4+class2[:,0]
class2[:,1] = 4+class2[:,1]

class3 = np.random.randn(1000, 2)
class3[:,0] = -4+class3[:,0]
class3[:,1] = 4+class3[:,1]

data = np.append( class1, class2, axis= 0)
data = np.append( data, class3, axis= 0)
print(data.shape)

# Plot the data
plt.scatter(data[:,0], data[:,1])
plt.show()

# Cluster
kmeans = KMeans(n_clusters=1, random_state=0, verbose = 1).fit(data)

# Plot clustered results
plt.scatter(data[:,0], data[:,1], c=kmeans.labels_)
plt.scatter(kmeans.cluster_centers_[:,0], kmeans.cluster_centers_[:,1], c = 'r')
plt.show()

# Show the cluster centers
print(kmeans.cluster_centers_)

  Initialization complete Iteration  0, inertia 81470.055 Iteration  1,
  inertia 48841.695 Converged at iteration 1: center shift 0.000000e+00
  within tolerance 8.140283e-04


Increase accuracy of classification problem I am trying to build a classifier that predicts the compiler given some operations of assembly code. Here is the pandas dataframe:

What I do is using a TfidfVectorizer and select the features that have most predictive power by doing:

tfidf_vectorizer=TfidfVectorizer(max_features=500)

so using max_features=500 to select the 500 features with the highest idf. The problem is that the accuracy is still low, infact it is around 0.69. I would like to arrive at least at 0.9, but I dont't know what else do.

I am using support vector machines and this gives me the accuracy of 0.69. I also tried random fores and I was around 0.75.

My code is the following:

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf_vectorizer=TfidfVectorizer(max_features=1000)
df_x = df['opcodes']

X_all = tfidf_vectorizer_vectors=tfidf_vectorizer.fit_transform(df_x)
y_all = df['compiler']

X_train, X_test, y_train, y_test = train_test_split(X_all, y_all,
      test_size=0.2, random_state=15)

from sklearn import svm

model = svm.SVC(kernel='linear', C=1).fit(X_train,y_train)

y_pred = model.predict(X_test)

acc = model.score(X_test, y_test)
print("Accuracy %.3f" %acc)

moreover the dataset is balanced, infact I have 3 compilers with 1000 samples each.

I don't know what other startegy to try to increase accuracy and get at 0.9.

Can somebody please help me?  Thank's in advance.

I'm not familiar with scikit but I'm assuming that TfidfVectorizer represents bag of words features right? By this I mean that it treats all the instructions in an instance as a set, i.e. doesn't take into account their sequential order.

I'm also not familiar with compilers but I'm guessing that the order of the instructions could be a relevant indication? I.e. a compiler may generate particular sequences of instructions.

Based on these remarks I would try to represent instances with n-grams of instructions rather than individual instructions. Then you can still use some kind of bag-of-ngrams representation, possibly with TFIDF, but I would start with simple binary or frequency features. A simple feature selection step with something like information gain might be useful.

[edit] N-grams take order into account locally. In a bag of words model, words (or instructions in your case) are considered individually of each other: for instance the sequence push, push, mov is the same as push, mov, push. With bigrams this sequence would be represented as (push,push), (push,mov) whereas the second one is (push,mov), (mov,push). This means two things:

Higher level of detail about the instance, which can help the model capture the relevant indications
More features so higher risk of overfitting (the model taking some random details as indication, which lead to errors on the test set).


deepAR RNN from AWS Sagemaker - should I clean the data first? I test the Sagemaker AWS solution for RNN: deepAR.
Previously I used sklearn for this and obviously I cleaned the data to avoid highly correlated time series (KBest, PCA, etc.)
Do you think it is needed here, or RNN will handle internally correlated series.
PS. This is typically the case with blackbox commercial solutions. They look fantastic, but when it comes to details you have nowhere to consult.

I wouldn't do much to the data before using as input to DeepAR. I would just make sure the timeseries I'm using as input have some variance (are not stationary for long).

DeepAR uses a simple yet efficient sampling technique to handle timeseries of different velocity.

DeepAR's paper state some of the capabilities the architecture should handle. https://arxiv.org/abs/1704.04110

In any case, it is always a responsible thing to try the simple first and use that as a baseline.


Are parquet files compressed? Parquet File Format Hadoop. Parquet, an open-source file format for Hadoop. Parquet stores nested data structures in a flat columnar format. Compared to a traditional approach where data is stored in the row-oriented approach, parquet is more efficient in terms of storage and performance.

What is the advantage of a parquet file?
Are parquet files compressed?

Yes, its compressed, read more for advanteges:

Parquet, is an open source file format for Hadoop, it stores nested data structures in a flat columnar format. Compared to a traditional approach where data is stored in row-oriented approach, parquet is more efficient in terms of storage and performance. Parquet stores binary data in a column-oriented way, where the values of each column are organized so that they are all adjacent, enabling better compression. It is especially good for queries which read particular columns from a “wide” (with many columns) table since only needed columns are read and IO is minimized.

When we are processing Big data, cost required to store such data is more (Hadoop stores data redundantly I.e 3 copies of each file to achieve fault tolerance) along with the storage cost processing the data comes with CPU,Network IO, etc costs. As the data increases cost for processing and storage increases. Parquet is the choice of Big data as it serves both needs, efficient and performance in both storage and processing.

To conclude main advantages of parquet:

Organizing by column allows for better compression, as data is more homogeneous. The space savings are very noticeable at the scale of a Hadoop cluster.
I/O will be reduced as we can efficiently scan only a subset of the columns while reading the data. Better compression also reduces the bandwidth required to read the input.
As we store data of the same type in each column, we can use encoding better suited to the modern processors’ pipeline by making instruction branching more predictable.


Running examples from scikit-learn tutorials I am new to scikit-learn. I went through the examples given in the docs and I downloaded the script for recognizing images of hand-written digits. When I made the script to run on my laptop, I got the following errors:
Traceback (most recent call last):
   File "C:\Python34\plot_digits_classification.py", line 22, in <module>
    from sklearn import datasets, svm, metrics
    File "C:\Python34\lib\site-packages\sklearn\__init__.py", line 57, in<module>
    from .base import clone
   File "C:\Python34\lib\site-packages\sklearn\base.py", line 11, in <module>
  from .utils.fixes import signature
  File "C:\Python34\lib\site-packages\sklearn\utils\__init__.py", line 11, in <module>
   from .validation import (as_float_array,
   File "C:\Python34\lib\site-packages\sklearn\utils\validation.py", line 16, in <module>
   from ..utils.fixes import signature
  File "C:\Python34\lib\site-packages\sklearn\utils\fixes.py", line 324, in <module>
   from scipy.sparse.linalg import lsqr as sparse_lsqr
   File "C:\Python34\lib\site-packages\scipy\sparse\linalg\__init__.py", line 109, in <module>
   from .isolve import *
  File "C:\Python34\lib\site-packages\scipy\sparse\linalg\isolve\__init__.py", line 6, in <module>
  from .iterative import *
  File "C:\Python34\lib\site-packages\scipy\sparse\linalg\isolve\iterative.py", line 7, in <module>
  from . import _iterative
  ImportError: DLL load failed: The specified module could not be found.

Please help me. Also, I want to know if I want to load data for prediction, how should I do that? For example, if I want to test a hand written digit that is stored somewhere on my disk, how to prepare that data for loading and passing into this model for prediction?

Seems like you've got to install the module, for example with the pip package manager. Perhaps you could install anaconda, which is highly recommended


Is any feed-forward neural network a composable function? I am wondering if each feed-forward neural network can be written as a composition of the layer functions.
So if $F(x)$ is the output of a neural network F, do always exists layer functions $f_1$,...,$f_n$ with
$F(x)$ = ($f_1$ o .. o $f_n$)$(x)$? Here "o" denotes the composition symbol.
If the network consists only of layers where each layer is connected to exactly one previous layer, this is obvious. However, this is not always the case, e.g. if you have skip-connections.
So what about the "concat" layer and the "sum" layer?
So, what if $F(x)$ = $G_1(x) + G_2(x)$, or $F(x) = [H_1(x), H_2(x)]$, for some layer functions $G_1$,$G_2$,$H_1$,$H_2$?
Can $F$ still be written as a composition?

I suppose you could write $ h\circ [g\circ f\ |\ f]$ if you argue that matrix augmentation is itself a multivariate function.
But to avoid pedantic arguments, you could use multivariate function composition (although it is uglier).
Define a multivariate concatenate function for column vectors:
$ c(\vec{x_1},\vec{x_2},...,\vec{x_n})=\left[
\begin{matrix}
\vec{x_1} \\ \vec{x_2} \\ \vdots \\ \vec{x_n}
\end{matrix}
\right]$
In a network $f\rightarrow g\rightarrow h$ having skip connection $f\rightarrow h$, you could then write: $ h\circ c|_{g\circ f,f}$.


How to keep track of assumptions/design choices and avoid errors in complex projects I am part of a small team data analysts that has been trying to optimise the process we use to keep track of the assumptions and design choices that are made over the lifetime of a complex data science project (several months) - ultimately we want to (i) catch (conceptual) errors when they are made (ii) ensure that it is at all times clear what assumptions underlie the results being produced.

Our current process relies on code-generated and manually maintained excel spreadsheets - the process is (i) time consuming/results in too many files, (ii) introduces a new source of error (the copying of results into the spreadsheet) and (iii) is not dynamic, i.e. cannot easily be adapted as a project evolves.

While there are a large number of questions/answers focused on version control, reproducible research and project organisation more broadly we are not looking for a new suite of tools but a light-weight process that addresses a very specific problem.

The ideal process:

Is easy to  maintain & as much as possible automated to reduce
errors
Captures all design choices and assumptions made
At a glance reflects the impact of any changes, i.e. reflects (changes
in) the 'key metrics' that resulted from the change
Links all changes with the related code file versions
Can easily be shared with other analysts & Principal Investigators

I would go for parametrised Rmarkdown reports. I think it adresses all your needs, it's simple to use and maintain, offers great flexibility, you can embed all kinds of things and it offers different output formats.


Keras LSTM Dimensions I am trying to build a RNN in Keras.
 I am inputting an array of 300K values.
 I have 4 independent variables (W,X,Y,Z) And 1 dependent variable f(W,X,Y,Z).

The array is then split 270K for training and 30K for validation.

When I try to put my data into the network, it says "expected ndim=3, found ndim=4".

My model looks like this

model=Sequential()
model.add(LSTM(4, input_shape=(270000,4)))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer ='adam')
model.fit(X_tr,Y_tr, validation_data=(X_val, Y_val),
          epochs=10, batch_size=30000, verbose=1)

When creating an RNN, we generally assume there is some temporal correlation, for example: that the data has a time-series nature, like the price of a stock. We might consider a window of 30 days as the timestep for a single sample.

When creating input data for an LSTM layer, you need to consider how many timesteps are included in one of your samples, and put the data into that shape. The input shape should be (number_of_sample, num_timesteps, num_features):

Take a look at this answer for more information.

Your data then might be able to be reshaped:

new_shape = data.reshape((None, 100, 4))       # new input shape to LSTM

None means you let the number of samples be whatever it must in order to fit your original data shape.
100 means you pass 100 sequential datapoints/timesteps as a single sample
4 is your number of independent variables (features)

That reshape might throw an error if your data cannot be reshaped to make the dimensions you want. You might need to trim your data (i.e. leave some of teh oldest samples out)


How to structure and train an LSTM for real-time predictions in Keras? I am trying to create an AI to play a 3D melee fighting game using an LSTM. The NN receives as input the relative positions of the enemies' joints (e.g. head, hands, legs) and should output the correct movements for each timestep. Currently, I am using a vanilla feed-forward NN made with Keras and it performs well, but it needs information from multiple timesteps to be more effective.
I am now trying to implement an LSTM for this problem. I want to receive an output at each timestep rather than feeding in a whole sequence of data before receiving output.
I believe that a stateful LSTM is suited for this task, but I am unsure of how to train it and when to reset the state. Ideally, the AI would have no fixed number of timesteps of memory and would just be able to continuously look back as it's making predictions in real time for each timestep, without having its state reset abruptly during fights.
My training data is one long, continuous sequence of enemy positions (input) and keystrokes (output) for each timestep. How would I structure and train an LSTM to learn from this data to make real time predictions at each timestep using Keras?

The tough part about this problem is evaluating what moves are "correct" per se. In a fighting game sequence, there may be 2 or more moves which both work in theory in the current frame. If you are optimizing for knocking the other character out of frame, it would be useful to build a reward set which optimizes for this.
It may be worthwhile to build a data set that takes into account corresponding stick positional data and then labeling past and future time locations as advantageous/disadvantageous based on simulation, similar to a chess computer.
First you have to build something that can identify perfect combos- situations in which the enemy, regardless of their inputs, has no reliable way to escape the combo which results in high % or death. And then the problem becomes easier because the ai can just identify positions to set up this combo, and then executes them.
Train a cpu to combo via reinforcement, and then integrate with the identifier, and you'll have a pretty damn good bot.


Is there any other probability distribution model than gaussian for multivariate data Whenever we talk about the probability distribution of data having more than one feature, we have only one option i-e multivariate normal distribution. Is there any other probability distribution model exist for multivariate data? If yes, then how can we find its parameters using MLE in MATLAB

There are quite a number of multivariate distributions, in theory. At the outset, we categorize them as multivariate discrete distributions and multivariate continuous distributions.

To start with, you may explore the following link for :
multivariate discrete distributions and the following for multivariate continuous distributions.

There are entire volumes printed on the subject, for example, books by:  N. Balakrishnan, Norman L Johnson, and Samuel Kotz and these books are published by John Wiley and Sons, Inc.


Difference between normalization and zero centering I am working on some pre-processing for lung CT images. I see a nice tutorial in here. Two of them are normalization and zero centering. I wonder what is the difference between these two steps? If one normalizes the image pixel values to range [0, 1], is it have a benefit to also do zero centering? Is there a difference if I do normalization first or zero centering first?

what is the difference between these two steps?

In that specific Notebook that you linked, normalization means: shrink a numerical distribution in the [0,1] interval. It is commonly referred to as Min-Max Scaling. Shrinking the distribution in the [0,1] interval moves its mean somewhere between 0 and 1.

Zero-centering instead means: to "shift" the values of the distribution so that its mean is equal to 0.

--

  if one normalize the image pixel values to range [0, 1], is it have a
  benefit to also do zero centering?

Zero centering variables improves ML models performance. Activation functions tend to be very responsive to weights' changes especially around zero. I found some good explanations here.

--

  Is there a difference if I do normalization first or zero centering
  first?

The order matters, since "normalization" would move the mean away from zero.


R's mice imputation alternative in Python What is Python's alternative to missing data imputation with mice in R? Imputation using median/mean seems pretty lame, I'm looking for other methods of imputation, something like randomForest.

You can use the library HERE. You are able to impute the values of your dataset using linear models with an Sklearn type interface.