Weighted Binary Cross Entropy Loss -- Keras Implementation I have a binary segmentation problem with highly imbalanced data such that there are almost 60 class zero samples for every class one sample. To address this issue, I coded a simple weighted binary cross entropy loss function in Keras with Tensorflow as the backend.

def weighted_bce(y_true, y_pred):
  weights = (y_true * 59.) + 1.
  bce = K.binary_crossentropy(y_true, y_pred)
  weighted_bce = K.mean(bce * weights)
  return weighted_bce

I wanted to ask if this implementation is correct because I am new to Keras/Tensorflow and the optimizer is having a hard time optimizing this. The loss goes from something like 1.5 to 0.4 and doesn't go down further. Normal binary cross entropy performs better if I train it for a long time to the point of over-fitting. Before anyone asks, I cannot use class_weight because I am training a fully convolutional network.

The code is correct. The reason, why normal binary cross entropy performs better, is that it doesn't penalize for mistakes on the smaller class so drastically as in weighted case. To be sure, that this approach is suitable for you, it's reasonable to evaluate f1 metrics both for the smaller and the larger classes on the validation data. It might show that performance on the smaller class becomes better. And training time can increase, because the model is forced to discriminate objects of different classes and to learn important patterns to do that.


Generating Dinosaur names with Tensorflow RNN I try to adapt "Text generation with an RNN" tutorial to generate new dinosaur names from a list of the existing ones. For training RNN tutorial text is divided into example character sequences of equal length:

# Read, then decode for py2 compat.
text = open(path_to_file, 'rb').read().decode(encoding='utf-8')
# The unique characters in the file
vocab = sorted(set(text))
idx2char = np.array(vocab)
# Creating a mapping from unique characters to indices
char2idx = {u:i for i, u in enumerate(vocab)}
# The maximum length sentence we want for a single input in characters
seq_length = 100
examples_per_epoch = len(text)//(seq_length+1)
# Create training examples / targets
char_dataset = tf.data.Dataset.from_tensor_slices(text_as_int)
# Convert to sequences of the same length
sequences = char_dataset.batch(seq_length+1, drop_remainder=True)
# Sequences as text
for item in sequences.take(2):
    print("----")
    print(repr(''.join(idx2char[item.numpy()])))

Output:

----
'First Citizen:\nBefore we proceed any further, hear me speak.\n\nAll:\nSpeak, speak.\n\nFirst Citizen:\nYou '
----
'are all resolved rather to die than to famish?\n\nAll:\nResolved. resolved.\n\nFirst Citizen:\nFirst, you k'

My problem differs from tutorial in that I have a list of names of different length instead of monolith of text:

aachenosaurus
aardonyx
abdallahsaurus
abelisaurus
abrictosaurus
abrosaurus
abydosaurus
acanthopholis

In my case character sequences are names. As long as I can't train RNN on sequences with different length (please, correct me if I am wrong here) I need to pad all my names with spaces to a size of a longest name, which is 26.

My longest name is lisboasaurusliubangosaurus, so, for example, aardonyx shoud be padded as:

"lisboasaurusliubangosaurus"
"aardonyx                  "

I tried to pad my sequences with:

# Convert individual characters to sequences of the desired size.
sequences = char_dataset.padded_batch(seq_length+1, padded_shapes=seq_length, drop_remainder=True)

Which results in error:

ValueError: The padded shape (26,) is not compatible with the corresponding input component shape ().

Questions:

Is it possible to train Tensorflow RNN with sequences of variable length?
How to pad short sequences?

Thanks!

Check out this answer:

https://stackoverflow.com/a/60230236/12642230

Alternative Solution:

Tensorflow provides a method pad_sequences() to do that:

https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/sequence/pad_sequences

The default value of padding is 'pre', you might wanna change that to 'post' to do what you want, along with providing the maximum length, which is 26 in your case. You also would need to add a special padding character to you dictionary of characters to indices, and use its index to provide padding value for the method.


Data repositories like UCI Are there any other data repositories like UCI and mlData, for biological data?? I want to know about mostly biological data set.

I do biological network analysis and here are some networks.


Complex HTMLs Data Extraction with Python Does anybody know a way of extracting data with python from more convoluted website structures? For example, I'm trying to extract data from the players in the ATP profiles, but it's just so complicated I quit. I think they're pulling data from some database in the script and I suspect that even if I tried I wouldn't be able to get it.

I then started using a specialized software called ParseHub, which pulls the data somewhat visually. It's a pretty good software, but they make it slow on purpose just so you buy it, and it is particularly not cheap.

I ended up using BeautifulSoup to do the job. The final code is not that clean - as I'm kind of a newbie - but it does what I wanted. You can find the source code and the dataset I've extracted so far in this repo. You can also check out an article I did on this in my website: fanaro.com.br.


What are recommended methods for multi-task prediction? Currently, we are working on a school project which is trying to predict the number of crimes in some area/neighbourhood.

There are 8 different categories for crimes and we've tried to find the correlation among those categories and now we only have 4 left.
Instead of building a model for each category, we want to predict these 4 categories simultaneously by some multi-output algorithm.

Our sample size is around 27,000 for 6 years (from 2011 to 2016, 4000+ for each year). We are going to use (maybe) cross-validation to build/test our model.

Would you please list 2-3 algorithms which already have fully or partially implemented library in Python (preferred) or R you would recommend to use with our dataset scale?
I only found scikit-learn with this algorithm. But it's for classification rather then prediction numbers.

This is a intro-level ML course project, the group is not very experienced in the field and the time is limited so we don't want to implement an algorithm from scratch.

For the project, do you need to explain exactly how the input variables gave you the predictions? If so, trees based methods (R library 'tree', 'rpart') and Logistic Regression would be easy to explain to your stakeholder / lecturer.

If this explanation is not that important, or you could get by just showing the relative predictor importance, and you'd like to focus on accuracy. XGBoost is my default recommendation for multi-class predictions that I use at work on a daily basis. If XGBoost has too many parameters for your liking, you could try randomForest which only has 2 main parameters to tune. Both XGBoost and randomForest have functions for variable importance measures. (R Library XGBoost and randomForest)


Score matrix string similarity I have a load of documents, which have a load of key value pairs in them. The key might not be unique so there might be multiple keys of the same type with different values.

I want to compare the similarity of the keys between 2 documents. More specifically the string similarity of these values. I am thinking of using something like the Smith-Waterman Algorithm to compare the similarity.

So I've drawn a picture of how I'm thinking about representing the data -

The values in the cells are the result of the smith-waterman algorithm (or some other string similarity metric).

Image that this matrix represents a key type of "things" I then need to add the "things" similarity score into a vector of 0 or 1. Thats ok.

What I can't figure out is how I determine if the matrix is similar or not similar - ideally I want to convert the matrix to an number between 0 and 1 and then I'll just set a threshold to score it as either 0 or 1.

Any ideas how I can create a score of the matrix? Does anyone know any algorithms that do this type of thing (obviously things like how smith waterman works is kind of applicable).

If your goal is to transform your matrix into a number (your similarity measure), you may want to use a matrix norm.

For instance, using the Frobenius norm on your example would return 1.488086.


Very Deep Convolutional Networks for Text Classification: Clarifying skip connections Question RE this research paper if anyone has experience with CNN's, pooling & skip connections: https://arxiv.org/pdf/1606.01781.pdf

In figure 1, the input to the first convolutional block has shape (batch_size, 64, s)
The output from block 1 must be (batch_size, 64, s)
The output from block 2 must be (batch_size, 64, s)

However the output from the pooling step has shape (batch_size, 128, s/2). How can a pooling step increase the number of parameters on axis 1 from 64 to 128?!

My guess is that the input to the pooling layer is actually a concatenation of 2 of the previous layer's outputs. In this case it would have input shape (batch_size, 128, s). However, the paper does not appear to clearly specify at what outputs are concatenated...

Can anyone clarify how this is the case?

As you mentioned, the paper doesn't clarify. However, my guess is that this is not due to concatenating 2 previous layers (I don't really see an specific reason to do this here) but because of concatenating the ResNet shorcut.

Generally, all conv layers have a number of filters, thus determining the output size (num_filters, size) regardless the inputs. On the other hand, MaxPooling does keep the input num_filters (though in this case reducing the size). In the paper, note that the num_filters is doubled at the output of all convlayer except for the one that does not keep the ResNet shorcut (last 512 conv layer). So my guess is that they are concatenating the output of the conv layer and the shorcut which would explain the output size.

Hope this helps!


Transcription of Hindi text to English text I am working on Hindi speech.

I want to convert Hindi text to English language text in Python. I am not interested in translation.

Example of the desired effect: input 'मेरा नाम मोहन है' to output 'mera naam mohan hai'.

seq2seq RNN models:

Basically, we need to transform the script of the text from Hindi to English. We can do this with the help of seq2seq RNN models. These models are widely used in Neural Machine Translation. Know more from here.

  seq2seq RNN models can take the sequence of Hindi words as input and output their corresponding English words.

Note: As these models have done their best in translation between languages, I assume that they can transcript between languages in a pretty well manner.

The above shows how a seq2seq model is used for language translation. Such a model can be used for transcription as well. See this notebook for creating such a system in TensorFlow Keras. Refer to these links for more:

https://towardsdatascience.com/how-to-implement-seq2seq-lstm-model-in-keras-shortcutnlp-6f355f3e5639
https://www.geeksforgeeks.org/seq2seq-model-in-machine-learning/
https://www.analyticsvidhya.com/blog/2018/03/essentials-of-deep-learning-sequence-to-sequence-modelling-with-attention-part-i/

Limitations:

You can get training data for translation easily. But, there is no data available for transcription between Hindi and English.
For the data, you can try posting a question on OpenData StackExchange.

Tip:

There's a Google Cloud API for transcription. See here for more details.


Dimension reduction techniques in R that do not use the full distance matrix I try to apply non-linear dimension reduction in R. As usual in machine learning I have a large data set (100 K rows). I tried the packages
RDRToolbox and vegan:

library( RDRToolbox )
res.lle = LLE(as.matrix(temp), dim = 3 , k = 10)
res.iso = Isomap(as.matrix(temp), dim = 3 , k = 10)

library( vegan )
isomap(dist(temp), ndim=10)

The first algoritms use the distance matrix internally and crash as they want to allocate 32GB. The second approach needs the distance matrix and crashes too.

Are there any tricks to do non-linear dimension reduction either with these algorithms? Or are there packages that do not use the raw and full distance matrix? Such approaches have to crash in my setting on my machine.

The only algorithm that works for me is Rtsne that appearantly does not use the whole distance matrix. largeVis was too slow so far to let it finish.

You can try using an auto-encoder which also is a non-linear dimension reduction technique.  It uses a neural network framework to find the most efficient transformation from $p$ dimensions down to whatever you choose.  It then finds how well it can reconstruct the origianl $p$ variables, and keeps tuning until it can optimally recreate the original values as "good" (lowest MSE) as possible.

In this case the input has 5 variables, and it reduces it down to 2 variables using non-linear transformations.  Once you get it down to the reduced variables (the light blue nodes) it then tries using non-lienar transformations to recreate those original five inputs using a similar nn structure.  The right-five red nodes are then compared to the left-five original inputs to see how much information is encoded in those 2 middle light blue reduced variables.

I've found the h2o implementation is pretty easy to use, but finding the correct parameters (number of layers, nodes, learning rate, etc) is still tough to select depending on your data.

Heres a little blog post that goes through the h2o autoencoder with the MNIST data set.


Measure of correlation for term frequency I'm trying to write a framework to compare a set of labels such as (for a sample of 5 yes/no answers to a question)  [0, 1, 1, 1, 0] to a series of features to determine correlation.  For numerical non-sparse features, like "number of words" or "average word length", I know I can use a variance-covariance matrix and get a sense for whether or not "number of words" or "average word length" is an informative feature for a model to answer the question.

I'd like to be able to do the same thing for term frequency (let's say using CountVectorizer in scikit-learn), but the resultant covariance matrix will be rather large and will only indicate whether or not that particular term is an informative feature.  How do I get some kind of "collapsed" or "aggregate" measure of correlation?  Is this even possible?

If you did a simple linear regression for each of the 5 outputs, your r-squared would be a good measure of the "aggregate correlation." If you wanted to compare to the other results, it would be helpful to do the same for each of your single features as well.


time series forecasting - sliding window method I have been trying to understand this sliding window technique but to no avail and really unsure as to how I would implement it.

My dataset: I have hourly values for the electric load for a year (over 8700 data points) - image below. I'm going to split the dataset into a training set  (1 jan to 30 sept) and a test set (1 oct to 31 Dec).

Im going to use supervised learning techniques such as Regression trees and random forests (basically anything that is available to me in scikit learn), train them on the training set then make predictions on test set.

I understand that I need to use historical known values  as input features to input into a model. As a result i created "Load_lagN".

Am I correct in saying that because I have created 10 lagged variables (Xt-1 to Xt-10) , this is the equivalent of using a sliding window of size 10?

How would I then  just simply train the model on the training set and make predictions on the test set without using the sliding model?

With the sliding window model, does that assume only the past n values (10 in my case) are relevant?

If I have split my dataset into a training and test set, how does the regression tree become trained and then make predictions in the test set (I'll stick to one step ahead forecasting for now) whilst implementing this sliding window technique?

http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html#sklearn.model_selection.TimeSeriesSplit
Ive come across this method - TimeSeriesSplit - is this what I need to use for the sliding window technique or is it only a cross validator?

My understanding of the sliding window method:
As  you can see in the image below, I use values of 10:00 to 19:00 to predict the value at 20:00, move the window so that this new value is now included, then predict the value for 21:00. This keeps happening until I have exhausted the training set. I then make predictions.

What are your thoughts?

Try this:

Make the data stationary (remove trends and seasonality).
Implement PACF analysis on the label data (For eg: Load) and find out the optimal lag value. Usually, you need to know how to interpret PACF plots.
Apply the sliding window on the whole data (t+o, t-o) where o is the optimal lag value.
Apply walk forward validation to train and test the models.

The way to escape sliding window is to use Recurrent Neural Networks but believe me, the method I suggested is worth it.

If you want the code:
https://machinelearningmastery.com/time-series-forecasting-supervised-learning/

This can help too: Master Thesis research applicable to your question


Next year forecasting with monthly data from many, correlated, non-monotonic trends I have trend data from many health departments in a local territory (eg. cardiology, orthopedics, etc...). These trends represent health service (visits, diagnostic, admissions) production, service value (proxy of type of service), and ratio of such service consumed by patients in our territory versus those consumed in other health providers than us, per department (eg. how much our patients have hip surgery in our hospitals versus other hospitals).

I would like to predict for each department or group of departments (all diagnostic or all surgery departments) those values for the next year. I have monthly data from the previous ten years.

Since there are many unmeasured sources of variability I can't expect to make long term predictions, by just one year is enough. Also, these trends have many change points (change of organization, personnel, technology), so I can mostly hope to predict the next year value once a new trend has started, predict for how long it will grow, reach a tipping point or going down.

I guess the model for each department could learn from previous growth and fall dynamics from his own past trend data and from the trends of other departments with similar dynamics and adjust for month related activity level, plus from information on where the service was provided.

Which model should I use for such a problem? I thought about a multilevel model for time series, but I guess there are more raffinate solutions.

One approach could be to simply use standard time series models, like ETS/TBATS/ARIMA for each department. These methods will, with proper care, take into consideration specific trends you might see (as well as seasonality) albeit predicting when a trend will fall/how long it will last is an incredibly difficult task. This wouldn't be an invalid approach, but the problem here is that each time series will be fit in isolation and you seemingly have large amounts of similar time series along with valuable exogenous data, much of it that does not depend on time. The approach above would mean fitting a potentially large amount of time series models which is largely impractical and tedious (in my opinion) and in the case of ARIMA you would not be able to include variables that don't vary with time. There are multivariate extensions of ARIMA methods that could help, but if you have many different time series convergence is often a problem.

This is where I would suggest you look into regression algorithms (statistical or ML) with time (in some fashion) as a covariate. Your dataset would basically be expanded so that one row is the observed response variable at any specific month + year, for any given department. Your time covariates could be two columns; one with the year, another with the month, or simply a continuous variable like 2019 + 1/12, 2019 + 2/12, etc.

You would then use "department" as a categorical variable, along with perhaps lagged values of the response (to perhaps incorporate current trend information) or perhaps variables like "average health service production in last k month(s) by department" or "median health service production in past year over all departments = over all the data", etc. The choice of how many months to take, along with the level at which you aggregate to (over a department? over all departments? over specific departments only?) will be determined based on what you think is reasonable.

If you have time related events, you could model this with indicator variables, and also perhaps use variables like "months since this time related event has started/ended", etc. to perhaps capture lagged effects. Seasonality, if it exists, could be easily captured with indicator variables/Fourier series transforms. To obtain more trend features, you might also run another regression algorithm using the past couple months of data (from the time period for that specific observation/row of data) to get the overall current trend (at that point in time). Non-time related continuous variables (like "how much our patients have hip surgery in our hospitals versus other hospitals") is a simple join to the expanded dataset by department.

In many of these feature engineering ideas, you will end up with NA values at the start of the series which implies giving up maybe a few months of data at the beginning (I think this shouldn't be too big of a deal if you have time series' that are 120 units long). Regardless, I'd recommend using time series cross validation, using the most recent data as testing indices, and walking forward until you run out of data. In your case, each test set would be comprised of 12 months worth of data, and you would add one month of data at a time to the training set (moving the test sets time period one month ahead).

I hope this helps.


R: Checklist for data checking I am new to data science and new to R. I see a lot of resources on data cleaning (i.e., dplyr, tidyr, tidyverse), but not so much on raw value verification. What are the "best practices" I should apply to verify the accuracy of the raw values or detect possible coding errors before cleaning? For example, I have been told you should ensure NAs are true NAs, or ensure there are no spaces in front of numerical or character values after you import data into R.

Thanks in advance.

UPDDATED with second resource

In general as I understand this is question about 'data quality'. There are lot of manuals, i like suggestions from this one - this is lets say semantic level:

SIX CORE DATA QUALITY DIMENSIONS
    COMPLETENESS
    UNIQUENESS
    TIMELINESS
    VALIDITY
    ACCURACY
    CONSISTENCY

This M&E Attribute: Data Quality Assessment checklist is more detailed.

Then, of course, you should go deeper, to the syntax level.


Maths question on mean squared error being dervied to bias and variance I am reading a book and have difficulty in understanding the math on bias-
variance tradeoff. Below is the section that I am having trouble with:

  Given a set of training samples $x_1, x_2, ..., x_n$ and their targets $y_1, y_2, ..., y_n$, we want to find a regression function, $\hat{y}(x)$, which estimates the true relation $y(x)$ as correctly as possible. We measure the error of estimation, how good (or bad) the regression model is by mean squared error ($MSE$):

I can derive mean squared error with partial derivative and the concept of slope. I also understand that $MSE$ is to minimize the total error. I also understand basics statics on expected value.

Yet, I have been stuck in finding the relevant math and statistical concepts behind this formula for a week.

The question is, what are the relevant math and statistical concepts behind this formula?

For example, how

$MSE = E[(y-\hat{y} )^2]$

becomes:

$= E[(y-E[\hat{y} ] + E[\hat{y}] - \hat{y} )^2]$

Thank you! I can see that the first component after adding and subtracting E[y^] is unchanged. Then the formula operated according to
$(a+b)^2 = a^2 + 2ab + b^2 $
where

$2ac = +E[2(y - E[\hat{y}])(E[\hat{y}] - \hat{y})]$

Why 2ac becomes

$2(y - E[\hat{y}])(E[\hat{y}] - E[\hat{y}]) $

Because it's a costant, everything that is a costant value remains unchanged by the E, that's why you can "move" it outside.

For example:

If y is a costant or it is known, the E doesn't affect it, so you can "move" it outside the symbol, and write just y.

The only thing that is unknown is the estimator yhat, in fact you have  and not just .

That's what's happening here.

I found this on CrossValidated that might be more clear.


Amount of data needed and hypothesis for SVD I was looking into the definition of SVD and trying to understand which are the conditions needed to be met in order to be able to use it.

Is there any hypothesis concerning the distribution of the data that I want to apply SVD on ?
Is the condition (number of data vectors > the dimension of the vectors space) strict ? Or is there always a way to decompose a dataset using SVD ? In the definition of SVD, all websites that I could find start by the same condition without fearther explanitions :

  given an $m\times n$ real matrix A with $n>m$, A can be written as ...

SVD is operation of decomposing a matrix $M$ into the matrix product $M=U\Lambda V$, where $U$ and $V$ are unitary matrices ($U^TU=UU^T=I$, etc.), and $\Lambda$ is diagonal.

So the formal answer is:

No, there are no assumptions about distribution of $M$. It does not to be random at all, it is just a matrix of any nature.
Condition $n>m$, where $(n, m)$ is shape of $M$, is not necessary. Indeed, if you have $n>m$, you can transpose the equation  $M=U\Lambda V$ to get  $M^T=V^T\Lambda U^T$, and it will be also a valid SVD decomposition.

Now the less formal part.

In data mining SVD has two main applications: for computations (like matrix inversion, least squares, etc.), and for dimensionality reduction (e.g. to compress the user-item matrix in recommender systems). For computations, only algebraic properties of SVD (shown above) matter.

For dimensionality reduction, truncated SVD is used: the small elements of $\Lambda$ are discarded, and only $k$ largest are kept. This operation is equivalent to finding a $k$-dimensional hyperplane such that projection of the data ($M$) on this hyperplane is closest (in Euclidian distance) to the original data.

In the last case, the analyst would like the decomposition to generalize well to unseen data, and this problem can be formulated well in probabilistic language. If we state that $M$ consist of $n$ IID $m$-dimensional random variables, then it appears that SVD works best if they are joint normal. That's because:

a. multivariate normal is indeed distributed nearly a hyperplane, and

b. for multivariate normal, Euclidean distance is tightly connected with probability density. Therefore truncated SVD (or PCA, which is mathematically identical) can be viewed as maximum likelihood estiomation of multivariate normal distribution with $k$ independent components. For more details, see the article by Bishop.


How the squared Euclidean distance is an example of non-metric function? I am reading a book on Pattern Recognition (by Prof V Susheela Devi and Prof Murty) where in the chapter of data representation 2.3.3 the non metric similarity function is defined as those which do not obey either the triangular inequality or symmetry.

In that context, it further adds The squared Euclidean distance is itself an example of a non-metric, but it gives the same ranking as the Euclidean distance which is a metric.

Is the squared Euclidean distance different from the Euclidean distance?

How is the squared Euclidean distance non-metric?

Let $x, y \in \mathbb{R}^n$. The Euclidean distance $d$ is defined as
$$
d(x,y) = \sqrt{\sum_{i=1}^n (x_i - y_i)^2}.
$$
The squared Euclidean distance is therefore
$$
d(x,y)^2 = \sum_{i=1}^n (x_i - y_i)^2.
$$

We know that Euclidean distance is a metric.
Let us check whether squared Euclidean distance is also a metric. I will use the definition from Wikipedia (Ankit Seth's definition is equivalent).

Non-negativity: $d(x,y)^2 \ge 0$. This one is obvious.
Identity of indiscernibles: $d(x,y)^2 = 0$ if and only if $x=y.$ This is true because $d$ is a metric, and $d(x,y)^2 = 0$ if and only if $d(x,y) = 0$.
Symmetry: $d(x,y)^2 = d(y,x)^2$. This is again true because $d$ is a metric and therefore $d(x,y) = d(y,x).$

(You can also see that 2. and 3. are true directly from the definition of $d(x,y)^2$).
Looks fine so far. But:

Triangle inequality: Do we have $d(x,z)^2 \le d(x,y)^2 + d(y, z)^2$ for all $x, y, z \in \mathbb{R}^n$? No. Pick an arbitrary $x \in \mathbb{R}^n \setminus \{0\}$ and set $y = 2x$ and $z=3x$. Then
$$
d(x,z)^2 = \sum_{i=1}^n (x_i - 3x_i)^2 = 4 \sum_{i=1}^n x_i^2
$$ and
$$d(x,y)^2 + d(y,z)^2 = \sum_{i=1}^n x_i^2 + \sum_{i=1}^n x_i^2 = 2 \sum_{i=1}^n x_i^2.
$$
Since $4\sum_{i=1}^n x_i^2 > 2 \sum_{i=1}^n x_i^2$, we have found a counterexample that shows that the triangle inequality does not hold.

What does it mean that squared Euclidean distance gives the same ranking as Euclidean distance?
Suppose we have $x, y, z$ such that $d(x,y) < d(x,z)$.
Then $d(x,y)^2 < d(x,z)^2$ as well: It ranks points in the same way as Euclidean distance.

This is good to know. For instance, this tells us that the $k$-nearest neighbors classifiers gives the exact same results for squared Euclidean distance and Euclidean distance.


Algorithms that can determine whether a string is an English sentence? In particular, I'm looking for something that can distinguish between complete sentences and sentence-like grammatical structures like clauses and phrases. It would also be very helpful if a library with this functionality was shared, but even a link to a paper would be appreciated.

For example- I would like "False" to be returned for strings like:

'Via The Daily Currant,'

“Waiting for the rain to stop”

'查看更多请返回网站主页。'

'*  *  *'

I would like "True" to be returned for things like:

"This is a complete sentence."

"I don't think there will ever be a Half Life 3, unfortunately."

Thank you!

There is a library for python

https://pypi.org/project/langdetect/

This field is called "language detection" and it is quite active.


In random forest, what happens if I add features that are correlated? I'm training a random forest, trying to predict market shares of future stores on geographical areas. I have many features for these areas, some of which tell similar but different things about one thing.

For example, I know the total number of $accommodations$ in the area, and I also have 5 others columns which are all linked in the following way:

$main \space accommodations + secondary \space accommodations + holiday \space accommodations = houses + flats = accommodations$

I have the feeling that including them all in my model would be wrong... but including them might be important... Any hint on how I should handle this?

Would it be a good idea to include $accommodations$ as absolute value and include all the other five but as % (of $accommodations$) and not as absolute values?

In a similar fashion, I also have the total number of $households$ of the area, the $total \space income$ of the area, and the $average \space income$ of households in the area (so that $households * average \space income = total \space income$). I have the feeling using the average and not the total income would be a better idea, but how can I be sure I'm right ?

(I guess I could train three random forests using the average income only, the total income only, and both, and see how they perform on cross validation, but is there a rule of thumb that I should know of which can make me go faster ?)

(In case it's relevant, I'm using R and the randomForest package)

Random forests don't suffer from correlated variables like linear regression models do. Random forests randomly pick from a subset of variables at each split (hence the "random" in "random forests"). This means that correlated variables are less likely to show up together when the trees are being trained. But even when correlated variables show up in the same random subset of variables, it's still not much of an issue because the variables aren't assigned coefficients.

Correlated variables are mostly an issue for linear models that try to hold all other variables constant when calculating coefficients during training. The variable selection process is much simpler for trees and tree-based algorithms like random forests and gradient boosting. When a random forest is being trained and a tree's split is being evaluated, the algorithm will simply pick whichever feature most reduces error on that particular split of the tree. Once a variable is picked, there is no coefficient, just a greater-than/less-than split point, so the problem of "exploding coefficients" doesn't apply.


Grouping already clustered data (with a pre-defined x and y) I have an already clustered data set (I wanna keep my x and y), where there's clearly a small group of elements in the middle that don't follow the expected patterns.

I can select them manually, but I wonder if there's a way of automating the selection part of these elements, efficiently.

Something like using just the grouping part of a clustering algorithm, I've been trying it with a threshold, but it doesn't produce good results in cases that won't form a circular cluster.

It would be helpful to know which clustering technique are you using.

You can use

Partition-based Clustering: for example K-Means Clustering, not that good with outliers.
Hierarchical-based Clustering: Produces trees of clusters (Agglomerative, Divisive). You get a Dendogram.
Density-based Clustering: produces arbitrary shaped clusters, for example DBSCAN

If you are looking something other that a circular cluster and you need clusters within clusters, I would try DBSCAN. It locates regions of high density and separate outliers and it can find clusters within clusters.

If you are using Python you can use DBSCAN with sklearn

from sklearn.cluster import DBSCAN

I hope that helps!


Is the classifier confused? Binary Classifier:

Assuming I have built a binary classifier and decided on an operating point. And made the classifier live on production data. This classifier returns the probability score for each of the two classes.

Is it safe to say, for production/out-of-sample instances where the classifier returns a probability score close to the operating point, the classifier is confused for those instances?

Moreover, if the I collect those instances where the classifier scores are close to the operating point.
And compare it with my training data. There could be two cases:

The "confusing" instances have significantly different distribution as opposed to the training data. In which case, my classifier is not faulty. And If possible I should manually label these instances and refit the classifier.
The "confusing" instances have similar distribution to my training data. In which case my classifier is at fault. What could be the implications of this case? But I believe this case would have been captured while training itself. Such instances would be very less in cardinality.

If the above mentioned thought process is correct. How can we extend this to multi-class classifier?

The more accurate term would be "uncertain", but you are correct in your line of thought.
Your classifier (based on learned or engineered features distributions) can't determine with certainty which class to assign to this merginal cases.
It isn't necessary a case where they have a completely different distribution, more likelely that they are part of the same distribution but are located on its tail close to the tail of the distribution of the Second class (think 2 gaussians that at some area share similar density values).

Depending on the amount of such cases, you should examine them and attempt to trace the source of confusion. Maybe some clever preprocessing could help or different features (and maybe these are just outliers).

In the multi class case, you would probably get similar behavior, but each time on a subset of all classes. I.E. the model is certain that the sample is one of 2-3 classes (out of 10 total classes for example) but it can't be certain from which specific class. So you will get probabilities of similar values for these classes and very small probabilities for the rest.


Binary classification using images and an external dataset I currently have a project in which I must create a binary classifier to detect defective products. I have image data which has already been labeled (each part has been labeled as a pass or fail), as well as an external dataset which has specific measurements for each image (it contains parameters which could not be accurately captured by a camera, such as temperature, length of a particular piece, etc.). I have achieved decent accuracy by applying basic classification algorithms such as logistic regression and support vector machines (in scikit-learn) to only this external dataset, but I would like to incorporate the image data to improve accuracy. I have read about training a CNN on the images, but I do not know how to incorporate the external dataset to train the model (in fact, I don't even know if this is possible). Is there any way to use both the image data as well as the external dataset to classify these images? Any help is much appreciated.

Yes you can most certainly combine the two ways of information in CNNs.
Let's take a look at a CNN:

Roughly speaking, CNNs consist of two parts:

one that has convolutional layers and optinally pooling ones
and one that has fully connected layers.

The first is used for extracting features from the image, while the second is used for using these extracted features to classify the image. The last part essentially is like a regular fully-connected Neural Network.
If you want to add more information to the CNN, which is not an image, you can add this to the first fully connected layer of the network. This way this information bypasses the feature extraction step (which is relevant only for imaging data) and is used only for classification.


Deep Learning for high contrast images with small differences I am trying to make a deep learning classification on the gear part you can see in the images. The contrast is high and almost binary. I want to classify the images where the circle located closest to the centre circle is left of the circle located furthest from the centre circle as the good part.

When this is mirrored (the circle located closest to the centre circle is right of the circle located furthest from the centre circle) the part needs to be classified as bad.

The parts can be rotated over a range of 360 degrees. The parts are found beforehand and cropped with the centre of the part as the centre of the image, which makes translation not a problem. The parts do not vary in scale.

I am using Keras with TensorFlow as Backend. I have 3600 input images, with 1800 good and 1800 bad images, where the parts are rotated. I have build some simple CNNs with three convolutional layers up to the VGG16 model, without getting past 0.5 accuracy.

My knowledge on different kind of models is limited and therefore I don't know which arcitecture would suit my problem best. Can anyone help me with some guidance on the kind of model I could use or to tell me if this is at all possible like this?

If more information is needed please let me know.

I have solved my problem! I found a relevant article which describes an architecture suitable for my problem. I have changed it a bit however. The article: https://www.researchgate.net/publication/312328148_Binary_shape_classification_using_Convolutional_Neural_Networks.

The model in keras:

modelBS = Sequential()

modelBS.add(Conv2D(150, (3, 3), input_shape=(ImgSize,ImgSize,1), activation='relu', padding='same'))
modelBS.add(MaxPooling2D(pool_size=(5, 5), strides=(5, 5)))

modelBS.add(Conv2D(150, (5, 5), activation='relu', padding='valid'))
modelBS.add(Conv2D(150, (2, 2), activation='relu', padding='valid'))

modelBS.add(Flatten())
modelBS.add(Dense(rp, activation='softmax'))


Will a Count vectorizer ever perform (slightly) better than tf-idf? For the task of binary classification, I have a small data-set of a total 1000 texts (~590 positive and ~401 negative instances). With a training set of 800 and test set of 200, I get a (slightly) better accuracy for count vectorizer compared to the tf-idf.

Additionally, count vectorizer picks out the relevant "words" training the model, while tf-idf does not pick those relevant words out. Even the confusion matrix for count vectorizer shows marginally better numbers compared to tf-idf.

TFIDF confusion matrix
[[ 80  11]
 [  6 103]]
BoW confusion matrix
[[ 81  10]
 [  6 103]]

I haven't tried cross-validation yet though it came to me as shock that count vectorizer performed a bit better than tfidf. Is it because my data set is too small or if I have't used any dimensionality reduction to reduce the number of words taken into account by both the classifiers. What is it that I am doing wrong?

I am sorry, if it is an immature question, but I am really new to ML.

I would say 1000 documents is a bit less to draw any conclusion about the vectorization technique, Neither an increase in True positive by 1 would matter. As the size of the vocabulary increases, TfidfVectorizer would be better able to differentiate rare words and commonly occurring words while Countvectorizer would still give equal weight to all words which is undesirable. So, TfidfVectorizer will give you better performance than CountVectorizer as the size of the vocabulary increases


Confidence Score For Trained Sentiment Analyser Model I have trained a text based sentiment analysis model, using SciKit-learn and custom data. I have the model ready and it works fine in predicting a text to a class (Positive or Negative or Neutral). I have achieved over 85% testing accuracy and around 80% cross validation accuracy.

But I want to get the confidence score attached to each of my prediction to a new example data/text I feed to the classifier. This is just an extra parameter I want to show/output apart from just the predicted class.

I have no idea how to achieve this, I shall be really thankful if anyone can provide some helpful insights.

I was able to solve it myself after some further research.
I will be briefly describing my approach here.
Cheers!

The idea is to find the confidence interval which was also same as finding the distance from the decision boundary / hyper-plane, in my case.

If you are using the Scikit Learn API, there is a method called predict_proba() for several classification models like Logistic Regression, SVM, Random Forest, etc. If your classifier does not provide one, you can wrap it with the CalibratedClassifierCV which can be found in sklearn.calibration, then use the above method to calculate the distance from the decision boundary.

If you are looking for custom in-depth implementation, here are some papers / references that might help.

Ref - 1
Ref - 2


How to refit GridSearchCV on Multiclass problem I'm trying to use GridSearchCV for my Multiclass problem. For starters, wanted to test it on KNeighborsClassifier.
First, here's the code where I define the function which uses GridSearchCV:
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import KFold

def grid_search(estimator, parameters, X, y):
    scoring = ['accuracy', 'precision', 'recall']
    kf = KFold(5)

    clf = GridSearchCV(estimator, parameters, cv=kf, scoring=scoring, refit="accuracy", n_jobs=-1)
    clf.fit(X, y)

    i = clf.best_index_
    best_precision = clf.cv_results_['mean_test_precision'][i]
    best_recall = clf.cv_results_['mean_test_recall'][i]

    print('Best score (accuracy): {}'.format(clf.best_score_))
    print('Mean precision: {}'.format(best_precision))
    print('Mean recall: {}'.format(best_recall))
    print('Best parametes: {}'.format(clf.best_params_))

    return clf.best_estimator_

And, here's where I use it, when I try running a K nearest neighbors classifier:
from sklearn.neighbors import KNeighborsClassifier

parameters = {'n_neighbors': [1, 2, 5, 10], 'weights': ['uniform', 'distance'], 'metric': ['manhattan', 'euclidean', 'chebyshev']}

knn = grid_search(KNeighborsClassifier(n_jobs=-1), parameters, X_train, y_train)

Under the current state of the above code I'm getting the following ValueError
ValueError: Target is multiclass but average='binary'. Please choose another average setting, one of [None, 'micro', 'macro', 'weighted'].

As you may have guessed, this might be related to the value of the refit parameter for GridSearchCV which currently is set to refit="accuracy" and this cannot work because the problem is multiclass.
I changed it's value many times, tried True or other explicitly stated metrics and nothing fixed the problem. On some of those tries, the error message changed to:

ValueError: For multi-metric scoring, the parameter refit must be set
to a scorer key or a callable to refit an estimator with the best
parameter setting on the whole data and make the best_* attributes
available for that metric. If this is not needed, refit should be set
to False explicitly.

Any advice?

Nevermind, I realized my mistake.
precision and recall don't work with multiclass data. They should be replaced by precision_macro and recall_macro instead.
Speaking of which, the one place where I tried using them on the code should be also changed into:
best_precision = clf.cv_results_['mean_test_precision_macro'][i]
best_recall = clf.cv_results_['mean_test_recall_macro'][i]


How to factorize the Matrix in TensorFlow? (Recommender System) Given a user ratings matrix which is $n \times p$, where $n$ users rate $p$ movies, I already have a row matrix $n \times 10$ which characterises the user.

I ideally wanted to use the TF was method for optimisation, https://www.tensorflow.org/versions/master/api_docs/python/tf/contrib/factorization/WALSMatrixFactorization
but it looks like it creates the row matrix itself.

What I need is to create the column matrix - which is $10 \times p$ (not both), containing the relationship between hidden characteristics (10) to the movies (p).

How can I do this in TF?

If R is the rating matrix, U is the user matrix and M is the movie matrix, then note that there is almost certainly no matrix M that satsifes $R = UM$. U and M are too low rank. However you should be able to find the matrix M that minimizes $|R - UM|$.

There is no need to use an optimizer, though you could I guess, because this is a convex problem. You're just solving a large linear system. This is indeed just what ALS does repeatedly.

You've found an ALS solver and if you just need to solve one step and have the user matrix already, I think you just supply it as row_init and run one iteration? haven't used it, but conceptually that's all you are doing. You don't need weights either.


Smaller network width than output size? I am trying to figure out if it makes sense that the width of the network could be smaller than the input/output size?

So for example, I am giving the Neural Network 2048 numbers, and I am expecting 2048 numbers back. I would also like to use LSTM's, which take a lot of time/space etc to train, and having one or a few 2048 or larger LSTM layers connected to Dense layers would take a lot of space/time to train. But maybe that is the way to go?

Does anyone have any experience with this type of problem? Thank you!

In high dimensional spaces like the one-dimensional space, you have to employ a very simple network, maybe just a single neuron and investigate whether your data is linearly separable in that feature space which can not be visualized. If you observe that you do not have a good performance, you can increase the size or the layers of your network step by step. You can generalise what I've just referred to other networks like RNNs.

What I've referred means that your data may be in a way that is separable though it's in 1000 dimensional space and you just need one neuron to classify it.


How word2vec can be used to identify unseen words and relate them to already trained data I was working on word2vec gensim model and found it really interesting.
I am intersted in finding how a unknown/unseen word when checked with the model will be able to get similar terms from the trained model.

Is this possible?
Can word2vec be tweaked for this?
Or the training corpus needs to have all the words of which i want to find similarity.

Every algorithm that deals with text data has a vocabulary. In the case of word2vec, the vocabulary is comprised of all words in the input corpus, or at least those above the minimum-frequency threshold.

Algorithms tend to ignore words that are outside their vocabulary. However there are ways to reframe your problem such that there are essentially no Out-Of-Vocabulary words.

Remember that words are simply "tokens" in word2vec. They could be ngrams or they could be letters. One way to define your vocabulary is to say that every word that occurs at least X times is in your vocabulary. Then the most common "syllables" (ngrams of letters) are added to your vocabulary. Then you add individual letters to your vocabulary.

In this way you can define any word as either

A word in your vocabulary
A set of syllables in your vocabulary
A combined set of letters and syllables in your vocabulary


LSTM for text with different sentences size, but same input-output sizes Hello fellow Data Scientists

I'm trying to use a LSTM (using word embeddings) to generate a system that can tag each word of a sentence. For this, I give it a set of sentences of different sizes and using padding I make the sentences all the same size, so that the LSTM can process them.

The problem is, the LSTM results, after excluding the tags that correspond to the padding portion give outputs of different size than the input.

So my question is, is this an architecture problem or it might be the case that more training (epocs word bigger batches) is needed?

Thanks for the help!

Edit: More details:

In the following picture I show my LSTM Architecture

Upon running for a sentence of size n (using one hot encoding and word embeddings) I get a result, that is usually of diferente size; by result I mean final output of the lstm and corresponding translation into words.

Also, there are the variables I have at the moment, with which I've played around:

BATCH_SIZE = 164
EPOCHS = 50
LSTM_NODES = 128
NUM_SENTENCES = 3000
MAX_NUM_WORDS = 250000
EMBEDDING_SIZE = 300

Sample input:

1  Cada
2  obra
3  consome
4  1,5
5  tonelada
6  de
7  aço
8  (
9  US$
10 6
11 mil
12 )
13 mais
14 US$
15 10
16 mil
17 de
18 mão-de-obra
19 .

Sample output:

1  c-am-prd*
2  c-am-prd*
3  (c-v*)
4  (am-rec*)
5  (c-v*)
6  (c-v*)
7  (c-v*)
8  (c-v*)
9  am-dis*

The problem is, the LSTM results, after excluding the tags that correspond to the padding portion give outputs of different size than the input.

How do you get the "LSTM results"? Are the "LSTM results" extracted making a prediction for each word in the original sentence?

The LSTM is not a problem to do what you want to do. It might be an issue of your implementation. If you put a snippet of code maybe you could get some help.


Formatting Data for use in SKmultilearn I have a dataframe of X values and one with the y values in what I think is the proper format. I took the single feature y and converted it to dummies so it's binary and is m rows by n features.

from sklearn.model_selection import train_test_split

X = df.drop('target',axis=1)
y = pd.get_dummies(df['target'], sparse=True, drop_first=True)

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=99, train_size=0.60

I've been trying to figure out how exactly the data should look from the documentation but am not having much luck.

I first tried the following:

import skmultilearn.problem_transform import BinaryRelevance
from sklearn.naive_bayes import GaussianNB

classifier = BinaryRelevance(GaussianNB())
classifier.fit(X_train, y_train)

where X, y train sets are pandas dataframes. This results in the following error:

TypeError: no suppoerted conversion for types: (dytype('O'),)

Then I converted to matrices:

X_train = X_train.asmatrix()
y_train = y_train.asmatrix()

I received the same error. Then I tried

X_train = np.matrix(X_train)
y_train = np.matrix(y_train)

Same error again. Is there an easy way to go from dataframe to proper format for skmultilearn models?

Some of the features were categorical, so they had to be converted to integers.

Then, I ran this to get the data in the right format:

import scipy.sparse as sp
X_train = sp.csr_matrix(X_train)
y_train = sp.csr_matrix(y_train)


Original Meaning of "Intelligence" in "Business Intelligence" What does the term "Intelligence" originally stand for in "Business Intelligence" ? Does it mean as used in "Artificial Intelligence" or as used in "Intelligence Agency" ?

In other words, does "Business Intelligence" mean: "Acting smart & intelligently in business" or "Gathering data and information about the business" ?

This question was the topic of a debate among some fellows in our data-science team, so I thought to ask about it from other experts. One might say that both meanings are applicable, but I'm asking for the original intended meaning of the word as proposed in the 1980s.

An acceptable answer should definitely cite original references, and personal opinions are not what I'm seeking.

Howard Dresner, in 1989, is believed to have coined the term "business intelligence", to describe "concepts and methods to improve business decision making by using fact-based support systems.". When he was at Gartner Group. This is a common mantra, spread over the Web. I have not been able to trace the exact source for this origin yet. Many insist on he was not at Gartner group in 1989, which is confirmed in the following interview. In his 2008 book, Performance Management Revolution: Improving Results Through Visibility and Actionable Insight, the termed is defined as:

  BI is knowledge gained through the access and analysis of business
  information.

He says, at the beginning, that

  In 1989, for example, I started-some might say incited-the BI
  revolution with the premise that all users have a fundamental right to
  access information without the help of IT.

No apparent claim of the invention of the term on his side. Indeed, one can find older roots in H. P. Luhn, A Business Intelligence System, IBM Journal of Research and Development, 1958, Vol. 2, Issue 4, p. 314--319.

  Abstract: An automatic system is being developed to disseminate information to the various sections of any industrial, scientific or government organization. This intelligence system will utilize data-processing machines for auto-abstracting and auto-encoding of documents and for creating interest profiles for each of the "action points" in an organization. Both incoming and internally generated documents are automatically abstracted, characterized by a word pattern, and sent automatically to appropriate action points. This paper shows the flexibility of such a system in identifying known information, in finding who needs to know it and in disseminating it efficiently either in abstract form or as a complete document.

The author claims that:

  The techniques proposed here to make these things possible are:

  Auto-abstracting of documents;
  Auto-encoding of documents;
  Automatic creation and updating of action-point profiles.

  All of these techniques are based on statistical procedures which can
  be performed on present-day data processing machines. Together with
  proper communication facilities and input-output equipment a
  comprehensive system may be assembled to accommodate all information
  problems of an organization. We call this a Business Intelligence
  System.

He also gives the explanation of the terms "business" and "intelligence":

  In this paper, business is a collection of activities carried on for
  whatever purpose, be it science, technology, commerce, industry, law,
  government, defense, et cetera. The communication facility serving the
  conduct of a business (in the broad sense) may be referred to as an
  intelligence system. The notion of intelligence is also defined here,
  in a more general sense, as "the ability to apprehend the
  interrelationships of presented facts in such a way as to guide action
  towards a desired goal."

So the idea of "linking the facts" is already present in H. P. Luhn paper. To many sources, Howard Dresner has re-invented "Business Intelligence" to re-brand decision support system (DSS) and executive information system (EIS) when at DEC, and the term became famous throught the influence of the Gartner group.

Apparently, the term has already been used way before, as in the book Wholesale Business Intelligence and Southern and Western Merchants' Pocket Directory to the Principal Mercantile Houses in the City of Philadelphia, for the Year 1839.

As I could not fetch this source, I will stick to the Luhn/Dresner acception. It relates to the etymology of intelligence:

  late 14c., "faculty of understanding," from Old French intelligence
  (12c.), from Latin intelligentia, intellegentia "understanding, power
  of discerning; art, skill, taste," from intelligentem (nominative
  intelligens) "discerning," present participle of intelligere "to
  understand, comprehend," from inter- "between" (see inter-) + legere
  "choose, pick out

In Business Intelligence for Dummies (Scheps, 2008), the definition chapter plays on Military Intelligence:

  Business Intelligence Defined: No CIA Experience Required So what the
  heck is business intelligence, anyway? In essence, BI is any activity,
  tool, or process used to obtain the best information to support the
  process of making decisions.

  For our purposes, BI revolves around putting computing power (highly
  pecialized software in concert with other more common technology
  assets) to work, to help make the best choices for your organization.

  Business intelligence is essentially timely, accurate, high-value, and
  actionable business insights, and the work processes and technologies
  used to obtain them.

I would thus bend toward "Gathering data and information about the business", maybe more "to better conduct business". Additional historical comments can be found in Father of BI? Is he having a laugh?!


Choosing weights on random forest for imbalanced data with the aim to minimize false positives I am currently dealing with a binary classification task on imbalanced data with the following distribution:

y_train: 4981 positive / 863894 negative samples
y_test:  128  positive / 128309 negative samples

The goal is to aim for a high precision (as little false negatives as possible).

How do I go on about choosing the weights for the random forest?

I tried to balance out the y_train ratio by assigning weight 1 to "negative" and 173 to "positive", but that still caused all the samples to be assigned to negative.

At this point, should I already consider this a problem with the features used, or should I try to assign higher weights with GridSearch (roc_auc as scoring parameter) and set the decision-threshold higher first?

I totally missed out on the fact, that the RandomForestClassifier accepts the values “balanced” and “balanced_subsample” to automatically assign the weights as per documentation


Pretraining a neural network to teach it general information Let's say that I want to train a neural network to recognize symptoms of severe dehydration in distance runners visually. Runners tend to finish races either overhydrated (hyponatremic) or dehydrated, but the two present with fairly similar symptoms, so knowing the difference could be quite useful.

I've got myself a small dataset of a few thousand faces of dehydrated runners, a few thousand of overhydrated runners, and several million faces of the general population. I don't expect to be able to properly train a network to recognize dehydration with just a few thousand faces of the dehydrated, but I'd like to see if I can pretrain the network using the several million faces so it learns to distinguish facial details, then train it on the actual overhydrated/dehydrated faces. But if I take some other problem, like distinguishing men from women, and pretrain it on that, I'd imagine my hydration training phase would output something that distinguishes something between hydration level and gender.

Is there some standard way to pretrain a model to learn general details (of faces for example) before training on the proper (far smaller) dataset?

Considering two tasks, first, classifying the gender and second, whether its overhydration or dehydration. We can't create a model that perform these two tasks in a combined manner. Yes, a system could be developed for the same.

Discussing pre-trained models for facial recognition and related tasks, we have the FaceNet model as well as VGGFace. We can build a model on top of these highly trained models.

A pretrained model as discussed above will first, provide a bounding box for each face present in the image. The image could be a video frame from a real-time stream.
We will crop that part of the image, so we are only left with a smaller image which only contains the face of the athlete.
Another model, which will be trained on overhydrated vs hydrated faces, will classify the face into the two classes as needed.

For each step, we can train a model and place it in the pipeline. Also, try weight quantization for a faster inference speed.


What is purpose of partial derivatives in loss calculation (linear regression)? I am studying ML and data science stuff from scratch. As a part of the course, I am studying how the models are derived. And for most of them, starting with the simplest - linear regression, we take partial derivatives. I understood its implementation part, however, I am a bit confused about why we need to take partial derivative there.

Is there any specific reason behind it? Can we use any other methodology to compute linear regression loss function?

The underlying idea behind machine learning is to come up with more or less complicated algorithms such that, given a set of input data, one is able to produce some sort of output; this output in turn depends on some parameters (upon which the model is specified). The objective is to choose the aforementioned parameters so that the algorithm output is as close as possible to the actual result. Now let $y_i$ and $f(x_i, \beta)$ be the actual and the predicted value, respectively, (in correspondence of the variable $x_i$): the previous sentence translates into trying to choose the parameters $\beta$ (whatever they mean) such that the error you commit is the smallest, namely $L(y | x, \beta)$ is minimised, where the function $L$ is any way we decide to estimate the error between predictions and actuals. In the literature $L$ is referred to as loss function and for most practical purposes, especially for polynomial models like linear regression, it reduces to the sum of squares
$$
L(y|x) = \sum_{i=1}^N (y_i -f(x_i,\beta))^2.
$$
For each choice of parameters $\beta$ and function $f(x_i)$ the above takes different values; we are looking for, once we decide to fix the form of $f$, the set of $\beta$ such that the above is the smallest. Assuming the loss function is differentiable, local minima must satisfy the condition such that the collection of partial derivatives with respect to the variable in question (in this case $\beta$) must vanish; as such, at the end of the day one comes down to essentially taking derivatives and equating them to zero.

In case of linear regression one assumes $f(x_i, \beta) = \sum_{j=1}^M x_i^j \beta_j$: plugging this expression into the loss function and taking derivatives gives back the familiar expressions for the coefficients that one learns in school. Likewise for more complicated models: the form of the function $f$ might be more complicated, there might be analytical problems related to the computational minimisation of the loss, there might be a bunch of more parameters connected to each other in a somewhat complex way (for instance for neural networks), nevertheless the underlying argument still holds.


How to restrict the values of predicted variable to be positive? I am using Python Linear Regression to predict the weekly orders for a food deliver company. But some of my orders are coming out as negative. Is there any way to restrict the predicted values to be greater than 0,i.e positive?

Use a generalized linear model which allows you to account for this. I do not recommend discretizing your continuous variable, you will lose a ton of information doing this and the size of the bins are arbitrary.

A standard linear model can be rephrased as a generalized linear model under a normal distribution and identity link function. Simply switch the link function to log, and derive your predictions = exp(XB), where XB is the linear predictor. Alternatively, change the distribution to a distribution with support on [0, infinity] like the gamma or inverse gaussian and use a log link. If you have count data (1 order, 2 orders, 3 orders, ...) rather than a fully continuous variable (550.567 dollars of orders, 300.234 dollars of orders, ...) then use a generalized linear model with a Poisson/log link instead.

As another alternative, log your predictor and assume Y = ln(target) follows a normal distribution. Then exp(Y) follows a lognormal distribution. That is, log your target variable, predict the logged target variable (using a standard linear model, normal/identity link usually) and then use exponentiation to get back your predictions. Note, however, that this might introduce some bias when you convert back to the standard unit. You can correct this by calculating the sample variance of your logged target variable in your training set and then letting your final prediction = exp(XB + variance^2/2). This comes from the expected value of a lognormal distribution.


Best way to evaluate performance for my case I have dataset that looks like this

F1 F2 F3 F4 F5 F6 A1 A2 A3
1  0  0  0  4  3  X  X  X
0  3  9  0  0  0  X  A  X
0  0  0  0  1  0  X  X  X
0  0  3  2  0  0  X  X  X
0  5  0  0  0  1  X  X  X
0  0  7  0  1  3  X  X  X
0  0  0  1  1  0  X  X  B
1  3  0  2  0  1  X  X  X
7  0  0  2  1  3  X  X  X
4  0  0  0  5  0  B  A  X
0  0  7  2  1  3  X  X  X

where F1 to F6 are the features

and A1 to A3 are the prediction attributes.

as you can see most of the records are X X X

when I build my prediction algorithm and try to evaluate the performance I use f1_score

but due to most records have X X X values, I get high f1_score.

which I think this is not the right way to evaluate.

the main point of the evaluation is to find if the values A B and C are predicted correctly.

It's not possible to obtain a single f1-score when there are several classes, probably you are using micro or macro f1-score?

First don't use the micro F1-score, since it gives the same weight to every instance and therefore gives a lot of weight to the "X" class.

Macro F1-score should be closer to what you want to evaluate, since it gives every class equal weight. If you want you could even calculate the macro F1-score over only classes A,B,C (i.e. excluding X), but that seems unnecessary to me.


Pattern Recognition (Bishop) - Maximum Likelihood This refers to Chapter 2, section 2.2 (page 75) - Pattern Recognition and Machine Learning, Christopher Bishop.

The question is related to calculation of Maximum Likelihood variable:

Maximum likelihood is given as:
$$p(D|\mu) = \prod_{n=1}^{N}\prod_{k=1}^{K}\mu_{k}^{x_{nk}} = \prod_{k=1}^{K}\mu_{k}^{m_{k}} \tag{2.29}$$
where $m_{k} = \sum_{n}x_{nk}$ $\implies$ #occurrences of event $k$, $x_k$, over $n$ trials where, k={1,$\ldots$,K} and, n={1,$\ldots$,N}. Also, $\mu_{k}$ being probability,  and $D$, the observed data.

  Author applies Lagrangian multiplier $\lambda$ and take log-likehood
  of above equation to get: $$\sum_{k=1}^{K}m_{k}ln{\mu_{k}} + \lambda(\sum_{k=1}^{K}\mu_{k} - 1) \tag{2.31}$$  given the
  (probability) constraint): $$\sum_{k=1}^{K}\mu_{k} = 1$$

  Maximization of eqn. 2.31, w.r.t $\mu_{k}$ and equating it to 0,
  yields: $$ \mu_{k} = -m_{k}/\lambda \tag{2.32}$$
  Author substitutes the constraint in eq 2.32 and finally gets:
  $$
\mu_k^{ML} = \frac{m_k}{N} \tag{2.33}
$$

I am losing it in the last step - my (partial/ wrong) derivation of 2.31 w.r.t. $\mu_{k}$ yields:
$$ \sum_{k=1}^{K}\frac{m_{k}}{\mu_{k}} + \lambda\cdot K = 0$$

I am not able to understand:

Where am I going wrong
How to move ahead to get the result (only, if my derivation is correct)

The partial derivative you have calculated is incorrect. You are probably confused by the index k in the equation.
Consider expanding the Langrangian:
$$ L = m_{1}ln\mu _{1}+ m_{2}ln\mu_{2} + ... + m_{K}ln\mu_{K} + \lambda(\mu_{1} + \mu_{2} + ... + \mu_{2} - 1) $$

Now, take the derivative with respect to $ \mu_{1} $

$$ \frac{\partial L}{\partial \mu _{1}} = \frac{m_{1}}{\mu _{1}} + \lambda $$

Then, in general,
$$ \frac{\partial L}{\partial \mu _{k}} = \frac{m_{k}}{\mu _{k}} + \lambda $$

Equate it to zero to get $ \mu _{k} = -m_{k} / \lambda $

Now substitute it in the constraint
$$ \sum_{k=1}^{K}\mu_{k} = \sum_{k=1}^{K}\frac{-m_{k}}{\lambda} = 1 $$

Then $ \lambda = \sum_{k=1}^{K}-m_{k} = -N $

Finally, you get your answer
$$ \mu_{k} = \frac{m_{k}}{N} $$


What does Negative Log Likelihood mean? I have a data set which has continuous independent variables and a continuous dependent variable. To predict the dependent variable using the independent variables, I've run an ensemble of regression models and tried to compare them against each other. Here are the results for reference:

I can interpret what the R-squared value / Coefficient of determination for each of those models means. However, I can't understand what the Negative Log Likelihood means. Especially, why is it Infinity for Linear Regression and Boosted Decision Tree, and a finite value for a Decision Forest Regression?

Edit:
Data Description: The data that went into these three models is all continuous independent variables and a continuous dependent variable. There are a total of 542 observations and 26 variables.
These 542 variables are split 70 - 30 to get training and testing datasets. Therefore, the training dataset has 379 observations and 26 variables; the testing dataset has 163 observations and 26 variables.
No missing data.

Edit 2 Possible Explanation - (click here): Apparently, Linear Regression and Boosted Trees in Azure ML don't calculate the Negative Log-Likelihood metric - and that could be the reason that NLL is infinity or undefined in both cases.

Likelihood function is the product of probability distribution function, assuming each observation is independent. However, we usually work on a logarithmic scale, because the PDF terms are now additive. If you don't understand what I've said, just remember the higher the value it is, the more likely your model fits the model. Google for maximum likelihood estimation if you're interested.

Obviously, your input data is bad. You should give your model a proper data set. While I don't have your data set, we can take a look at the likelihood function for linear regression:

You will get infinity if the likelihood function is zero or undefined (that's because log(0) is invalid). Look at the equation, most likely your sample standard deviation is zero. If it's zero, the last term will be undefined. Have you given a data set that you copied and pasted the same data over rows?

Boosted trees should also be undefined if your sample deviation is zero. However, decision tree is estimated based on impurity and won't crash here.

Summary: please check and double check your data.

EDIT I think you just had a bug. Linear regression will always give you something here. Have you fitted the models in R with the same dataset? –


Random Forests with Big Data - number of trees v. number of observations I frequently use Random Forest, Regularized Random Forest, Guided Random Forest, and similar tree models.

The size of the data that I'm dealing with has grown beyond what I can work around using HPC and parallelism. It's typically large due to row length (observations) not columns (features). The data is also often not normally distributed.

I have to make a choice between:

Running a small number of trees (i.e. 50 or less) with either complete data or a relatively large and comparative sample
Running several times the number of trees, but with a correspondingly scaled down sample size

There are work-arounds and for any 1 case -- for instance, I can do some ad hoc tests to see which I think will work better, but what I'm wondering is if there is a good theoretical (or robust empirical) reasoning to either guide the choice of approach over the other or to describe the tradeoff being made?

In other words, I'm hoping that someone more comfortable with the math, statistics, and theory underlying this (type of) algorithm can offer some generalizable insight.

I would recommend using a combination of both options #1 and #2.

You could first try tuning your hyper-parameters to find out till what extent could you reduce the number of trees to a point where the random forest model's prediction starts deteriorating on the test set.

This is because changing the value of mtry, the randomly selected number of features for a new tree, is the only meaningful hyper-parameter that should impact accuracy of the model. Since averaging converges as the no. of trees increases, the no. of trees could be reduced to a point where its performance is not impacted as much. Hence, you need to iterate and choose a a limit beyond which very small number of trees may not produce a strong enough ensemble. A random forest needs works best by using more base learners for reducing the variance by averaging each individual tree's output.

It is not clear from your case whether you're using the Random Forest for a a classification or a regression problem. In case this is a classification problem, and if your data-set is imbalanced in terms of ratio of positive vs. negative classes; then you could reduce the size of the training set by under-sampling the majority class to bring it nearer to a 1:1 ratio. Since you have a large number of records, such class based sampling could improve accuracy as well as reduce data size for training.

Additionally, if you've got a fine tuned Random Forest with good performance, then you could also evaluate dropping features that are least important as determined by the algorithm on OOB samples. This would reduce the time taken to train the model.


How to find the various matrix sizes in designing a CNN I am trying to understand CNN especially the maths and working mechanism using Matlab as the coding language. I have few confusion regarding the concept and the associated programming and will be immensely grateful for an intuitive answer.

Below is the structure of my CNN for 5 classes. I could calculate only the output structure of the first Conv layer and stuck on determining the number of parameters i.e., number of neurons?

The output for the first convolution layer that I could calculate: In the first layer an input of size [50 50 2] is convolved
with a set of M_1 5-dimensional filters applied over all the input channels. The first 2 D convolutional layer is composed of M_1 = 20 filters of size [5x5x 1] having the step size (stride) for traversing the input vertically and horizontally as 1 creating a feature map of size {(h-f_h+1) x (w - f_w +1)x 1x M_1} = (50-5+1)x(50-5+1)x20 = [46x46x 20] So we have 20 channels.

For a CNN layer with input of dimensions h * w * d, kernel size k * k and number of kernel filters as f, we have the number of parameters as k * k * d * f, if we ignore the biases. If use biases then the number of parameters becomes (k * k * d + 1) * f

For e.g., the 1st conv layer has 5 * 5 * 2 * 20 parameters if we are ignoring the biases. With bias, the number of parameters would become (5 * 5 * 2 + 1) * 20.

Note that the number of parameters does not depend on the stride, padding, pooling, dropout nor on the spatial dimensions of the input or output!

To find the total number of parameters in the network one needs to add the parameters of all the individual layers in the model.


Network architecture of AlphaZero I am curious about the network architecture of AlphaZero. Can anybody tell me details about it? Is it the same as in AlphaGo Zero?

The AlphaGo Zero paper tests a few different architectures:

Seperate 12-convolutional-block policy and value networks (as in AlphaGo)
A single 12-convolutional-block network with dual policy and value outputs
Separate 20-residual-block networks
A single 20-residual-block network with dual policy and value outputs
A single 40-residual-block network with dual policy and value outputs

In the AlphaZero paper AlphaZero Go wins 60 games in a 100-game match against the 20-block single resnet AlphaGo Zero, but it is not explicitly stated which architecture AlphaZero is using. It is either the 40-block single resnet since this was the best, or it is the 20-block single resnet in order to make a fair comparison with AlphaGo Zero.


Sklearn Random Feature Importances Identical for Predicting Different Response Variables I have created four random forest models they have the same X data, but their y data are four different response variables. The sklearn random forest feature importance is identical for all four. All four models achieve their purpose and make different predictions, but their random forest feature importance is the same.
Has anyone experienced this before?
I created the models with a series of nested objects like illustrated below. I used the same code before without having identical random forest feature importances, there was however the difference that inside each object I ran a 3-fold CV to determine max_features, whereas here I just used the default which is all of them.
Current code:
class NoCVMethod:
    def __init__(self, X_train, y_train, X_test, y_test, y, Method):
        self.clf = Method
        self.clf.fit(X_train, y_train)
        self.predictions = self.clf.predict(X_test)
        self.rev_preds = rev_pred(y[-(13978+97):].values,self.predictions)
        self.residuals = y_test - self.rev_preds
        self.RMSE = np.mean((self.residuals)**2)**0.5

class Different_variables:
    def __init__(self, X_train, y_train, X_test, y_test, Method):
        self.TSS = NoCVMethod(X_train, y_train[y_train.columns.tolist()[0]], X_test, y_test[y_test.columns.tolist()[0]], y[y.columns.tolist()[0]], Method)
        self.NOx = NoCVMethod(X_train, y_train[y_train.columns.tolist()[1]], X_test, y_test[y_test.columns.tolist()[1]], y[y.columns.tolist()[1]], Method)
        self.NH4 = NoCVMethod(X_train, y_train[y_train.columns.tolist()[2]], X_test, y_test[y_test.columns.tolist()[2]], y[y.columns.tolist()[2]], Method)
        self.PO4 = NoCVMethod(X_train, y_train[y_train.columns.tolist()[3]], X_test, y_test[y_test.columns.tolist()[3]], y[y.columns.tolist()[3]], Method)

Old code:
class CVMethod:
    def __init__(self, X_train, y_train, X_test, y_test, y, param_dict, Method):
        self.pipeline = Pipeline([
            ('scale', StandardScaler()),
            ('clf', Method)
        ])
        self.param_grid = param_dict
        self.grid = GridSearchCV(self.pipeline, param_grid = self.param_grid, cv = 3, verbose = False, n_jobs = -1)
        self.grid.fit(X_train, y_train)
        self.predictions = self.grid.predict(X_test).ravel()
        self.rev_preds = rev_pred(y[-(13978+97):].values,self.predictions)
        self.residuals = y_test - self.rev_preds
        self.RMSE = np.mean((self.residuals)**2)**0.5

class CVDifferent_variables:
    def __init__(self, X_train, y_train, X_test, y_test, param_dict, Method):
        self.TSS = CVMethod(X_train, y_train[y_train.columns.tolist()[0]], X_test, y_test[y_test.columns.tolist()[0]], y[y.columns.tolist()[0]], param_dict, Method)
        self.NOx = CVMethod(X_train, y_train[y_train.columns.tolist()[1]], X_test, y_test[y_test.columns.tolist()[1]], y[y.columns.tolist()[1]], param_dict, Method)
        self.NH4 = CVMethod(X_train, y_train[y_train.columns.tolist()[2]], X_test, y_test[y_test.columns.tolist()[2]], y[y.columns.tolist()[2]], param_dict, Method)
        self.PO4 = CVMethod(X_train, y_train[y_train.columns.tolist()[3]], X_test, y_test[y_test.columns.tolist()[3]], y[y.columns.tolist()[3]], param_dict, Method)
``

It seems that your self.clf points to your Method. At the end, you are probably printing the features importance of a unique classifier.
Maybe you should copy it:
from sklearn.base import clone

class NoCVMethod:
    [...]
    self.clf = clone(Method) # only copy the estimator
    # OR
    self.clf = deepcopy(Method) # if you want to also copy the data estimator

See here (or here as you suggested) for more details about copying an sklearn estimator.


Convolutional Neural Network overfitting I built a CNN to learn to classify EEG data (only about 4000 training examples, 2 classes, 50-50 class balance). Each training example is 64x512, with 5 channels each

Ive tried to keep the network as simple/small as possible for testing:

ConvLayer (4 filters)
MaxPool
Dropout 50%
Fully connected (50 neurons)
Dropout 50%
Softmax

Im also using weight decay (L2 reg, lambda = 0.001)

The problem is no matter how I play with the filter parameters (size, stride, number) my network keeps overfitting. It fits the training data 100%, but no matter what I do I can't get the test accuracy over 65%.

Why is such a small network overfitting? I thought it was a sample size issue, but I've read a number of research papers on EEG and BCI and they occasionally have even smaller sample sizes than I do

What else can be done to regularize a CNN?

This is because you have very little amount of data. If not enough data is provided to CNNs, they are very likely to overfit. You can do the following things in order to overcome this problem:

Data Augmentation : It is a technique to create new examples from the training examples by doing some preprocessing on them e.g. Rotation, Scaling, etc.

Rigorous Dropout : Dropout is a very powerful technique to control overfittting. You can try using more rigorous dropout in your architecture


How to compare 10000 data frames in Python? I have 10000 data frames like this:
Each data frame corresponds to a different card game (and so different numbers in the table) and I want to compare these data frames all together. For example, I want to compare the heat maps of these data frames. Is there anyway to do this in Python? Is there any toolset so that I can compare all of them in one diagram or something like that? Because I want to see a trend in all of these 10000 data frames.

Simple approach:

Define an appropriate distance or similarity measure between two such data frames. It's unlikely that there is a standard measure for whatever game this represents. For example you could have a distance measure which sums every value in each data frame and then return the absolute difference between the two values, but this is unlikely to correctly represent the semantic of the game.
Implement this distance or similarity measure between two such data frames as a function
Call this function for every possible pair of data frames (that's 49,995,000 comparisons, it's doable).

Fair warning: a heatmap of an unordered matrix 10k x 10k might not be very exploitable.
Note that once you have the distance/similarity function, you could also:

use a clustering algorithm to group the data frames by similarity
detect outliers


Classification for 'not something' Neural Networks This is a very basic question about neural networks in general.

How do you have a classification for 'not something'?

Example: Say that you have to determine the accuracy of a certain simulation by comparing it to the actual scenario (in this case, comparing simulated and actual galaxy images). Theoretically, I can make a CNN and classify these images as 'galaxies' and 'not galaxies'. But what kind of images do I need to use for 'not galaxies'? I only have images for true galaxies.

Your problem resembles the learning task of 'one-class classification', otherwise known as anomaly detection. Essentially, you have a set of images all belonging to one class (galaxies), and you want a model to tell you if a new image is a galaxy or not, without providing any counter-examples to train with.

You should try a recently developed method called Deep Support Vector Data Description. In this method, a transformation is learned by a neural network to map the inputs to points contained in a small hypersphere in latent space. Then, examples falling inside the hypersphere can be considered 'galaxies', and outside can be considered 'not galaxies'.

You can read more about it here (quite technical), or just try to use the authors' PyTorch implementation here.


Text similarity with sentence embeddings I'm trying to calculate similarity between texts with various lengths. My current approach is following:

Using Universal Sentence Encoder, I convert text to a set of vectors.
I average these vectors to create the final feature vector.
I compare feature vectors using cosine similarity.

This gives me pretty good results for texts with roughly same sizes, but I was wondering if there is a better approach for the step #2 if texts have different lengths.

One approach is using Word Mover’s Distance (WMD). WMD is an algorithm for finding the distance between texts of different lengths, where each word is represented as a word embedding vector.

  The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to "travel" to reach the embedded words of another document.

For example:

Source: "From Word Embeddings To Document Distances" Paper

WMD can be modified to Sentence Mover’s Distance, comparing how far apart different sentence embeddings are to each other.


CNN - Is this a Toeplitz Matrix? I have been reading through Chapter 9 of www.deeplearningbbook.org, where convolutional networks are being described.

The following image represents the output of a 2D convolution, without kernel flipping.

The book goes on to describe this matrix as a Toeplitz matrix where,

  for univariate discrete convolution, each row of the matrix is constrained to be
  equal to the row above shifted by one element.

I fully understand this statement since w, x, y and z are constants in their respective columns with shifting elements.

However, there is no mention of diagonal-constants which are a key feature of such matrices. As per Wikipedia (link above) and several other sources:

  a Toeplitz matrix or diagonal-constant matrix, named after Otto Toeplitz, is a matrix in which each descending diagonal from left to right is constant.

Going back to the image above, does this mean that, for instance, aw == cx == gy? How can this be ensured when all elements are different?

First of all, be assured that the kernel matrix (the 2x2 matrix in the figure) in CNN is not constrained to have its diagonal filled with a unique value, and is thus not a Toeplitz matrix.

In www.deeplearningbook.org,  we can read:

  "Discrete convolution can be viewed as multiplication by a matrix, but the matrix has several entries constrained to be equal to other entries"

This means that the global operation of passing a kernel on the input data of a CNN could be expressed as the multiplication of this input data by a matrix. i.e. by a large and sparse Toeplitz matrix.

Authors mention the illustrative simple case of univariate discrete convolution. In univariate discrete convolution, we would be applying a 1-D kernel matrix of length $m$ on 1-D input data of length $n$. Let us take n=10 and m=3.
Doing the convolution could be done by multiplying the input vector data by a n*n matrix, whose diagonal would be composed of the 3 weights of the kernel, shifted of one column to the right at each new row.


Faster grid search with small dataset to derive best params instead of full dataset? I have a dataset of 300 000 rows and an ensemble model, which include grid search to find the best params of every algorithm. Unfortunately the grid search needs to long and I have problems to implement gpu running for the different algorithms (xgb,lightgbm..). Also the models which are from scikit-learn like random-forest dont run on gpu.
The idea is now, instead of 300 000 rows, I will create a small dataset with max. 500 rows, which need less time than the full dataset.

Could the minimal required sample size calculation help?

How big should the sample be? to get a good data distribution of the big dataset?

This idea of using a small sample to the data set to search for the hyperparameters is called multi-fidelity methods.  A good starting point is the book by Frank Hutter, Lars Kotthoff, Joaquin Vanschoren Automated machine learning: methods, systems, challenges which is open access.


Building a Tic Tac Toe game which learns by itself Let's assume we have a huge database gathered which contains winning sequences, which each player played and won. (Both the players are programmed, to obtain random numbers and attempt playing the game).

  What would be the best way to use these winning sequences each player
  has gathered for future plays? Are we going to program one occurrence
  to play without using historical data and few other occurrences using
  historical data?

  What would be the best way to save the winning sequences, the entire
  path which both the player One and Player Two has taken or only the
  sequence used by the winning player?

This is my first attempt to machine learning paradigm and hope some support could be taken!

I took an introductory computer science course on Coursera a few years back where we had to use a Monte Carlo algorithm to build a tic-tac-toe AI.  Instead of using historical data, we generated data using a function that would play through a series of games with random moves, but I think the principle is basically the same.

Each square on the board has a separate score associated with it. For each game completed, the scores change based on which player won.  If the player designated as the machine player won, the machine player's score for each square on the board where it placed a mark would increase and the "other" player's score would decrease for every square where it placed a mark.  Conversely, if the other player won, its score would increase for each square on the board where it placed a mark and the machine player's score would decrease.  If the game is a tie, the number of points for each square would not change.

So, to answer your question, I think it is important to track both the winning and losing player's moves because we need to know not only what works but also what doesn't work.

Edit:  I just remembered that my first implementation did not deduct points on the grid for games lost and because of this, my AI had an all-out "offensive" strategy.  In other words, it focused on getting three-in-a-row without making any effort to block the human player.  This is why it's important to track moves that result in a loss.


Creating a neural network, composed of n times a different network. Is it possible? I'm currently working on a project with a bunch of data of devices that can either belong to people, or not. The ultimate goal is to estimate a number of people detected.
Sadly, it is impossible to get labels for which device should be classified as human and which shouldn't. So a basic binary classification network will not be trainable.

I do however have for each moment in time a number of people, that can be correlated to the number of human-devices that should be detected (it is not the exact sum of devices that should be detected because we won't be able to detect all).

So... I though of this weird construct, where I have a binary classification neural network (green box in the picture) copied n times along itself, and then have its outputs propagate to an outer network which will perform a regression using both the outputs of the classifiers and some other variables (pink) that will determine the scaling (for example, if only 50% of the humans are detectable using the devices, then the scaling will have to be 2. But this is a non-linear function so it'll also be determined by the NN).

The green neural networks should all share the same weights, because they will perform identical work.

Another possibility would be to use the pink scaling variables from the outer (red) NN, to the green-NN's, and then let it result in a non-binary value, but more like a scaled value. So in the same example as above, a 1 would return 2.

Would this be possible in theory? Has it been done before? What would be the best approach for this?

I already had a talk with my adviser about this, but he was very skeptical about it.

I think it's possible. I don't know how well it will work out for you, but I think it's technically possible (which is the best kind of possible).

For starters, I would suggest using the functional API for Keras for this kind of thing. It makes it super easy to do composition of networks like this, and residual networks like this implementation on GitHub are composed in a similar fashion. Something like the following might get you going (note this isn't meant to be a full solution, sorry):

from keras.layers import Input, Dense
from keras.models import Model

def single_net(input_shape, filters):
    def f(input):
        h0 = Dense(filters, activation='relu')(input)
        h1 = Dense(filters, activation='relu')(h0)
        prediction = Dense(2, activation='softmax')(h1)
        return prediction
    return f

input_a = Input(shape=input_shape)  # whatever your input shape is
input_b = Input(shape=input_shape)
# ... repeat for however many people you have ...
shared_net = single_net(input_shape, filters)  # define once
network0 = shared_net(input_a)
network1 = shared_net(input_b)
# ... again repeat for each person
combined_nets = keras.layers.concatenate([network0, network1, ...])
dense = Dense(output_shape, activation='relu')(combined_nets)  # or however you want to combine them
model = Model(inputs=[input_a, input_b, ...], outputs=dense)
model.compile(optimizer='adam', loss='binary_crossentropy')  # whatever optimizer, loss you like
model.fit([data_a, data_b, ...], labels, ...)

By defining your repeated network only once (shared_net above) and then using that same object for each input, you are sharing those layers across all of those inputs. Basically when you back-propagate, all inputs/outputs will train a single instance of the network, even though you're basically using duplicates of it for each input.

The big question is how do you combine all of your individual models, and how do you deal with changing numbers of models. I don't know how to deal with an uncertain number of models, but I suggest looking at the shared layers section of the Keras functional API documentation for some help on combining, and dig in to some examples on GitHub. What's shown here ought to get you started, but there are probably some errors hiding throughout. Like I said, not a complete answer, sorry.


Confusion between doing .copy() and using [[ ]] I am reading "Hands-On Machine Learning with Scikit-Learn and Tensorflow" by Aurélien Géron.
Sometimes he does
xyz = DataFrame[["col1"]]

Which returns a dataframe, but other times:
xyz = DataFrame["col1"].copy()

Which reuturns a series
I do understand that both are ways of copying and not using a 'view', but when should we use each?
(ps: I only found information about views refering to arrayas in numpy... Are the arrays in numpy the same DataFrames in Pandas?)

Both are more or less the same thing.
xyz = DataFrame[["col1"]] is useful when you have to select one or more than one column from a dataframe. For example: xyz = DataFrame[["col1","col2"]]
While xyz = DataFrame["col1"].copy() or df.copy() is used for copying a complete/partial dataframe.
Note that both are exactly the same.


Best way for data preparation to have accurate prediction I'm trying to experiment if an opportunity will win or lose in Azure Machine Learning studio. However, am still in Data preparation method.

In my Data base I have opportunity table and products table.

For example, one opportunity has multiple products. Should I deal with the many products and put them in one record?

Will it affect the prediction if we have duplicate records for an opportunity like (a) or it’s better to have one record per opportunity in order to feed it to ML studio. And if yes which one will be better approach (b) or (c).

Approach a

Approach b

Approach c

oppid |first product |first technology|2nd product |2nd technology
1      out-services   active directory   TRN-items Adobe Acrobat

The simplification of the data may make the model more stable, but it will also remove its ability to use more specific input criteria. For example, the way you move from Approach A to Approach B you are aggregating specific products into product categories. This means that if your model is successful in Approach A, it will be able to predict based on specific products. On the other hand, if your model training succeeds in Approach B it will only be able to predict on product category (and you will have to convert products into its categories before supplying it to the model).

So to answer your questions, the number of data samples you have determine how much you have to aggregate and simplify your data. The data itself in its most detailed form could also fail to train the model properly, in which case the approach taken in Approach B is the best next step.