Machine Learning Identification and Classification, based on string contents: General advice I have just very recently started to develop an interest in machine learning, and I have a particular problem in mind that I would like to start to explore.

I would like to train a system to automatically classify various attributes of an item, based on what's in a string.

Let's say I have a long list of various mutual funds, like:

Ticker  Fund Name
------  ---------
ABNAX   ABC Bond Fund, Inc: Bond Inflation Strategy
ALYSX   ABC Bond Fund, Inc: Credit Long/Short Portfolio; Advisor Class
AGRXX   DEF Bond Fund, Inc: Government Reserves Portfolio; Class 1 Shares
HIYYX   FGH Bond Fund, Inc: High Yield Portfolio; Advisor Class Shares
HIYAX   FGH Bond Fund, Inc: High Yield Portfolio; Class A Shares
...

… And so on.

I have a large data set that contains "complete" classifications, which have Fund Names similar to the ones above, and – in addition – a human has already given the training set items certain attributes. For example:

AIISX   Allianz Funds Multi-Strategy Trust: AllianzGI International Small-Cap Fund; Class R6 Shares

Which will have the associated attributes:

Strategy: Multi-Strategy
Geography: International
Capitalization: Small-Cap
Share class: R6

The challenge for the machine learning system will be to assign the right value to an attribute, when there are values "competing" on the same attribute. Let's say that a certain fund can have Strategy: Long-Short and Strategy: High Yield at the same time – and both terms are present in the Fund name. The system should select the right one, based on exposure to historical bias present in the training data set.

Question

I am interested in getting a grasp of which machine learning methods and algorithms that would be able to "learn" how to classify an item, based on a large set of examples with human-classified attributes, as indicated above.

I am a complete beginner to machine learning, except for some basic knowledge of statistics, so I would just like to be pointed in a general direction.

Can/should this be accomplished with something like multiple regression, or are we looking at something else? Is some sort of natural language processing needed – or is basic keyword pattern recognition enough?

Lastly, which terminology or labeled area of expertise would summarize this problem description?

If the content/information is lengthy, I'd suggest you to use some NLP tasks for starters. I would suggest you to use some basic NLP based preprocessing because it makes our model perform better. So, the basic feature extraction can be used for this. Example, using Porter Stemmer, Lemmatizer to clean the data or removing stop words and then using ngrams for features seem to be a basic idea and a good start. There are various vectorizers which can be used to extract features the documents. For example, TfidfVectorizer calculates the frequency of a word in a document and also frequency across documents. This can be more useful than a naive Bag of words approach. Then, on top of this there are various classifiers which can be used like OneVsRestClassifier or others.

A simple approach could be selecting the input and target first. Select the parameters which are to be passed as input and the desired output. Then, decide to clean the input or not based on some NLP APIs(you can use nltk). Then decide on a classifier. You can then predict the values. Test on validation set and
try various classifiers for starters.

As for terminology, I can think of Multiclass Classification only now.


Predicting Missing Word in Text I know about BERT and other solutions when you masking some words and try to predict them.
But let say I have a text:

  Transformer have taken the of Natural Processing
  by storm, transforming the field by leaps and bounds. New,
  bigger, and better models to crop up almost every ,
  benchmarks in performance across a wide variety of tasks.

And I cannot in advance say to BERT where masking is. I am looking for an algorithm which can understand where missing words are and after that predict them.

It's not simple, but doable. I suggest you to create training data in the following way: take a text corpus, as large as possibile, and remove words sampled randomly. Then train an seq2seq RNN to map this "deteriorated" text with its original.

The RNN you need won't be too different from an NMT model, but it's goal is different of course.

It's the first time I encountered this kind of task, therefore I can't say what is the state of the art.


How choosing a value for random_state argument in sklearn.model_selection works? I have been using sklearn for quite some time and I understand using the same number say 100 or 200 as a value for the random_state argument will help me to produce the similar train and test data sets. However, how does it work?

Is it some kind of initialization? Do we need to pick a value based on data size? I want to know how random_state works.

In the documentation (for example for KFold) you can find the following explanation:

random_state : int, RandomState instance or None, optional, default=None
If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by np.random. Used when shuffle == True.

Obviously, the way you use it falls into the first use case. Thus, the number is used as a seed for the pseudorandom number generator. When you have the seed hardcoded as a parameter, you use the same seed for a second time. This will result in the same sequence of pseudorandom numbers, thus not random at all. Because of this, use numeric seeds only for testing purposes when you want to reproduce your "randomness". For learning experiments and production purposes, you can just leave the default None value.


Can AlphaGo Zero adapt to oponents skills/profile? I read the AlphaGo Zero paper and I didn't found nothing about it in there. But I would like to know if AlphaGo Zero can adapt to the way the oponent plays (oponent profile) or something like this. Thanks!!

It's not important to adjust to the opponent, as Go is only about winning or losing. There is no bigger reward the faster / the more obvious it wins.

Or to put it different: only the current board situation is important in a min-Max setting (although the value approximation of a state admittedly depends a bit on the opponent)


How to sum values grouped by two columns in pandas I have a Pandas DataFrame like this:

df = pd.DataFrame({
    'Date': ['2017-1-1', '2017-1-1', '2017-1-2', '2017-1-2', '2017-1-3'],
    'Groups': ['one', 'one', 'one', 'two', 'two'],
    'data': range(1, 6)})

    Date      Groups     data
0  2017-1-1    one       1
1  2017-1-1    one       2
2  2017-1-2    one       3
3  2017-1-2    two       4
4  2017-1-3    two       5

How can I generate a new DataFrame like this:

    Date       one     two
0  2017-1-1    3        0
1  2017-1-2    3        4
2  2017-1-3    0        5

pivot_table was made for this:

df.pivot_table(index='Date',columns='Groups',aggfunc=sum)

results in

         data
Groups    one  two
Date
2017-1-1  3.0  NaN
2017-1-2  3.0  4.0
2017-1-3  NaN  5.0

Personally I find this approach much easier to understand, and certainly more pythonic than a convoluted groupby operation.  Then if you want the format specified you can just tidy it up:

df.fillna(0,inplace=True)
df.columns = df.columns.droplevel()
df.columns.name = None
df.reset_index(inplace=True)

which gives you

       Date  one  two
0  2017-1-1  3.0  0.0
1  2017-1-2  3.0  4.0
2  2017-1-3  0.0  5.0


model predicting probability close to 50 for positive classes in imbalanced training dataset I have a binary classification model where I am predicting the positive class which is only 10% of whole training data set. The issue with this imbalanced data set is my model is predicting probability not more than 0.6 for the positive class.
Why is it happening?

I know it is due to imbalanced data set and there are some answers which suggests how to deal with imbalanced data sets.

I have dealt with imbalanced data sets before and have never encountered such a case where probability for a class is not more than 0.6.

More than the probability, what you can consider is the confusion matrix you get after applying the threshold on the probability. If, for example, your threshold is 0.5 and your model is perfectly classifying your validation examples, you shouldn't have a problem.

If you take average log loss as your training metric, with equal weights for positive and negative classes, your average probability values will get shifted towards the dominant side in the training set.

Summarizing, probability values in themselves may not be a cause of worry. Focus more on imbalanced data specific metrics like precision, recall(sensitivity) specificity, F1 score or Area Under the ROC curve.


neural network probability output and loss function (example: dice loss) A commonly loss function used for semantic segmentation is the dice loss function. (see the image below. It resume how I understand it)

Using it with a neural network, the output layer can yield label with a softmax or probability with a sigmoid.
But how the dice loss works with a probility output ?

The numerator multiply each label (1 or 0) of the predicted and ground truth. Both pixels need to be set to 1 to be in the green area. What is the result with a probability like 0.7 ? Does the numerator result in a floating number (ie with ground truth = [1, 1] and predicted = [0.7, 0], the "green" area of the numerator would be 0.7) ?

I think there is a bit of confusion here. The dice coefficient is defined for binary classification. Softmax is used for multiclass classification.

Softmax and sigmoid are both interpreted as probabilities, the difference is in what these probabilities are. For binary classification they are basically equivalent, but for multiclass classification there is a difference.

When you apply sigmoid, you make sure that all output neurons are between 0 and 1.

When you apply softmax, you make sure that all output neurons are between 0 and 1 AND that they sum up to 1.

This means, when the output is sigmoid, the input data can be in several classes at the same time. For softmax, you force the network to pick one of the classes.

The code you posted here is correct for binary classification, but for multiclass, there is some intricacy when it comes to combining the classes.

Since in your example the target consists of two pixels, each labeled either 0 or 1, you are dealing with binary classification and should use pixel-wise sigmoid in the first place, i.e. the probabilities from your model should be e.g. [0.7, 0.8] or something like that.

Pixel-wise softmax should only be used if each pixel could be in only one of many classes and softmax over all pixels does not make much sense, as this would force the model to pick one out of many pixels to label as 1.


Classification of vector sequences My dataset is comprised of vector sequences. Each vector has 50 real-valued dimensions. The number of vectors in a sequence range from 3-5 to 10-15. In other words, the length of a sequence is not fixed.

Some fair amount of the sequences (not vectors!) are annotated with a class label. My task is to learn a classifier that given a sequence of vectors, the class label for the whole sequence is computed.

I cannot tell the exact nature of the data but the nature of sequences is not temporal. Nevertheless, a vector $x_i$ cannot be interchanged with a vector $x_j$ without changing the label ($i \neq j$). In other words, the order of vectors is important. The vectors themselves are comparable, for example it makes sense to compute a dot product and use this similarity value.

My question is: what are the tools/algorithms that can help to classify such data?

UPDATE:
The data has such a property that one or very few vectors influence strongly the class label.

POSSIBLE SOLUTION:
After some research it looks like the Recurrent Neural Networks (RNN) fit the bill pretty naturally. The overarching idea is to pick a context size $k$, concatenate word vectors, do max pooling and feed that through classical NN. At each possible context window position in a sentence, a feature vector is built. The final feature vector is built using max pooling for example. The backpropagation is done to adjust the network's parameters. I already got some positive results (GPU is a must).

As you can't disclose much detail, I'm forced to be a bit generic in my answer. I hope it will be helpful nevertheless. First of all, I would only consider reducing the sequences before classification (be it by using the dot product or something else) if you can make sure that you don't lose information you need for classification afterwards. So this approach is only feasible if you have some insight into the nature of the classification. To give a simple example: if the class label is just the number of vectors in your sequence, you won't be very successfull in predicting the class label from the dot product.

Hence, I would take the full sequence as an input for classification, and impose a maximum on the sequence length you want to consider. You might do this by first finding the maximum sequence length m in your training set and then turning each sequence of 50-dimensional vectors into one vector of dimension 50*m, possibly with some missing values at the end if your sequence doesn't have maximum length. You will probably want to get rid of these missing values and you may want to simply replace them by zeros.

There's two roads you can go from here:
1.) You directly apply classification methods known to be suitable for high dimensions. Try something simple that doesn't need much tuning like naive Bayes. This way you can see whether this approach is feasible without losing too much time if it isn't.
2.) You try first to reduce the dimension and understand the nature of the classification better. You may want to use something like principal components analysis or analyse correlation/association between each vector component and the class label. If you're successful, you know how to properly reduce the dimension of your input before applying classification.

If you would like to follow any of these ideas, please keep in mind that the concrete details of your data and the classification may render any of the ideas proposed above infeasible. So please be careful to check against any details you know but can't post here before trying to make sure you're not wasting your time.


Finding gender affinity for businesses What are the different models I can use to find the gender affinity of businesses using yelp dataset-- https://www.kaggle.com/yelp-dataset/yelp-dataset . I need to find Probablity (Male buying from a merchant) and Probablity (Female buying from a merchant) where both probablities add to 1 . There is no information about gender so I can use the Genderize api to find gender using names of users. This is an unsupervised problem

Since you're only interested in the gender affinity of businesses in your dataset, no model is needed.  You can compute gender affinity directly:  For each business, find the total number of customers, the total number of male customers, and the total number of female customers.  Then $Pr(Male) = \frac{males}{total}$ and $Pr(Female) = \frac{females}{total}$


Can data analytics be a basis for artificial intelligence? I'm very passionate about how computers can be made able to think intelligently and independently (in our favour, of course!). I'm currently studying Bachelors science of information technology at UTS (University of Technology:Sydney). I have two months before I start my second year, and have not yet been able to decide on which major should I select that can lead myself towards dedicated study of Artificial Intelligence (which I love with my life).

I have the following majors available:

Internetworking and Applications
Data Analytics
(there are other two as well, but business oriented).

Here is the link to my subjects. I believe that being able to play with data is a sign of intelligence (I may be wrong too!). Will one of these subjects form me a good foundation for my further study in A.I.? Or should I jump into Engineering? Or Pure Science?

Classes related to Artificial Intelligence are typically taught in Computer Science departments. Looking at the IT Project Subjects offered by your university, I suspect Data Analytics would indeed be more relevant to AI than Internetworking and Applications.

Looking at the courses offered by your department, the following likely involve aspects of AI:

Image Processing and Pattern Recognition
Intelligent Agents
Building Intelligent Agents

For self-directed study in AI, I recommend starting with Russell & Norvig's essential textbook Artificial Intelligence: A Modern Approach.

As to what it will take to create a human-like strong AI, I recommend this collection of essays: The Philosophy of Artificial Intelligence... even though the material is getting a bit out-of-date by now.

Good luck!


Difference between LASSO penalty in neural network and just LASSO regression I wonder whether those two have any significant differences.

I think in neural network, the lasso penalty put on the loss function makes the model simpler and introduces more sparsity by automatically reducing some of the features to 0. I assume that's basically the lasso regression is doing?

I think Lasso Regression, is a type of regression model which implements the lasso penalty (L1 regularization) within its algorithm (different from normal regression algorithms).

Lasso penalty is more of a generalized penalty that can be put on any learning algorithm.

The concept behind both ideas are the same, but their area of implementation is different.

Hope this helps !!


Bounding box coordinates prediction How to pass multiple bounding boxes coordinates to CNN model?My goal is to predict the coordinates of texts in an image.

Please look into the Mask R-CNN (Faster CNN)
https://github.com/matterport/Mask_RCNN/


How to write out the definition of the value function for continous action and state space In the book of Sutton and Barto (2018) Reinforcement Learning: An Introduction. The author defines the value function as.

$$v_{\pi}(\boldsymbol{s})=\mathbb{E}_{\boldsymbol{a}\,\sim\, \pi}\left[\sum_{k=0}^{\infty}\gamma^kR_{t+k+1}\,\bigg|\,\boldsymbol{s}_t=\boldsymbol{s} \right]$$

If $\boldsymbol{a}\in \mathcal{A}$ and $\boldsymbol{s}\in \mathcal{S}$ are continuous I would think by using Bellman's equation for the state-value function that this can be written as the integral

$$v_{\pi}(\boldsymbol{s})=\int_{\boldsymbol{a}\in\mathcal{A}}\pi\left(\boldsymbol{a}|\boldsymbol{s} \right)\int_{\boldsymbol{s}'\in \mathcal{S}}p(\boldsymbol{s}'|\boldsymbol{s},\boldsymbol{a})\left[R_{t+1}+\gamma v_{\pi}(\boldsymbol{s}')\right]d\boldsymbol{s'}d\boldsymbol{a}.$$

Is this correct?

Also without using Bellman's equation does the integral definition of the state-value function look like this?

$$v_{\pi}(\boldsymbol{s})=\int_{\boldsymbol{a}\in\mathcal{A}}\pi\left(\boldsymbol{a}|\boldsymbol{s} \right)\int_{\boldsymbol{s}'\in \mathcal{S}}p(\boldsymbol{s}'|\boldsymbol{s},\boldsymbol{a})\left[R_{t+1}+\gamma \left[\int_{\boldsymbol{a}'\in\mathcal{A}}\pi\left(\boldsymbol{a}'|\boldsymbol{s}' \right)\int_{\boldsymbol{s}''\in \mathcal{S}}p(\boldsymbol{s}''|\boldsymbol{s}',\boldsymbol{a}')\left[R_{t+2}+\gamma\left[\cdots\right] \right]d\boldsymbol{s''}d\boldsymbol{a}'\right] \right]d\boldsymbol{s'}d\boldsymbol{a}$$

Are my integral versions correct?

this can be written as the integral, is this correct?

Yes. Your derivations imply that we have assumed a deterministic reward given current state-action $(\boldsymbol{s},\boldsymbol{a})$. An stochastic reward model would be $p(\boldsymbol{s}', r|\boldsymbol{s},\boldsymbol{a})$ which requires an additional integral over reward $r$ (for example, equation (3.14) page 47)

  Are my integral versions correct?

Yes. You are unfolding the recursive definition. An illustration would be the recursive definition for factorial:
$$f(n) = nf(n-1);f(0)=1$$
Which is unfolded as:
$$f(n) = n [(n-1) [(n-2)[...]]]$$
However, the difference is that the index in Bellman equation  is going forward since current value depends on future values not the previous ones.


Distinguish randomly generated texts from reasonable for human texts I have strings short texts of 2 types:
'23jd2032n0d2mn', 'fn830n30rn83', 'fhui29n4ok', 'qn4foml', ...
and
'sweetie23', 'king3prussia', 'maryjesus', 'lovedog4and_kitties', ...
Is there a way to distinguish one type from another?
I've tried to vectorize texts with word2vec and classify on these vectors with xgboost, but I didn't succeded to achieve got F1-score.

You could train a character-level language model, e.g. an LSTM, on the real short texts, and use the perplexity as the signal to know whether a piece of text is real or not.
In order to find an appropriate perplexity threshold, you can have a look at the distribution of perplexities over a validation holdout dataset.
UPDATE: There are multiple implementations of language models. For "classical" options, you can go for KenLM or if you have GPUs to train the model, you can use fairseq. Just remember to prepare your text to have character-level tokens (normally you just need to have a space between every letter).


Testing bimodality of data I am trying to see if my data is multimodal (in fact, I am more interested in bimodality of the data). I performed dip test and it does evidence against unmodal data. However, I want to see, in particular, if it is bimodal. I believe silver man's test can be used. However, I couldn't find the implementation of it in either r or in python. (The one in R is old and not working with the current version of R).
Also, assuming that I have a bimodal data and that I am able to get the two components (using mixtools in R), how do I figure out how to find the point of intersection of the two components. For example, here is the histogram (overlaid with its density estimation) of the entire data.

Here are the two components:

I want to get the the x value where the curves intersected. I could have uploaded the data, but the length of the vector is rather long. Any general thought and idea is welcome including the R and/or Python packages are welcome.

Thanks

There has been an answer on this question, please check it :

Test for bimodal distribution


Fraud detection using machine learning I would like to implement a machine learning algorithm for data like this.

id                Date     Amount isFraud
1  100 2018-03-20 10:20:10  110.0      No
2  101 2018-03-25 11:20:20    2.0     Yes
3  101 2018-03-25 11:20:22    2.5     Yes
4  101 2018-03-25 11:20:25    1.7     Yes
5  101 2018-03-25 11:20:27    2.1     Yes
6  101 2018-03-25 11:20:30    2.9     Yes
7  102 2018-03-29 11:00:11  290.0      No
8  110 2018-04-15 08:10:05    1.5     Yes
9  110 2018-04-15 08:10:09    2.3     Yes
10 110 2018-04-15 08:10:13    2.7     Yes
11 110 2018-04-15 08:10:17    1.9     Yes
12 110 2018-04-15 08:10:21    2.9     Yes
13 110 2018-04-15 08:10:27    2.2     Yes
14 110 2018-04-15 08:10:45    2.1     Yes
15 113 2018-04-15 10:10:05  190.0      No

Algorithm needs to work in the following way. Transactions with lower amount are classified as scam. Frequent transactions from same id with same amount or lower amount are also scam. The above dataset was created for this purpose. For these kinds of problems, should we write our own algorithms by specifying the conditions? Or can we use any existing algorithms?

should we write our own algorithms by specifying the conditions?

If you know the exact conditions that result in fraud, you can implement a decision tree for that. But, I believe it is going to be really difficult to come up with that set of conditions that can give clear decision boundary. The data you have created makes the problem look easier than it actually would be. Someone who did a transaction of low amount quite a few times might not be a fraud.

  Or can we use any existing algorithms?

Yes.

This is a classification problem which can be handled through machine learning techniques like logistic regression, support vector machines, neural networks (Multilayer perceptron), Random forest etc.

You can also try Naive Bayes Classifier, which can outperform complex techniques like support vector machines, neural networks.


The effect of an linear layer? I have the last couple of month worked with an regression problem, of turning a framed audio file into a set of mfcc features, for a  speech recognition application

I tried a lot different network structures, Cnn, different normalisation techniques, different optimizer, adding more layers and so on..

but finally i've got some decent result, but i don't understand why..
What i did was i added a linear layer as output, and somehow that minimised the error tremendeosly, and bit puzzled why a linear layer would have that much effect?...

I mean am still tried to fit the actual output to the desired output?..
Why would the activation function matter here?... I mean the weight are being adjusted based on the error, so why is the neural network better at adjusting for the error when the output is linear rather than non-linear (such as: tanh, Relu).. ?

If you are performing regression, you would usually have a final layer as linear.

Most likely in your case - although you do not say - your target variable has a range outside of (-1.0, +1.0). Many standard activation functions have restricted output values. For example a sigmoid activation can only output values in range (0.0, 1.0) and a ReLU activation can only output positive values. If the target value is outside of their range, they will never be able to match it and loss values will be high.

The lesson to learn here is that it is important which activation function you use in the output layer. The function must be able to output the full range of values of the target variable.


Neural Network cannot learn nonlinear function I am currently creating a neural network to learn a function of the following form Data that I want to learn x corresponds to x axis and y to y axis(one dependent and one independent variable)

I am using both keras and tensorflow and with both scripts I get the following result

I am currently creating a neural network to learn a function of the following form Data that I want to learn x corresponds to x axis and y to y axis(one dependent and one independent variable)

I am using both keras and tensorflow and with both scripts I get the following result Predictions orange line Data blue line. Somehow my neural network doesn't capture the non-linearity of the data and only tries to fit a linear function. Do you maybe have a suggestion what I am doing wrong? Also is the architecture appropriate for the following task or there exists some problems.

Additionally as an information I also include a snippet of the architecture that I am using in keras

    def individual_model(keys, labels, config):
model = Sequential()
model.add(Dense(32, input_dim=1))
model.add(LeakyReLU())
for i in range(2):
    # if str(i) not in config:
    #     break
    model.add(Dense(32))
    model.add(LeakyReLU())
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse', metrics=[
              max_absolute_error, 'mse', 'mae'])
model.fit(keys, labels, epochs=100, batch_size=32, verbose=1)
return model

You should try using a sigmoid activation function, as leakyRelu is very close to a linear function, and remove one hidden layer. Just one hidden layer should be engough as you have only one input, with a very small length, maybe 3 or 5 neurons.


PyTorch is too heavy for deployment, is there a light weight pytorch for running neural networks? We deploy PyTorch models in docker container, which massively increased the size of the docker container by more than 1G.

But when we deploy the model the training has already been done, so technically we don't need to include the machinery involved in training.

We don't even need backpropagation, we just need to run the neural network to get the outputs.

Is it possible to include only part of PyTorch (or another totally different product) that executes a neural network and nothing else? So deployment is light-weight.

One thing to try is to create a lean version of Pytorch that only has what you need. I did something similar for an AWS Lambda layer, here's what I was able to delete (this was for Pytorch 1.1, things may have changed since):

find . -type d -name "tests" -exec rm -rf {} +
find . -type d -name "__pycache__" -exec rm -rf {} +
rm -rf ./{caffe2,wheel,pkg_resources,boto*,aws*,pip,pipenv,setuptools}
rm ./torch/lib/libtorch.so
rm -rf ./torch/test

If that's not enough, you need to look into using ONNX or TorchScript/torch.jit. ONNX exports your model to a binary protobuf file. TorchScript exports it to a serialized form that can be run in a C++ environment.


Tensorflow speech-to-text training on single file with tf.nn.ctc_loss not converging I want to train speech-to-text model on tensorflow.

Code at:
https://github.com/bikramjitroy/speech-to-text/blob/master/TrainingUnitTest-CTCTrain.py

A sample "audio.wav" and "label.txt" file also present in repo.
https://github.com/bikramjitroy/speech-to-text

I am loading a wav file. Getting 20 mfcc feature per frame. Passing input to one layer RNN (256 hidden unit) and then added a fully connected layer and then using ctc_loss.

The accuracy is not increasing even with 500 steps with one training example :

Error = 70.932137   # Expecting it should go near zero

Accuracy = 0.62068963     # expecting this will go to zero

I have tried bi-directional rnn as well but that is diverging.

What am I doing wrong in code?

I have found the solution.

Earlier I have a Dropout layer after single layer of RNN. I have changed the rnn_layer function as below.

Removed the DropoutWrapper from the RNN layer - This helped me to overfit
Added 2 layers of LSTM cell. Single layer LSTM cell taking time to
converge but rather than making LSTM layer wide changed it to deep.
This helped me achieve faster convergence than making network wider.

Below is the code of rnn_layer.

def rnn_layer(input_tensor, n_cell_units, dropout, seq_length, batch_size):
    lstm_fw_cell = tf.contrib.rnn.BasicLSTMCell(2*n_cell_units, forget_bias=1.0, state_is_tuple=True)
# Dropout layer dropped
#    lstm_fw_cell = tf.contrib.rnn.DropoutWrapper(lstm_fw_cell, input_keep_prob=dropout, output_keep_prob=dropout)
# Multi layer RNN
    lstm_fw_cell = tf.contrib.rnn.MultiRNNCell([lstm_fw_cell] * 2, state_is_tuple=True)

    outputs, output_states = tf.nn.dynamic_rnn(cell=lstm_fw_cell,
                                               inputs=input_tensor,
                                               dtype=tf.float32,
                                               time_major=True,
                                               sequence_length=seq_length)

    return outputs


Confusion Matrix and AUC in univariate Anomaly Detection In the code I upon a csv file which only has one column. The data in there in not that important just normal numbers.

# Pandas - data handling
import pandas as pd

# Numpy for mathematical operations
import numpy as np
import pandas as pd

# Scikit learn for the DBSCAN algorithm
from sklearn.cluster import KMeans

# Matplotlib - plots
import matplotlib.pyplot as plt

import seaborn as sns

data=pd.read_csv('C:/Users/data.csv', usecols=["meaning_txt", "valueData_value"], sep=';' )

# Add manually an anomaly value
for i in range(0, 50):
    all_data = all_data.append({all_data.column.max() + np.random.randint(1000, 2000) }, ignore_index=True)

clf = KMeans(n_clusters=2, init='k-means++', max_iter=300, n_init=10, random_state=1)

clf.fit(all_data.column.values.reshape(-1, 1))

all_data_prd = clf.predict(all_data.column.values.reshape(-1, 1))
all_data_prd

plt.scatter(all_data_prd, all_data.valueData_value, c=all_data_prd)

My question is if it is possible to have a confusion matrix and roc curve in this case when I only  have once column.

To this column I have added some anomalies.

The goal:

Add these anomalies that are way way bigger than the maximum value
of    the column and then check how many out of the 50 added
anomalies, how    many the K-Means finds
Add these anomalies that are not that much bigger than the maximum
value of the column then check how many out of the 50 added
anomalies are found. At the end I want a Confusion matric and a ROC
graph to see how good they perform.

It's difficult to answer precisely without knowing the data and the task.

Assuming it's a single column of values with no order involved, it boils down to finding the optimal threshold to separate regular values vs. anomalies. Given that you know which ones are the anomalies (labelled data), the problem can be treated as a binary classification task: a value is either regular (negative) or an anomaly (positive).

The idea is simply to evaluate the performance for every possible threshold, i.e. count the number of True/False Positive/Negative for every value. Let's say you take the set $S$ of all the distinct values in your column. For every value $t \in S$ you calculate the confusion matrix (i.e. how many TP/FP/TN/FN) obtained by predicting every value $x$ as positive if if $x>t$, negative otherwise.

From there you can plot a ROC curve and/or find the optimal threshold according to some evaluation measure (typically the one which maximizes the F1-score).

I assume that there are libraries which do that automatically, but it's simple enough so personally I would do my own code for this kind of thing.


Which recommender system: Content based or Collaborative filtering? I want to build a recommender system for a coupons website which should do the following:
Given the past purchase behaviour of a user, recommend coupons which the user is likely to buy.
The data does not have any ratings for coupons by the user. It tells if a user bought a certain coupon or not and what is the gender and location of the user who bought these coupons. Between content-based and collborative-filtering what would be the best recommender system to choose from in this scenario. Personally, I think that content based recommender system fits the requirement because I want to look only at user's own historical behaviour. What would you guys advice? Thanks.

You are right. Content-Based Recommender System is the best approach to tackle such problems using historic information about the users.


Explaining XGBoost functioning to non-technical people I have been tasked to explain the principle of the XGBoost algorithm to non-technical people (think 1-2 slides in a powerpoint presentation to upper management).

I am currently working with the original papers : here for the paper specific to XGBoost and here for the orignal boosting idea by Friedmann.

I have come up with a short explanation for the boosting principle : the model is built iteratively, by concentrating on where the previous model made errors. XGBoost take advantage of the tree architecture and tools (splitting, pruning) to do so.

What would you add to this short description ?

The xgboost docs come with a nice introduction and examples. If it is really only about a few slides, this may be a good start: https://xgboost.readthedocs.io/en/latest/tutorials/model.html


which algorithms can be used to extrapolate non-linear data? I have a dataset, where target value changes in time in following way:

I need to predict target value for upcoming month, however I struggle to find a method to extrapolate the function that defines target. It is certainly not linear.

I tried to extrapolate the max/min peaks separately using linear regression, but not sure what to do with all other values.

That function can be decomposed into composite components:

General trend
Repeated frequencies (looks like there are 3 major cycles)
Random noise

A Fourier transform can decompose a function into its constituent frequencies.


How to generate a sentence with exactly N words? Thanks to GPT2 pretrained model now it is possible to generate meaningful sequence of words with or without prefix. However a sentence should end with a proper endings (.,!,?). I am just wondering how to generate a sentence (with proper ending) of length N?

One possible approach is post-processing, that is process many sequences and choose the ones the serve the purpose! However, it could be a really daunting task to use in any pipeline.

Is there any suggestion, perhaps a secondary algorithm, to tune the hyper-parameter such that it produces sentence of desired length with higher probabilities.

Limit outputs od decoder to N. Not sure how easy it would be, probably a bit digging into official implementation but after that the main "skeleton" of the GPT2 is usable, meaning that all of the pre-training can be reused to produce meaningful sentences.


Model stacking with instance attributes From what I have understood about model stacking: the meta estimator trains to combine the N-models predictions to fit the ground truth. Once trained, it combines the 1st level output to approach the ground truth.

The meta estimator is a model of type :
$ (y_{pred1}, y_{pred2}, y_{pred3})\rightarrow y_{pred-stack}$

So the combination is only based on the values of 1st level predictions. However, each line of the stacking data is also linked to other attributes: "Brand", "Model", "Power". Why won't we take those attributes to determine the optimal combination? So if the model 1 is the best when the brand is "NaN", the meta will learn it and redirect every prediction having NaN brand to model 1.

So the meta estimator I propose is as follow :
$ (y_{pred1},y_{pred2},y_{pred3},$brandIsNull$)\rightarrow y_{pred-stack}$

Does this approach exist?
If not, would it be a good or bad idea?

The rationale for stacking learners is to combine the strengths of the individual learners.

On the one hand your idea makes sense: if learners have different strengths, adding some features might help the meta-model detect when to give more importance to a particular learner. It's totally possible that overall the meta-model will work better this way in some cases.
On the other hand, generalizing this idea would often defeat the purpose of stacking:

if we know in advance that a particular model is particularly good in some specific identified cases, then it's likely optimal to switch entirely to this model in such cases (no stacking).
if we don't have any particular knowledge, then this would require to add many/all of the original features to the meta-model, just in case they help. But by adding features it's likely that (1) the model will try to use the features directly, hence collapsing into an individual learner itself, and (2) the model will be more complex, therefore more prone to overfitting and likely not to make the best use of the individual learners' output.

To sum up: the stacking approach relies on "making things simple" for the meta-model, so that it only has to make a call about the answers of the individual learners. This way the meta-model can "focus" solely on optimally use these answers. The more features we add to it, the more risk that it will not to its job correctly.


Has anyone tried to use the hierarchy of ImageNet? The classes of ImagNet have a hierarchy. Did anybody try to build a hierarchy of classifiers to use this fact?

Searching for "multistage classification" leads to different results.

Yes, multiple papers have used this. I've heard of multiple ways to exploit this hierarchial structure. This paper Hierarchical Deep Convolutional Neural Network for Large Scale
Visual Recognition uses multiple levels by predicting the more coarse distribution and I think it then passes this as features to the more low level classification. YOLO9000 actually uses the ImageNet hierarchy for the problem of object detection. Hierarchial classification is the name of the problem you are describing.


Action Recognition for multiple objects and localization I want to ask questions regarding the action detection on the video with proposed frames. I've used Temporal 3D ConvNet for the action recognition on video. Successfully trained it and can recognize action on videos.
When I do inference, I just collect 20 frames from video, feed it to the model and it gives me the result. The point is that events on different videos are not similar in size. Some of them cover 90% of the frame, but some May 10%. Let's take as an example that two objects collided and it can happen on a different scale, and I want to detect this action.
How provide to the model exact position for the action recognition, if it can happen on a different scale with different objects? What comes to mind is to use Yolo to collect Regions of Interest and feed collected frames every time the 3D convnet. But if there are a lot of objects, the speed will be very slow. How to handle it?
Are there any end-to-end solutions for the action recognition with the object location proposal for the action recognition network?
I've already looked at papers and blogs, what people suggest, couldn't find the solution for the localization issues, so the action recognition model got the correct frames.

So finding actions from videos happens to be a tricky task. I have no idea about temporal 3D convnet but in order to tackle a problem like this, I would couple the CNN layer on individual frames of video and then feed the frame timeline into another layer of LSTM in order to find the context of the video.

As the action being performed on the video covers anywhere from 10% to 90% of the frame, you can perform TestTimeAugmentation on the video in order to find the action with a higher confidence rate. Similar approach could be found in this video by Google.


How to write a generator for keras fit_generator? This question is a further step of this question.

My data inputs are tens of .csv files, I have already read csv input data until the following format:

# train_x is data, train_y is label
print(train_x.shape)       # (2000000,10,100)  3D array
print(train_y.shape)       # (2000000,)    labels

I already can fit & evaluate them using:

model.fit(train_x, train_y, batch_size=32, epochs=10)
model.evaluate(train_x, train_y)

It works well if the dataset is LESS than RAM size. But if dataset is too BIG then "large dataset do not fit the memory". Most online suggestions are to use fit_generator( ) instead of fit( ) (also suggested from keras website).

fit_generator(generator, steps_per_epoch=None, epochs=1, verbose=1, callbacks=None, validation_data=None, validation_steps=None, validation_freq=1, class_weight=None, max_queue_size=10, workers=1, use_multiprocessing=False, shuffle=True, initial_epoch=0)

How to write a generator function (the 1st parameter of fit_generator)?

I only know the generator function aims to feed data batch by batch.

  As the name suggests, the .fit_generator  function assumes there is an underlying function that is generating the data for it.

What should be included in this geneator function? What should be returned? Any related example?

Mark: I have read several online examples (e.g., this and this). They use images as example, which is not my case (csv data only), and not easy to understand.

Ill supply two tutorials I used when I first started using fit_generator. That being said the first thing to remember is that a generator is essentially like any other function your write that returns something with the exception is that the function runs a continuous loop that is designed not to exit. For example, in a normal function, you would use return to return some chunk of data every time that function is called. In a generator, the function returns data in chunks continuously until there is no data to return. This is what the yield statement does. On my project, I have a couple of terabytes of signal data. Even on a powerful server that is a little bit too much data for hardware I use. So the generator function serves up chunks of data in batches, it can be run in parallel as well to increase speed. This approach also greatly reduces your memory usage and usually, your system's memory will hit some point and not really change until all data has been fed to the DNN. I apologize for the poor explanation but I am pretty new to this as well :-) I am sure someone else will provide a better answer but this should get you started.

Here is one good tutorial https://www.pyimagesearch.com/2018/12/24/how-to-use-keras-fit-and-fit_generator-a-hands-on-tutorial/

and a very short one https://medium.com/@fromtheast/implement-fit-generator-in-keras-61aa2786ce98

also, look at the docs
https://keras.io/models/sequential/

and finally here is one I use on a current project. It is not very pretty but it works well

def batch_generator(X_train, Y_train):

    while True:
         # samples_counter = 0
        for fl, lb in zip(X_train, Y_train):
            sam, lam = get_IQsamples(fl, lb)
            max_iter = sam.shape[0]
            sample = []  # store all the generated data batches
            label = [] # store all the generated label batches
            i = 0
            for d, l in zip(sam, lam):
                sample.append(d)
                label.append(l)
                i += 1
                if i == max_iter:
                    break
            sample = np.asarray(sample)
            label = np.asarray(label)
            yield sample, label


Class asks me to give self for Naive Bayes Model python I try to use the following code but when I try to use fit function with my X_train and y_train,
I get the following error:

fit() missing 1 required positional argument: 'self'

I do not know much about classes but I know it should not ask for self. I found somethings about instantiation but could not figure it out.

class BernoulliNB(object):
    def __init__(self, alpha=1.0):
        self.alpha = alpha

    def fit(self, X, y):
        count_sample = X.shape[0]
        # group by class
        separated = [[x for x, t in zip(X, y) if t == c] for c in np.unique(y)]
        # class prior
        self.class_log_prior_ = [np.log(len(i) / count_sample) for i in separated]
        # count of each word
        count = np.array([np.array(i).sum(axis=0) for i in separated]) + self.alpha

        smoothing = 2 * self.alpha
        # number of documents in each class + smoothing
        n_doc = np.array([len(i) + smoothing for i in separated])
        print(n_doc)

    def predict(self, X):
        return np.argmax(self.predict_log_proba(X), axis=1)

when I try

b = BernoulliNB()
b.fit(b, X_train,y_train)

this time, I receive

fit() takes 3 positional arguments but 4 were given

Then I changed it to

BernoulliNB().fit(X_train,y_train)

but this time, this error occurs:

The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

b = BernoulliNB()
b.fit(b, X_train,y_train)

This doesn't work because you're calling the fit method of the BernoulliNB class object you instantiated with BernoulliNB(). This argument says you have an extra parameter because it only accepts X, y.

It's hard to diagnose your second issue without seeing your data. But your y_train is probably formatted incorrectly.


Time series plot using netcdf data in python I am trying to plot a time series for a 3d plot. I know that the variable must be made 1d inorder to make it viable for the time series plot. Actually I want the data within a box of latitude and longitude. So if anyone can help me in this regard of how to plot it will be much appreciated.

You may want to try these out and see if they work for your use-case.
Plotting geospatial data in Python:
https://towardsdatascience.com/geopandas-101-plot-any-data-with-a-latitude-and-longitude-on-a-map-98e01944b972
Using matplotlib for simple animations:
https://www.geeksforgeeks.org/using-matplotlib-for-animations/


Is it a acceptable way to write a loss function in this form? I found a loss function of a perceptron on a book is in this form

$$ L(w,b) = - \sum\limits_{x_i \in M}y_i(wx_i+b) $$

In the context of machine learning loss functions, is it acceptable to put $x_i$ on the bottom of a summation notation?

It just means to sum over all $x_i$ in $M$. That is completely acceptable notation.


Hill Climbing Algorithm - Optimum Step Size I am implementing a standard hill climbing algorithm to optimise hyper-parameters for a predictive model. The hill climbing algorithm is being applied as part of a two-stage approach:

Apply grid search with large values applied to the hyper-parameter to find a 'best' starting point
Apply hill climbing algorithm in this space with a large number of different, random start points to find a local optimum

The large values that are passed in the first step, the grid search are

1*10^seq(-4, 5, by=1)
 [1] 1e-04 1e-03 1e-02 1e-01 1e+00 1e+01 1e+02 1e+03 1e+04 1e+05

So I am struggling to choose an optimum step size that isn't too large that is skips the peak or too small that it takes too long to converge. I don't think a single value for the step size is appropriate for all values passed in the grid search since, for example the difference between 1e-04 and 1e-03 is vastly different to 1e03 and 1e04. So I want the step size to be proportionate to the grid search start point. I know the search space I'm looking at is

grid_search_optimum/10 to grid_search_optimum*10

My question, therefore, is what is an accepted value for the step size in relation to the search space? I haven't found any opinions in literature around this and, in general, the only advice is to choose a step size that is "sufficiently small". Any advice or pointers to relevant papers would be greatly appreciated!

Usually it worked for me that if the search space was know then annealing rate (divide the size size with number of iteration)helped to decrease/increase the step size gradually to get to local max/min but the draw back is it might get stuck in local and might need some "momentum" to go on, another draw back it it might be very slow.however it doesn't seems like the case with your question that you bother about it.

Please refer-
https://courses.cs.washington.edu/courses/csep573/11wi/lectures/04-lsearch.pdf

This paper proposes self adaptive step size search-
https://link.springer.com/chapter/10.1007%2F3-540-34783-6_56


Binary Classification without machine learning tools I have to write a binary classifier for my company that should be as simple as possible and doesn't use machine learning libraries (and I also should not code too sophisticated algorithms by myself).

I just came up with the following idea: If I used a boxplot of my features and if the feature value of a new sample is bigger than the whiskers of the boxplot, I say the sample is in category A and if it is smaller than the whisker, it is in category B.
Is this a good approach? (probably not) Are there classification algorithms that are simple enough so that it is feasible to code them without libraries?

How about implementing one of the most basic binary classifiers by hand without any libraries, Logistic Regression?
Here is the explanation of the Logistic Regression in subtle terms:

https://www.youtube.com/watch?v=6tByJTacCOc&index=6&list=PLBAGcD3siRDguyYYzhVwZ3tLvOyyG5k6K

Here is the description of writing a simplified version of a binary classifier in Python, which will take a few buckets of lines in the vectorized form:

https://www.youtube.com/watch?v=ERIFzbTdIm0&index=14&list=PLBAGcD3siRDguyYYzhVwZ3tLvOyyG5k6K

If you feel uncomfortable with the implementation or confused about something, I'm sure that if you watch the videos in between the two that I provide, you will be able to easily write the algorithm that the video describes. I hope I could help; if you are not happy with the offered solution, I will be around for alternatives. Have fun.


Pandas merge column duplicate and sum value How to merge duplicate column and sum their value?

What I have

A   30
A   40
B   50

What I need

A   70
B   50

DF for this example

d = {'address': ["A", "A", "B"], 'balances': [30, 40, 50]}
df = pd.DataFrame(data=d)
df

You may use

df2 = df.groupby(['address']).sum()

or

df2 = df.groupby(['address']).agg('sum')

If there are columns other than balances that you want to peak only the first or max value,  or do mean instead of sum, you can go as follows:

d = {'address': ["A", "A", "B"], 'balances': [30, 40, 50], 'sessions': [2, 3, 4]}
df = pd.DataFrame(data=d)
df2 = df.groupby(['address']).agg({'balances': 'sum', 'sessions': 'mean'})

That outputs

         balances   sessions
address
A              70       2.5
B              50       4.0

You may add as_index = False to groupby arguments to have:

  address  balances  sessions
0       A        70       2.5
1       B        50       4.0


is the logit transform ever actually computed in modeling process of logistic regression? i've been tying to wrap my head around logistic regression, the logit transform, and the sigmoid function.

logit transform:

from what i understand, in practice all we want to do is maximize the following log likelihood (to get the sigmoid parameters that best fit the data):

which involves some form of gradient descent to find the weights that maximize the above expression.

what i'm a bit hung up on is i don't see the logit transform anywhere here, ie, it looks like it isn't even used in the modeling process.

so is the logit transform actually ever computed in logistic regression? or is it just used to set up the problem, as a rationale for maximizing the log likelihood (rather than computing linear least squares)?

its a bit similar to the question below, i understand that the logit is the inverse of the sigmoid, and in that sense it is "used" in the modeling process, but i don't see the log odds anywhere in the modeling process. thanks,

What is the purpose of Logit function? At what stage of model building process this logit function is used?

Basically the Logit function is a series of transformations from ordinary regression. Transformations from ordinary regression are often called "link functions" It goes:

Ordinary Least Squares (OLS) -> Logistic Function -> Logit Function

You probably already know these equations but I am going to write them out anyway:

OLS gives you a single predicted value

OLS = β_1 X_1+β_2 X_2+⋯β_i X_i

The Logistic Function gives you the probability that Y = 1

Logistic Function = 1/(1+exp[-1(OLS)])

The Logit Function gives you the log odds that Y = 1.

Logit Function = log_e(Logistic Function / (1-Logistic Function))

You are asking whether or not the logit function is ever actually used and the answer is yes, but you probably don't see it because a model summary will display the coefficients (betas) but not the equation that they're in. Your preferred language will certainly use these equations when estimating your coefficients, but there is no reason to show them to the user because the user is the one who specified the link function in the first place.

glm(y ~ x1 + x2, data = my.df, family = binomial(link = "logit"))


Is there a difference between tf.nn.conv1d and tf.nn.convolution in Tensorflow? I want to know what is the difference between a these two. For me, they are the same function, so I do not see the reason of existance two same functions.

In TensorFlow 2, they accept different arguments:

  tf.nn.convolution(input,...)

  Computes sums of N-D convolutions (actually cross-correlation).

  input: An (N+2)-D Tensor of type T, of shape [batch_size] + input_spatial_shape + [in_channels]

While the second one is more specific:

  tf.nn.conv1d(input,...)

  Computes a 1-D convolution given 3-D input and filter tensors.

  input: A 3D Tensor.input: A 3D Tensor.


Why does my linear regression model converge to a non-zero gradient value? I have a basic 2D Linear Regression model coded out (using gradient descent), yet it doesn't seem to work as well as it should.

What I expect is that m and c should approach 4 and 3 respectively, and m's slope or c's slope should tend to 0; yet what is actually happening is that c's slope approaches a non-zero value, and c itself approaches a value depending on the epoch (around 0.5 with an epoch of 100.)

If I look at the graph of c, it very slowly tends up over time, though.

Code here:

import random, math
import matplotlib.pyplot as plt

def linreg(x, y):
    """ Performs linear regression: input x, output y. """
    n = float(len(x))
    m = random.random()
    c = random.random()
    dm, dc = [], []
    rate = 0.00001
    epoch = 100
    for run in range(epoch):
        d_m = 0
        d_c = 0
        for i in range(len(x)):
            d_m += (y[i] - m*x[i] - c)*x[i]
            d_c += (y[i] - m*x[i] - c)
        d_m *= -2/n
        d_c *= -2/n
        m -= d_m * rate
        c -= d_c * rate
        dm.append(d_m)
        dc.append(d_c)
    return m, c, dm, dc

x = [i for i in range(400)]
y = [4*i + 3 for i in x]

m, c, dm, dc = linreg(x, y)

print(m, c)

plt.grid()
plt.scatter(x, y)
plt.plot(x, [m*i + c for i in x], color='red')
plt.show()

plt.grid()
plt.plot([i for i in range(len(dm))], dm)
plt.plot([i for i in range(len(dc))], dc, color='red')
plt.show()

those x values are way larger than it should. If you normalize your x and increase the learning rate, it will converge easily. General assumption is that the input comes from a normal distribution so we transform the input.

Here is the code(notice that I only subtracted the mean from x and divided it by its standard deviation and I changed the rate to 0.05):

import random, math
import matplotlib.pyplot as plt
import numpy as np

def linreg(x, y):
    """ Performs linear regression: input x, output y. """
    n = float(len(x))
    m = random.random()
    c = random.random()
    dm, dc = [], []
    rate = 0.05
    epoch = 100
    for run in range(epoch):
        d_m = 0
        d_c = 0
        for i in range(len(x)):
            d_m += (y[i] - m*x[i] - c)*x[i]
            d_c += (y[i] - m*x[i] - c)
        d_m *= -2/n
        d_c *= -2/n
        m -= d_m * rate
        c -= d_c * rate
        dm.append(d_m)
        dc.append(d_c)
    return m, c, dm, dc

x = [i for i in range(400)]
x_mean = np.mean(x)
x_std = np.std(x)
x = [(i - x_mean) / x_std for i in x]
y = [4*i + 3 for i in x]

m, c, dm, dc = linreg(x, y)

print(m, c)

plt.grid()
plt.scatter(x, y)
plt.plot(x, [m*i + c for i in x], color='red')
plt.show()

plt.grid()
plt.plot([i for i in range(len(dm))], dm)
plt.plot([i for i in range(len(dc))], dc, color='red')
plt.show()

the output in my case:

3.999898446167822 2.999938454995635


Network analysis classic datasets There are several classic datasets for machine learning classification/regression tasks. The most popular are:

Iris Flower Data Set;
Titanic Data Set;
Motor Trend Cars;
etc.

But does anyone know similar datasets for networks analysis / graph theory? More concrete - I'm looking for Gold standard datasets for comparing/evaluating/learning:

centrality measures;
network clustering algorithms.

I don't need a huge list of publicly available networks/graphs, but a couple of actually must-know datasets.

EDIT:

It's quite difficult to provide exact features for "gold standard dataset", but here are some thoughts. I think, real classic dataset should satisfy these criteria:

Multiple references in articles and textbooks;
Inclusion in well-known network analysis software packages;
Sufficient time of existence;
Usage in a number of courses on graph analysis.

Concerning my field of interest, I also need labeled classes for vertices and/or precomputed (or predefined) "authority scores" (i.e. centrality estimates). After asking this question I continued searching, and here are some suitable examples:

Zachary's Karate Club: introduced in 1977, cited more than 1.5k times (according to Google Scholar), vertexes have attribute Faction (which can be used for clustering).
Erdos Collaboration Network: unfortunately, I haven't find this network in form of data-file, but it's rather famous, and if someone will enrich network with  mathematicians' specialisations data, it also could be used for testing clustering algorithms.

What you are looking for can be found in KONECT (the website is down as I'm writing this but it should be fixed soon!). It's almost the most comprehensive data collection for network analysis. But the question is which one is more standard to use?

Well, there is no clear answer except of Zachary's Karate Club!

If you do a literature review in Community Detection algorithms you'll see that almost all shining papers use different networks. My suggestion is going through what Andrea Lancichinetti and Santo Fortunato did for benchmarking graphs. They proposed some benchmark graph generation algorithms e.g. this one.

Hope it helps :)


Training models from sklearn using tf.distribute.MirroredStrategy I want to distribute the training of a simple model, such as a support vector classifier like sklearn.svm.SVC() across some or all CPUs and GPUs on a single device. I have never utilized a GPU before and I'm confused as to how this works or if using tensorflow is even the right choice for this simple task. What i think I need to do is something like this:

import tensorflow as tf
from sklearn.svm import SVC
from sklearn import datasets
from sklearn.model_selection import train_test_split

strategy = tf.distribute.MirroredStrategy()

with strategy.scope():

    iris = datasets.load_iris()
    X = iris.data
    y = iris.target
    class_names = iris.target_names

    X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

    classifier = svm.SVC(kernel='linear', C=0.01).fit(X_train, y_train)

The actual dataset I'm using is $\approx 3 \times 10^8$ training examples with 11 features. Does this code do what I think it does? If not, what would be the best way to go about this task? If so, is there anything that can be improved?

EDIT:

After doing some more googling I discovered that sklearn does not support GPU utilization. See this post:
https://stackoverflow.com/questions/41567895/will-scikit-learn-utilize-gpu

I'm still not sure how I can go about utilizing a GPU for simple ML models.

There are a few approaches that allow you to do basic ML modelling using a GPU.

First of all, in the code as you presented it, the tensorflow MirroredStrategy unfortunately has no effect. It will only work with tensorflow models themselves, not those from sklearn. In fact, sklearn does not offer any GPU support at all.

1. CUML

An Nvidia library that provides some basic ML model types and other things, often offering the exact same API (classes and functions) as SciKit-Learn. Coming from Nvidia, this of course means everything is built with the GPU in mind.

This is part of the larger RAPIDS toolset (incubated by Nvidia). Maybe there are other tools there that can be helpful, like their XGBoost library.

2. Tensorflow / PyTorch + NumPy

These framework are not just for complicated Deep Learning, you can really use them to perform any basic modelling and leverage their GPU support. Their documentation contains examples, otherwise something like Hands On Machine Learning (a book with an accompanying set of Jupyter notebooks) is a nice way to dig in.

These frameworks work well with the normal scientific stack in Python (such as NumPy, Scipy, Pandas) because numpy arrays and the frameworks' Tensor objects are plug-and-play for most cases.

3. Another option:

Stick to sklearn while you are learning about the models, how they work and so on. If you want to just do anything with the goal of learning about about GPU usage, the two options above are the most modern ways to get started.


What are the inputs to the first decoder layer in a Transformer model during the training phase? I am trying to wrap my head around how the Transformer architecture works. I think I have a decent top-level understanding of the encoder part, sort of how the Key, Query, and Value tensors work in the MultiHead attention layers. What I am struggling with is the decoder part, specifically the inputs to the very first decoder layer.
I understand that there are two things. The output of the final encoder layer, but before that an embedded (positional encoding + embedding) version of... well something.
In the original paper in Figure 1, they mention that the first decoder layer input is the Outputs (shifted right). I am a little confused on what they mean by "shifted right", but if I had to guess I would say the following is happening
Input: <Start> How are you <EOS>
Output: <Start> I am fine <EOS>
and so the input to the first decoder layer will be [<Start> I am fine].
What is the need for shifting the sequence? Why would we not just input the target sequence itself? I was thinking maybe it's because of the auto-regressive nature of the decoder part, but then the only difference between the sequences would be the <EOS> token if I am seeing this correctly.
As you can probably tell I am a little bit confused by how some of the parts work, so any help to get to a better understanding would be much appreciated.

Following your example:

The source sequence would be How are you <EOS>
The input to the encoder would be How are you <EOS>. Note that there is no <start> token here.
The target sequence would be I am fine <EOS> . The output of the decoder will be compared against this in the training.
The input to the decoder would be <start> I am fine <EOS>.

Notice that the input to the decoder is the target sequence shifted one position to the right by the token that signals the beginning of the sentence. The logic of this is that the output at each position should receive the previous tokens (and not the token at the same position, of course), which is achieved with this shift together with the self-attention mask.


After training and saving a model can we give more information as input? Suppose my data is a time series with multiple features such as wind, temperature, holidays, etc.. and I'm predicting a target variable Y.
After I go through the whole process of splitting data into training and test and after having good predictions/forecasts I save my model.
Can I now give extra information as input to my model? If I want to predict tomorrow's Y values and I know the wind and temperature forecasts (from weather services) for tomorrow, can't I use them to get a better prediction for tomorrow?

Yes, by customizing the model with new information. But, you have to run atleast one training, with complete cycle and export output.
the general example from tensorflow is here


RandomForest - Reasons for memory usage / consumption? Which factors influence the memory consumption?

Is it the number of trees (n_estimators) or rather the number of data records of the training data or something other?

Both the amount of data and the number of trees in your forest will take up memory storage.

For the amount of data you have, this is obvious why it takes up storage - it’s data, so the more of it you have, the more space it takes up in memory. If your dataset is too large you may not even be able to read it all into memory - it may need to stay on the disk for training (I don’t think scikit supports this).

For the number of trees, each tree that is grown is trained on a subset of random features (so that the forest can avoid variance and avoid all the trees growing the same way). Once the forest is trained, the model object will have information on each individual tree - it’s decision path, split nodes, thresholds etc. This is information, and the more information you have, the more storage it will need. Therefore more trees = more information = more memory usage.

There are other factors that determine how much memory the model object (the trained forest) will take up; for example, max_depth which sets the maximum number of layers / levels any tree can grow to. If this is large, trees can grow deeper and therefore the model object will be bigger and therefore will need more storage. This shouldn’t be too large though, to avoid overfitting.


scale_pos_weight Xgboost My question is rather simple what does the parameter scale_pos_weight in xgboost do? I know typically it should be $\frac{sum(negative cases)}{sum(positive cases)}$.
Does it oversample the minority class by that ratio or does it undersample the majority class by inverse of that ratio? Or something else?
Also I would like to know if during cross validation in xgbcv, does the sampling happen on the test part of the cross-validation also or only the train part is affected by scale_pos_weight? Because i've heard that sampling should never be applied to test as it gives over-optimistic results.

I guess while computing total error from errors of individual samples, if sample happens to be of a positive class, its error is multiplied by the scale_pos_weight factor. Therefore setting high scale_pos_weight urges optimizer to treat more important (mostly due to its rareness) class with more respect due its higher contribution to the total error value.


How to cluster multiple time-series from one data frame I have a data.frame which has multiple time series in it, in the following manner:

01.01.16 Value1-Time-Series-1 Value2-Time-Series-1 Value3-Time-Series-1 ID-1
01.02.16 Value1-Time-Series-1 Value2-Time-Series-1 Value3-Time-Series-1 ID-1
01.03.16 Value1-Time-Series-1 Value2-Time-Series-1 Value3-Time-Series-1 ID-1
01.01.16 Value1-Time-Series-2 Value2-Time-Series-2 Value3-Time-Series-2 ID-2
01.02.16 Value1-Time-Series-2 Value2-Time-Series-2 Value3-Time-Series-2 ID-2
01.03.16 Value1-Time-Series-2 Value2-Time-Series-2 Value3-Time-Series-2 ID-2

Meaning:

There is one timeframe which is the same for every time series
There are multiple (thousands) of time series within that time
All are made up by the same type of values (columns)

I want to cluster this dataset in order to find the time series objects similar to each other.
I want to find those time series(s) which changed the same with respect to the values mentioned. I'm obviously interested in the measured distance. But using a simple hierarchical clustering algorithm is not cutting it since I need to tell the (whatever clustering algorithm I end up using) what is a time series and what not.

Any ideas how to this?

I'm using R for this. If there might be a better option with Python, I'm totally open to use it.

Thanks for any kind of help!

For an applied solution to your problem, I highly recommend reading the following: TSclust: An R Package for Time Series Clustering which can be found here. It presents the TSclust package in R and provides code.

Note that time series data is special, and cannot be treated like other data. So typical clustering techniques are not appropriate.


Solution is to transform the matrix:

JavaRDD<Vector> rows = jsc.parallelize(Arrays.asList(
    new DenseVector(new double[]{2, 2}),
    new DenseVector(new double[]{0, 1}),
    new DenseVector(new double[]{1, 1}),
    new DenseVector(new double[]{1, 0}),
    new DenseVector(new double[]{0, 1}),
    new DenseVector(new double[]{2, 1}),
    new DenseVector(new double[]{1, 1}),
    new DenseVector(new double[]{1, 1})
));

RowMatrix mat = new RowMatrix(rows.rdd());
List<Vector> sims = mat.columnSimilarities().toRowMatrix().rows().toJavaRDD().collect();
for(Vector v: sims) {
  System.out.println(v); //(2,[1],[0.8215838362577492])
}


When do we scale features and should it be done to label encoded features? So, I am trying to make a linear regression model for predicting car prices for which I have the following data set:  Data Set

Since the unique values are a lot for the categorical features, I label encoded the categorical features and converted the data set like this: Modified Data Set

Thereafter, I scaled the continuous features ( Engine, Power) like so:

# Scaling continuous features (No scaling for categorical variables)

X_train[['Engine', 'Power']] = scale.fit_transform(X_train[['Engine', 'Power']])

est = sm.OLS(y_train,X_train).fit()
print(est.summary())

X_test[['Engine', 'Power']] = scale.transform(X_test[['Engine', 'Power']])
y_predicted = est.predict(X_test)
y_predicted.sort_values()

# R2 value for y_test and y_predictions

from sklearn.metrics import r2_score

r2 = r2_score(y_test, y_predicted)
print(r2)

and got an R2 value of 84.8% compared to 33.5% if I also scaled label encoded features. I want to know why this happened? Also, my predicted values using this model on testing data is resulting in negative prices as well and most of these prices are not predicted well with the R2 score of predicted and true data being 68%.

You should not use Label Encoding for Categorical data unless there is a known ranking and that also in the specified ratio between the level values.
In this case, the model will assume 10 as 2 times of 5.

One-hot will add a lot of dimensions as I can see in your data.

You must try other Categorical encoding techniques esp. Sum Coding Or Helmert.

You should also try Dimension reduction techniques post-One-Hot to see if it reduces the dimensions.

Please do read these references -
Beyond-one-hot
Library
Kaggle tutorial
Feature Engg


How to install Polynote on Windows? I've been searching around the Internet for a while but I have not been able to find detailed instructions on how to install Polynote (the polyglot notebook
with first-class Scala support) for Windows with mixing multiple languages, Python and Scala.

Github Link for Polynote.
Official Website.

According to the official website:

  Polynote is currently only tested on Linux and MacOS, using the Chrome browser as a client. We hope to be testing other platforms and browsers soon. Feel free to try it on your platform, and be sure to let us know about any issues you encounter by filing a bug report

I would really appreciate it if anyone here can share his/her method if he/she
had successfully installed Polynote on Windows, either from Virtual Machines
(VMware/Virtualbox) or directly.

Slightly different from VMware or Virtualbox, but Polynote in Docker is supported by the developers and they have created a number of ready-made Docker images - more info here, although the instructions are targeted at a Linux host. Below are the instructions modified for a Windows host:

Once you have Docker installed, make a folder somewhere on your host system, e.g. C:\polynote:

cd \
mkdir polynote

Then create a text file config.yml in that folder, with the contents:

listen:
  host: 0.0.0.0

Then, you can get the container running with a single command - although make sure you customise the paths to your config file and working folder:

docker run --rm -it -e PYSPARK_ALLOW_INSECURE_GATEWAY=1
        -p 127.0.0.1:8192:8192
        -p 127.0.0.1:4040-4050:4040-4050
        -v C:/polynote:/opt/config
        -v C:/path/to/your/working/folder:/opt/polynote/notebooks
        polynote/polynote:latest
        --config /opt/config/config.yml

This exposes some ports to the host machine, mounts the folder with your config file into the container, mounts your working folder in the container at /opt/polynote/notebooks, pulls the latest Polynote Docker image and tells Docker to use your custom config file.

Once the container is spun up, you can use your web browser to access Polynote at http://localhost:8192/.


Using Low Frequency Labels with High Frequency Features I am trying to build a model (most likely a regression or random forest regression) for quarterly financial data. My training data has a daily cadence, but I am not sure how to work with these to predict a quarterly dependent variable. Clearly I could summarize these by quarter (avg, mean, etc.) but that would result in few total observations, and it feels like that is throwing out some valuable variance to train on.
Thanks for the help!

One way is to transform the daily data into dynamic quarterly data, eg by averaging over a moving window (that represents a quarter) and similarly for other statistics.
These windows can be overlapping, eg by a factor $o$%, no need to be non-overlapping.
This way the data are transformed to quarter period statistics while not actually reducing amount of data (but simply transformed).
Then you feed your regression model these quarterly data instead of the daily data in order to get quarterly results.
It is clear that to get results over quarter periods one needs inputs that represent quarter periods in the first place. So this is a natural way to achieve it.


What is Quantum Convolutional Neural Network? I just came through the TensorFlow Quantum library they introduced Quantum Convolutional Neural Network.

What is Quantum Convolutional Neural Network?
What is the difference between CNN vs QCNN?

This answer might not make a lot of sense without a little background on quantum computing

A QCNN (https://arxiv.org/abs/1810.03787) is a type of quantum model that the authors in this paper use to model quantum data. At the core it is just a quantum circuit acting on a set of qubits in order to model quantum data that is on those qubits. The authors use it to predict a particular property of their quantum data. Here is a piece from the paper where they explain the architecture at a high level:

  The circuit's input is an unknown quantum state $\rho_{\text{in}}$. A convolution layer applies a single quasi-local unitary ($U_i$) in a translationally-invariant manner for finite depth. For pooling, a fraction
  of qubits are measured, and their outcomes determine unitary rotations ($V_j$) applied to nearby qubits. Hence, nonlinearities in QCNN arise from reducing the number of degrees of freedom. Convolution and pooling layers are performed until the system size is sufficiently small; then, a fully connected layer is applied as a unitary $F$ on the remaining qubits. Finally, the outcome of the circuit is obtained by measuring a fixed number of output qubits. As in the classical case, circuit structures (i.e. QCNN hyperparameters) such as the number of convolution and pooling layers are fixed, and the unitaries themselves are learned.

To contrast this a little bit with the classical CNN: The QCNN is best used when dealing with quantum data that you wouldn't normally be able to deal with (easily) classically. The authors made a very smart choice here in choosing to model data that could be prepared on a quantum computer, data which might be harder for a classical computer to prepare. If you have classical data (like MNIST or CIFAR) then the CNN is still your best bet over the QCNN for performance. If you have quantum data on a quantum computer the QCNN has a much stronger case.

If you've been using TensorFlow Quantum, one of the tutorials shows this exact performance comparison where a quantum computer modelling classical data has a really hard time keeping up with the scaling of the classical computer (https://www.tensorflow.org/quantum/tutorials/mnist).

We are hoping to use TensorFlow Quantum to better understand quantum data and it's not too likely any QCNNs are going to pop up in serious classical ML pipelines for NLP or Image classification at scale that are going to drastically improve performance..... yet :P


SageMaker DeepAR access data Can SageMaker DeepAR algorithm only deal with training data that resides in Amazon S3 buckets?

Yes, it appears to be so. Look here for more details.
Here is a small, but important, extract:

"Just like other Amazon SageMaker algorithms, it [deepAR] relies on Amazon
Simple Storage Service (Amazon S3) to store the training data and the
resulting model."

Hope this helps.


Active learning on image classification model Say we have a trained image classification model. Theoretically, is it possible to update the model with only a sample without retraining? If not, is there any kinds of active image classification DL model that can allow incremental update?

Yes, that is possible. In transfer learning you have at least three options to handle transferred weights:

Freeze transferred weights
Retrain transferred weights from scratch
Fine tune transferred weights (further training but not from scratch)

The 3rd one is what you intend to do. And of course different options can be applied to different layers. Also see this paper:

  The usual transfer learning approach is to train a base network and then copy its first n layers to
  the first n layers of a target network. The remaining layers of the target network are then randomly
  initialized and trained toward the target task. One can choose to backpropagate the errors from
  the new task into the base (copied) features to fine-tune them to the new task, or the transferred
  feature layers can be left frozen, meaning that they do not change during training on the new task.
  The choice of whether or not to fine-tune the first n layers of the target network depends on the
  size of the target dataset and the number of parameters in the first n layers. If the target dataset is
  small and the number of parameters is large, fine-tuning may result in overfitting, so the features
  are often left frozen. On the other hand, if the target dataset is large or the number of parameters is
  small, so that overfitting is not a problem, then the base features can be fine-tuned to the new task
  to improve performance. Of course, if the target dataset is very large, there would be little need to
  transfer because the lower level filters could just be learned from scratch on the target dataset.

I have marked the sentence addressing your question bold.


How can the Adaboost technique be called an ensemble learning technique? I have read that in ensemble learning we use the outputs of various classifiers to make the predictive modeling better but in Adaboost we just use one classifier and we make it a strong learner but how it is a part of ensemble learning.

AdaBoost or Adaptive Boost is a boosting ensemble model which works by learning from it's previous mistakes, ie: misclassified data points.

We specify the number of decision trees to be generated while training and during each training step, it calculates the following :

The weighted error rate of the trained decision tree
The decision tree's weight in the ensemble = learning rate * log((1-e)/e)
Update weights of wrongly classified points

This process repeats until all the trees are trained. In the end, AdaBoost makes all of them "vote" by adding up the weight (of each tree) multiplying by the prediction (of each tree).

Therefore, this becomes an ensemble of multiple decision trees.


How is Amazon using Deep Learning in the new Amazon Go? In the Amazon Go page they claim to use "deep learning".

Where is deep learning used in the "Go" service? Is it computer vision or inventory management?

" Our Just Walk Out technology automatically detects when products are taken from or returned to the shelves and keeps track of them in a virtual cart. "

Out of this sentence I understand that camera's will record everything you do. Based on these videos they will record the things you bought etc. To get the information out the video deep learning will be used.

A quick search on google about "video analysis deep neural network" will give you results like: https://arxiv.org/abs/1512.03131


Multiple binary dummy features Vs Multi-values single feature I found a script on Kaggle's titanic competion in wich the creator convert Multi-values single feature (namely Pclass = {1,2,3}) to 3 binary features.

What are the pro/cons of such a conversion? Does it really improve something, or is it worse? Or doesn't change anything?

Despite taking longer to compute, I also worry that it could introduce a bias in the model.

As @K3 suggested, as @Sean Owen told me the name (One Hot Encoding), I can provide at least some elements to answer myself.

As found on sklearn documentation :

  [...]integer representation can not be used directly with scikit-learn
  estimators, as these expect continuous input, and would interpret the
  categories as being ordered, which is often not desired.

It is implemented in sklearn as OneHotEncoder and in Pandas as get_dummies .


How to visualize mass spectrometer data with plotly? Mass spectrometers output analyzation data usually in form of a mass chromatogram. I need a visualization of the output data. The x-axis represents the retention time, the y-axis represents the signal intensity and the z-axis represents the mass frequency. I need the signal intensity vs. retention time (y vs. x) and the signal intensity vs. mass frequency (y vs. z). Consider this short introductional video about how the visualization should look like.

The data may be represented as profile or centroid data. But w.r.t. data representation this does not make any difference because the measurement samples are still discrete w.r.t. z-axis (mass frequency).

I'd like to use plotly Dash to visualize the output data in a dashboard. However I could not find a suitable 3D chart example. How do I get the visualization I am aiming for?

There is a Needle Plot example here (dash-bio.plotly.host) or here (dash-gallery.plotly.host) which is suitable to visualize the data in 2D as well as a 3D Stem Plot example to visualize the data in 3D.


Which model to chose based on learning curve I trained my model using different regression techniques, and I'm not sure which model to choose based on the learning curve.

1) Should I choose Lasso, since train and CV converge at the end
2) Should I  choose Gradient Boosting since train and CV converge in the middle, and achieves a lower RMSE score than Lasso

You should go for LightGBM which has the lowest training and cross validation RMSE. By the way, score is a bad name for RMSE because you would expect a higher score to mean that your model is better but here the opposite is true.


Public datasets that show "cyclical" behavior I am looking for any publicly available dataset that has a "cyclical" structure to it, in the sense that if I plot the data in a certain way, a loop becomes visible.  A good example of this would be the Lotka-Volterra predator-prey model, which has a very pronounced cycle, shown below.  Are there any other good examples that demonstrate this?

The search term you are looking for is limit cycle. You can find some examples (legged-locomotion, digital control systems, etc.) here: https://www.sciencedirect.com/topics/engineering/limit-cycle.


Finding out why your model is doing better? I fitted a logistic regression model on a data set and got an AUC score of .70.
I added some additional out-hot encoded categorical features to the model and the AUC improved slightly to .74.
How do I assess how the model improved? What plots/other analyses are used to to assess the performance gain?
I understand that the model improved, but I want to be able to explain why adding those features improved the model.
This is just a general data science question.
Thanks!

First, it makes sense that adding features will improve your performance, just make sure you do evaluation carefully and not overuse the same validation dataset (and if yes try to re evaluate it on an unseen independent different test set) to ensure you are not overfitted.
After that, you can use Shapley values in its aggregate look to see which features (or specific values in your encoding) impacted the model decision making


Is it possible to find a model that minimises both false positive and false negative? Is it possible to come up with a model that minimises both false positive and false negative?

Minimising can be done to a point, such as the Bayes error threshold.

Yes and No! depending on what do you mean by minimization. When you say minimizing $f$ and $g$ according to something, you are actually looking for a point which minimizes both. It does not mean that this point necessarily finds the minimum of $f$ or $g$. So yes in this sense.

But in case you mean a point in which both of them are in their minimum, this case is not realistic in practice as usually increasing one of them means decreasing the other one. Have a look at the plot bellow. FRR means False Rejection Rate, FAR means False Acceptance Rate and EER means Equal Error Rate.This is the terminology used in Biometric, a ML-based field studying the person identification/recognition.

It illustrates what I explained above.

Note

You should consider the importance of each error. In some use-cases (including some biometric use-cases) you may sacrifice one error for the other one. In those cases you minimize the error you need to be minimum without really caring about the other one (of course trying to keep the other one as low as possible as well but not at top priority).

For example I have a Face Recognition system which opens the door of a highly confidential military site to permitted users. Should I try to keep the EER low? Sure not. If sometimes the door is not open for a permitted person it's ok (at most they complain a bit) but door MUST NOT be opened for someone who is not allowed so here you care about keeping FAR low.

Hope it helped


Quickest way of multi-labelling images? I want to make a new dataset containg thousands of (different-sized) images. Now I need to assign multiple labels to each image.

Of course I already looked at github etc. and there are good labelling programs there. The problem is it seems like you can always only assign one label to an image.

Do you know a simple way of assigning multiple labels to images and saving them in a dedicated file?

Your question is relatively devoid of details, so all I can really suggest is to use a standard convolutional network classifier.

The problem you seem to be having is that most such models output a one-hot vector, i.e., categorical distribution, corresponding to a single label: $p=(p_1,\ldots,p_n)$, where $\sum_i p_i = 1$ and $p_i$ is the probability of assigning label $i$ (and only label $i$) to the input. Usually this is computed via $p = \text{SoftMax}(f_\theta(x))$ for some input image $x$ and neural network $f_\theta$.

To change this to handle multiple labels, let $q=(q_1,\ldots,q_m)$, where $q_i\in[0,1]$ is the probability of assigning label $i$ to the input. You can think of each $q_\ell$ as parameterizing a Bernoulli distribution over a single label. You can compute $q$ via, for instance, $q = \sigma_E(f_\theta(x))$, where $\sigma_E$ is the elementwise sigmoid function. You can then assign labels to $x$ by, for example, thresholding each $q_j$, i.e., assign label $k$ to $x$ if $x_k \geq T$, where $T$ is a hyper-parameter that can be, say, 0.5.

In other words, take the classifiers on Github and replace the softmax with an elementwise sigmoid, change $n$ to $m$, and use your training data; that's it.

Hopefully that's what you meant.


Multi task learning architecture for Multi-label classification I am working a classification problem. The dataset was collected from  Painters by number, a competition hosted by kaggle. The task is to identify painter,style and genre given paintings.

So far, I trained individual models to predict painter,style,genre given paintings. Now i would like to incorporate Multi task learning (i.e) Developing a single model which can predict all three tasks.

The issue i am currently facing is in designing architecture.

      Model                         No of classes (Softmax)
   (Individual Models)
   --------------------             -----------------------
 Model predicts painter                     8 (8 - painters)
 given paintings

 Model predicts style                       10 (10 style classes)
 given paintings

 Model predicts genre                       23 (23 genre classes)
 given paintings

I don't know how to combine the above models in keras. Any suggestions or feedback would be helpful.

You should design a multi-task model (MTM).
MTM has the ability to share learned representations from input between several tasks. More precisely, we try to simultaneously optimize a model with m types of loss function, one for each task. Consequently, MTM will learn more generic features, which should be used for several tasks, at its earlier layers. Then, subsequent layers, which become progressively more specific to the details of the desired task, can be divided into multiple branches, each for a specific task.

You need an architecture like the following:

# Your input and hidden layers
inp = Input(...)
x = Layer1(...)(inp)
x = Layer2(...)(x)
...
x = Layer_N-1(...)(x)

# Your 'm' output layers
out_1 = Layer_N(x)
out_2 = Layer_N(x)
...
out_m = Layer_N(x)

# your model
MTM = Model(inputs=inp, outputs=[out_1,out_2,...,out_m])
MTM.compile(loss=[loss_func_1, loss_func_2,..., loss_func_m],
      optimizer='xxx',
      metrics=xxx)

MTM.fit(train_data, [train_labels_1,train_labels_2,...,train_labels_m],
          batch_size = xxx,
          epochs = xxx)

If you want a real example in Keras, I have implemented a 2-task ConvNet for my project here.


I am not getting classification output by predict_generator() I am trying to classify pre-downloaded images from my dataset to "Rock" , "Paper" , "Scissor" classifications but I am getting outputs as numbers.

I have already divided the dataset to Train folder where 3 folders with the names of the categories exist.

Is it because I am not using a convolutional neural network?

I don't understand what to do here .

Code:

from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Dense, Flatten
import numpy as np

train_directory = 'D:\D_data\Rock_Paper_Scissors\Train'
training_datgagen = ImageDataGenerator(rescale = 1./255)
training_generator = training_datgagen.flow_from_directory(
    train_directory,
    target_size = (28,28),
    class_mode = 'categorical', classes = ["Rock", "Paper" , "Scissor"])

validation_directory = 'D:\D_data\Rock_Paper_Scissors\Train'
validation_datagen = ImageDataGenerator(rescale= 1./255)
validation_generator = validation_datagen.flow_from_directory(
    validation_directory,
    target_size = (28,28),
    class_mode = 'categorical',
    classes = ["Rock", "Paper" , "Scissor"]
    )

model = Sequential()
model.add(Flatten(input_shape = (28,28,3)))
model.add(Dense(128,activation = 'relu'))
model.add(Dense(64, activation = 'relu'))
model.add(Dense(16, activation = 'relu'))
model.add(Dense(3, activation = 'softmax'))

model.compile(optimizer = 'adam', loss = 'categorical_crossentropy',metrics = ['accuracy'],)
filenames = validation_generator.filenames
nb_samples = len(filenames)
desired_batch_size = 1

model.fit_generator(training_generator,epochs=20,validation_data = validation_generator)
predict = model.predict_generator(validation_generator, steps = np.ceil(nb_samples/desired_batch_size))

print(predict)

Output:

[[2.2792105e-01 7.7206087e-01 1.8017196e-05]
 [2.8221733e-03 9.9715865e-01 1.9176909e-05]
 [5.1788869e-04 9.9948126e-01 8.8551565e-07]
 ...
 [9.9283433e-01 7.1645100e-03 1.1474091e-06]
 [9.9999678e-01 3.2666808e-06 4.5816462e-09]
 [9.9640071e-01 3.5960779e-03 3.1497514e-06]]

m.predict_generator returns a numpy array. you can take the maximum probability of a class by numypy argmax function or use predict_classes. This link might help-

How to predict class label from class probability given by predict_generator for testdata?


What are the consequences of not freezing layers in transfer learning? I am trying to fine tune some code from a Kaggle kernel. The model uses pretrained VGG16 weights (via 'imagenet') for transfer learning. However, I notice there is no layer freezing of layers as is recommended in a keras blog. One approach would be to freeze the all of the VGG16 layers and use only the last 4 layers in the code during compilation, for example:

for layer in model.layers[:-5]:
    layer.trainable = False

Supposedly, this will use the imagenet weights for the top layers and train only the last 5 layers. What are the consequences of not freezing the VGG16 layers?

from keras.models import Sequential, Model, load_model
from keras import applications
from keras import optimizers
from keras.layers import Dropout, Flatten, Dense

img_rows, img_cols, img_channel = 224, 224, 3

base_model = applications.VGG16(weights='imagenet', include_top=False, input_shape=(img_rows, img_cols, img_channel))

add_model = Sequential()
add_model.add(Flatten(input_shape=base_model.output_shape[1:]))
add_model.add(Dense(256, activation='relu'))
add_model.add(Dense(1, activation='sigmoid'))

model = Model(inputs=base_model.input, outputs=add_model(base_model.output))
model.compile(loss='binary_crossentropy', optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),
              metrics=['accuracy'])

model.summary()

I think that the main consequences are the following:

Computation time: If you freeze all the layers but the last 5 ones, you only need to backpropagate the gradient and update the weights of the last 5 layers. In contrast to backpropagating and updating the weights all the layers of the network, this means a huge decrease in computation time. For this reason, if you unfreeze all the network, this will allow you to see the data fewer epochs than if you were to update only the last layers weights'.
Accuracy: Of course, by not updating the weights of most of the network your are only optimizing in a subset of the feature space. If your dataset is similar to any subset of the imagenet dataset, this should not matter a lot, but, if it is very different from imagenet, then freezing will mean a decrease in accuracy. If you have enough computation time, unfreezing everything will allow you to optimize in the whole feature space, allowing you to find better optima.

To wrap up, I think that the main point is to check if your images are comparable to the ones in imagenet. In this case, I would not unfreeze many layers. Otherwise, unfreeze everything but get ready to wait for a long training time.