Multitask learning NN only trains on a few tasks I am fairly new to the Machine Learning field, but I am working on building a multi-task network for a project I am working on. Unfortunately, despite all my best googling and digging through documentation, blogs and research papers, I am so far unable to get my network training on more than a few of the tasks at a time.

For context, the problem entails a set of images of metallic test samples, all taken under standard lighting and at a standard distance. For confidentiality reasons, I cannot share the dataset itself. The goal of the network is to be able to score 3 different surface defects in 3 distinct regions each, a total of 9 outputs, each with an integer rating score from 0 to 5 (6 possible outcomes for each output).

I have been writing the NN in Python using Keras and visualising the metrics and graph with TensorBoard.

The problem is that, as I attempt to train the network, it only (at best) seems to learn for up to 4 of the 9 tasks, even though each image has a rating for each task. To illustrate my point, here are two screenshots of the TensorBoard metrics for one of the more successful training runs along with the graph showing the network structure:

As you can see, the tasks WCP_3T and WCP_5T are training well, but nothing else really is (the high accuracy on the WCP_F task is due to heavily weighted data, and not much else I'm afraid). The plateau in accuracy for each of the other tasks seems to me to be the network quickly finding the rating that is most common for that task, and remaining there. Thus, the varying accuracies for each task is an artifact of the data for that category.

I have tried deepening (and shallowing) the other two main branches (BS and BD), I have experimented with altering learning rate, batch size, optimizer function (currently using Adam) and activation functions (currently using relu for the WCP branch and LeakyReLU for the other two, but I have also tried ordinary relu for them with no benefit). I have altered the number of filters in the various convolutional layers, and I have increased and decreased the dropout on various layers also. Still, nothing I seem to try can bring the network to train on all the tasks.

It is possible that the size of the dataset is a problem, but I only have a small number of images available and I don't think it is possible to get many more. I am currently using a training set of 506 images and a batch size of 32, which only leaves about 170 images for each testing and validation. Data augmentation to provide more images may also be out of the question as that could conceivably alter the scores placed on the different defects, and I would not know how (they were graded by experts, and the data was then passed to me for this task).

If anyone has had a similar problem before and could suggest something else I could try, or could at least point me towards somewhere that might help to solve my problem, I would be extremely grateful!

So, it turns out that my problem had a lot more to do with network structure than anything else. I was trying to share too many lower levels, and then still not leaving enough room for the later branches to learn individual features/locations. In the end, I re-structured my network so that there was only 1-2 shared convolutional layers, then each primary branch shared 1 further convolutional layer before again each branching into 3 sub-branches (one for each output), each with their own convolutional, dense, flatten and then softmax layers. I also possibly had too many filters on each convolutional layer.

This enabled 'training' across each of the 9 outputs, but has left me with some serious overfitting to try to overcome, due to the depth of the network and the small data set. I am currently trialing methods of correcting for this.


Why use SOM for clustering? Suppose I have a dataset with $M$ observations of dimension $N$.

I've seen many people use Kohonen networks (or self-organizing maps, SOMs) to do this task. In this approach, the neurons have weights of the same dimension of the observations, $N$. The idea is to train the network so that each node $w_i$ becomes a centroid of a cluster, and thus identify the clusters with the neurons of the network. As we don't know the number of clusters a priori, one should establish a large number of neurons, $W$ such that $W<< M$, and then identify the nodes that represent the centroids (checking which ones are located in more dense areas and other criteria similar to that one).

What I don't understand is why one would do this. I could just say that I have $M$ possible centroids (each datapoint) and then use the discriminating criteria to check which ones are the actual centroids. In this approach there are $M$ centroid candidates and in the SOM one, $W$. But in both cases the dimension is $N$. Why is then using a SOM favorable? I believe I would want to have more data when performing clustering, so I think that having $M$ samples instead of $W$ would be better. Also, the dimensionality of the problem stays the same, so no advantage there. On top of it, it would be much faster to do it without the SOM, as training it may take a while, but the output (i.e. the values of the neuron weights) would just be pretty similar to a subset of the original dataset, which could be obtained just by identifying possible centroids from the $M$ observations.

Why is it then that SOMs are used for clustering?

I disagree with the assertion that SOM are frequently used. IMHO they are difficult to use and rather obscure.

Nevertheless they have some interesting properties, although I'd rather call them a visualization than a clustering technique.

What you are ignoring is the"map" property. Neurons aren't independent but connected in a particular layout. Without this, the method is useless AFAICT. You need nodes to usually not be empty, but rather the average of many points, and thus M<


What's the intuition behind nonlinear predictors in multivariate regression? I'm learning statistical learning with the well known ISLR (Introduction to Statistical Learning with Applications in R) and doing the exercises, right now in the linear chapter regression. Despite being linear regression, non linear transformations are also mentioned because of the high bias issues.
It's quite straightforward to understand the meaning of a linear regression model with multiple predictors, e.g.:
Y = β0 + β1X1 + β2X2 + β3X3

So as X1 increases by one unit, predicted response Y will increase by β1 units (assuming X2 remains constant), same for the rest of predictors.
Also for regression with interaction terms:
Y = β0 + β1X1 + β2X2 + β3X1 X2

Which in this case indicates there's a synergy between X1 and X2 predictors, removing the additive assumption.
I'm also getting familiar with F-statistic, t-value or p-value for assessing the statistical significance of the predictors and decide whether or not there's an actual relationship with the predicted variable.
In the applied exercises section for this chapter, one of the exercises used with a dataset with several values, contains the following question:

Try a few different transformations of the variables, such as log(X), sqrt(X), X^2. Comment on your findings.

How should one try nonlinear transformation of variables following an intuition? It's easy to get a sense of this for simple regression with a single predictor (just by plotting the dataset), but what's the technique here without just randomly try stuff, considering that there are several predictors to try them all?
Moreover, given the following hypothetical model:
Y = β0 + β1X1 + β2X2 + β3X3 + β4X1^2 + β5log(X2)

That is, a predictor having both a linear and non linear relationship with the response? What can be a real-world example to understand this? As said, this would be very easy with a single predictor, but I don't find an intuition with multivariate regression.

Not quite sure if this is what your question actually is. However, when you have a data generating process such as $y = 3 + 2x$ (with some zero-mean "gaussian noise"), your model could look like:
$$ y = \beta_0 + \beta_1 x + u,$$
and you will find the $\beta_0 = 3$ and $\beta_1 = 2$.
When you have a data generating process such as $y=3+2x+3x^2$ you need to do a "basis expansion" in your model to capture the additional complexity of the function and you could write a model such as:
$$ y = \beta_0 + \beta_1 x + \beta_2 x^2+ u.$$
The second model will capture the non-linear (quadratic) DGP "perfectly".
df = data.frame(x1=seq(-10,20,by=0.2))

df$v = df$x1^2
df$err = rnorm(nrow(df), mean = 0, sd = 1)
df$y = df$v + df$err

plot(df$x1,df$y)

reg1 = lm(y~x1,data=df)
summary(reg1)
lines(df$x1,predict(reg1,newdata=df),col="red")

reg2 = lm(y~poly(x1,2,raw=T),data=df)
summary(reg2)
lines(df$x1,predict(reg2,newdata=df),col="blue")


What is the purpose of multiple neurons in a hidden layer? On the surface, this sounds like a pretty stupid question. However, i've spent the day poking around various sources and can't find an answer.

Let me make the question more clear.

Take this classic image:

Clearly, the input layer is a vector with 3 components. Each of the three components is propagated to the hidden layer. Each neuron, in the hidden layer, sees the same vector with 3 components -- all neurons see the same data.

So we are at the hidden layer now. From what I read, this layer is normally just ReLus or sigmoids.

Correct me if I'm wrong, but a ReLu is a ReLu. Why would you need 4 of the exact same function, all seeing the exact same data?

What makes the red neurons in the hidden layer different from each other? Are they supposed to be different? I haven't read anything about tuning or setting parameters or perturbing different neurons to have them be different. But if they aren't different...then what's the point?

Text under the image above says, "A neural network is really just a composition of perceptrons, connected in different ways." They all look connected in the exact same way to me.

To explain using the sample neural network you have provided:

Purpose of the multiple inputs: Each input represents a feature of the input dataset.
Purpose of the hidden layer: Each neuron learns a different set of weights to represent different functions over the input data.
Purpose of the output layer: Each neuron represents a given class of the output (label/predicted variable).

If you used only a single neuron and no hidden layer, this network would only be able to learn linear decision boundaries. To learn non-linear decision boundaries when classifying the output, multiple neurons are required. By learning different functions approximating the output dataset, the hidden layers are able to reduce the dimensionality of the data as well as identify mode complex representations of the input data. If they all learned the same weights, they would be redundant and not useful.

The way they will learn different "weights" and hence different functions when fed the same data, is that when backpropagation is used to train the network, the errors represented by the output are different for each neuron. These errors are worked backwards to the hidden layer and then to the input layer to determine the most optimum value of weights that would minimize these errors.

This is why when implementing backpropagation algorithm, one of the most important steps is to randomly initialize the weights before starting the learning. If this is not done, then you would observe a large no. of neurons learning the exact same weights and give sub-optimal results.

Edited to answer additional questions:

The only reason the neurons aren't redundant is because they've all been "trained" with different set of weights, hence, give a different output when presented with the same data. This is achieved by random initialization and back-propagation of errors.
The outputs from the Orange neurons (use your diagram as an example), are "squashed" by each Blue neuron by applying the sigmoid or Relu function with the trained weights and the output of the orange neurons.


Is time series forecasting possible with a transformer? For my bachelor project I've been tasked with making a transformer that can forecast time series data, specifically powergrid data. I need to take a univariate time series of length N, that can then predict another univariate time series M steps into the future.
I started out by following the "Attention is all you need" paper but since this paper is meant for NLP I had to make some changes. Instead of embedding each word to a random point in d_model-dimensional space I use a linear layer embed the data. I've also tried using a nn.Conv1d layer with a kernel size of 1 to embed, but these approaches fail to make a non-linear prediction of the data and instead only predict a straight line through the average of the data.
First I though that the problem was my implementation of the transformer, but even when I use Pytorch' build in nn.Transformer module I get the same results. I then tried different types of positional encoding like the "Time2Vec" paper that approximates the data by using different sinus functions.
I feel like I've tried a lot of different things to make this transformer work but to no avail. So my question is, do transformers alone work for multistep forecasting with univariate data. And if so are there any articles, papers, repositories etc. that forecasts time series data with succes? If not which approach should I take going forwards to see if I can get my transformer to work.

I'm unclear whether transformers are the best tool for time series forecasting. Transformers at the end of the day are just the latest in a series of sequence-to-sequence models with an encoder and decoder. This means that transformers change something to something else.
With time series you aren't changing something to something else, you're trying to find statistical patterns in variations across time e.g do these variations correlate with each other, do they follow trends, cycles, etc. Then you're trying to use these statistical patterns to forecast for future dates. Time series is a lot like regression in that sense, where transformers try to capture relationships between words, and not just in left to right direction (the positional encoding is just extra information for the network).


is it reasonable to consider Horse Colic dataset as a binary classification problem? I am learning Horse Colic dataset.

Well documented attributes; 368 instances with 28 attributes (continuous, discrete, and nominal); 30% missing values

there are 3 type of outcomes, 'lived', 'died', 'was euthanized'.

since 'was euthanized' is another form of 'died', is it reasonable to consider this problem as a binary classification problem?

It depends on your goal.

If you're interested in the horse's status then this is a multiclass classification problem.

Or if you're only concerned if a horse died and not concerned about how it died, then you can treat it as a binary problem (after grouping 'euthanised' with 'died').


ValueError: Tensor Tensor("activation_5/Softmax:0", shape=(?, 2), dtype=float32) is not an element of this graph There seem to be an issue with predicting using my keras model. I had trained it using the following keras code:

model = Sequential()
model.add(Conv2D(32, (3, 3), input_shape=(150, 150,3),padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='same'))

model.add(Conv2D(32, (3, 3),padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='same'))

model.add(Conv2D(64, (3, 3),padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='same'))

model.add(Flatten())  # this converts our 3D feature maps to 1D feature vectors
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(2))
model.add(Activation('softmax'))

model.compile(loss='binary_crossentropy',
          optimizer='rmsprop',
          metrics=['accuracy'])

However when i predict it on my local system after training with the shape (1,150,150,3) . It predicts accurately with an accuracy over 90%. however when i load my model on my raspberry pi and input the image of the same shape (1,150,150,3) it returns an error. Below is the code loaded on the raspberry pi to predict from the keras model.

    data = numpy.fromstring(stream.getvalue() , dtype = numpy.uint8)
    image5 = cv.imdecode(data , 1)
    print(image5.shape)
    #cv.imwrite('uhhu.png',image5)
    img = cv.resize(image5,(150,150))
    x = img_to_array(img)
    x = x.reshape((1,) + x.shape)
    x = x/255
    x = numpy.float32(x)
    print(x.shape)
    score = loaded_model.predict(x)
    print(score)

The solution to this issue is predict from the keras model when running a tensorflow graph as default.

import tensorflow as tf
graph = tf.get_default_graph()

global graph
with graph.as_default():
    result = loaded_model.predict(x)


Face dataset organized by folder I'm looking for a quite little/medium dataset (from 50MB to 500MB) that contains photos of famous people organized by folder.

The tree structure have to bee something like this:

├── bfegan
      └── ...
├── chris
      └── ...
├── dhawley
      └── ...
├── graeme
      └──...
├── heather
      └──...

Do you know some dataset that have this tree structure?

I want to understand how my very basic neural network is able to detect face in a 'real-world' situation.

https://github.com/alessiosavi/PyRecognizer

I've found a dataset the meet the specification here:

https://www.kaggle.com/atulanandjha/lfwpeople/data#lfw-funneled.tgz

Some people in this dataset have only very few image.

We can create a new one dataset using the following bash command, in order to exctract only the people that contains more than 5 images:

# Extract only the people that have more than 5 photos (-gt 5)
for i in $(ls); do a=$(ls $i |wc -l); if [ "$a" -gt 5 ]; then echo $i ; fi ; done > people_ok
# Create a directory for store the images
mkdir -p /tmp/faces
# Copy the filtered directory in the new one
for i in $(cat people_ok  | xargs echo -n) ; do cp -r $i /tmp/faces/ ; done


How to optimize input parameters given target and scoring parameters I'm new to machine learning/optimization, so I apologize in advance if this has been answered before. I don't know which search terms to use.

I have a large dataset where I have a number of input parameters $I_1$, ... $I_n$ ($n$ up to 10), a target parameter $T$ and a scoring parameter $S$. There are roughly a million rows of real-life data, where each row has the input parameters, the target parameter and the scoring parameter. The data is operational data from a piece of machinery where $T$ is output power, $S$ is fuel consumption and $I_1$, ... $I_n$ are tuning parameters for the machinery. The relationships of the tuning parameters to $S$ and $T$ are unknown but likely nonlinear. So I basically want to understand how to tune the machinery to produce the desired output power as fuel-efficiently as possible given the parameters I can tune.

What I need is some way of getting to a function where I input $T$ and get out the optimal combination of $I_1$, ... $I_n$ that minimizes $S$ for that value of $T$. I work in Python, and I assume that it's some combination of sklearn and scipy, but I haven't been able to figure out the steps to take for this type of problem. Thanks in advance.

If your data have simple linear relationship (or you can transform it such that this is the case) you can do regression on $T$ and $S$, and after that you should have the linear relationship between input parameters and target parameters. After that you can use scipy minimize or any other library to perform constrained optimization. If I follow from my example my idea would be to perform a predefined trade-off between $T$ and $S$ for example $\max 0.8 T + 0. 2S$. This should be easy to code with scipy.


why is more layers better in deep learning? i understand mathematically that deep learning has more than one hidden layer, whereas regular machine learning hs just one. is that right? if so, why and how is it better to have more than one layer, that give deep learning the edge over machine learning?

You can think of Neural Networks (however deep) as an approximation of an ideal function. The more layers/nodes are available, the more the Neural Network successfully approximate that ideal function. There is a theorem which states it.

Let's say you have to recognize human faces in pictures: there is at least one (ideal and purely hypothetical) mathematical function that is able to do the job. Than, a Neural Network can approximate that mathematical function in a more or less precise way.

Now: if you have the collection of every possible picture which can exist in the world since ever and forever (a so-called complete set), you could choose whatever number of layers and nodes, in whatever configuration, and you will never fall in an overfit situation. Instead, the more layers/nodes are available, the better your network will perform.
But in this limited real world, you can get just a tiny subset of that main set.

So you have to choose the optimal number of layers and nodes, given a configuration, to avoid to fall in the situation of overfit. The more data samples you have, the more you can add up layers and nodes to the configuration, with the result of having better performances, i.e. a Neural Network which better approximate the (ideal and purely hypothetical) mathematical function introduced above.


Can metadata be used to adapt parsing for an unescaped in field use of the delimiter? I have data coming from a source system that is pipe delimited. Pipe was selected over comma since it was believed no pipes appeared in field, while it was known that commas do occur. After ingesting this data into Hive however it has been discovered that rarely a field does in fact contain a pipe character.

Due to a constraint we are unable to regenerate from source to escape the delimiter or change delimiters in the usual way. However we have the metadata used to create the Hive table. Could we use knowledge of the fields around the problem field to reprocess the file on our side to escape it or to change the file delimiter prior to reloading the data into Hive?

So, a few of your rows will have too many columns by one or more as a result. That's easy to detect, but harder to infer where the error was -- which two columns are actually one? which delimiter is not a delimiter?

In some cases, you can use the metadata, because it helps you know when an interpretation of the columns can't be right. For example, if just the one column can have a text value, and all the others must be numeric, it's unambiguous where the error is. Any additional columns created by this error occur right after the text column.

If they're all text, this doesn't work of course.

You might be able to leverage more than the metadata's column type. For example you may know that some fields are from an enumerated set of values, and use that to determine when a column assignment is wrong.


Modelling regressor of historic data on basic features test set I am using a historical dataset of sales of items in shops and I need to predict the sales of the next month of the period.

I performed feature engineering, and now I have 10 features in the train set - median, avg of sales (prices and quantities).
I want to perform Linear Regression, The test set contains only 2 features - Shop_id and Item_id. How can I perform prediction from fitted regressor with 10 coefficients, and the test has only the 2?

I am not talking about doing PCA - the historic data has few historic sales  features and the test only has the basics

Dataset: Competitive data science predict future sales

Well, what you have here is a time series problem.

In a time series problem, you don't have the data parallel to the variable you are trying to predict. That means your data is over at the time $T$ and you have to predict based on the data from times $1$ to $T-1$.

Also, Shop_id and Item_id are not regressor which have a coefficient, because they are IDs, and using IDs to predict causes you model to learn-by-memory your data for the ID's, it gives a overfitted model.

Imagine the real-life situation: The store asks you to predict the sales of the next month, of course, you don't have data like the average price of the next month either, so you have to predict with historical data of both (all) variables.

The solution you are looking for is an ARIMAX model: The ARIMAX model allows you to use variables like Sales, Avg price, and others with information from the past.

$$Sales_t = \alpha_1 Sales_{t-1} + \alpha_2 Sales_{t-2}+\dots+\beta_1AvgPrice_{t-1}+\dots+\beta_mAvgPrice_{t-n}$$

The model is an example of how your predictions could be made, the use of the IDs can be "a model for every store". And every $Sales$ variable in the model could be a set of products.


Should you use the same algorithm in your feature selection as your model My question is should you use the same algorithm in feature selection as your model?
If I'm using a KNN model for classification should I also use a KNN algo when running feature selection? Or lightGBM algo for feature selection and lightGBM model?
Or is there a best algo for feature selection that you can use those features with any model?
I know there are several different feature engineering techniques I just don't know which techniques should be used with different machine learning algorithms.

Yes - You can use the same the same algorithm for both feature selection and prediction. The most common examples are L1 regression and tree-based algorithms. Those algorithms both find the most important features and predict targets using the same fitting mechanism.
The difference between those algorithms and your examples is that those algorithms do both steps concurrently and your examples are sequential.
There is no single best algorithm for feature selection that can also be used with other models for prediction. Different algorithms will perform better on different datasets.


XGBOOST : model.predict_proba() and model.predict() conflicting behaviour I have two classes : 1 and 2

The output of model.predict_proba() -> [0.333,0.6667]

The output of model.predict() -> 1

This is happening for around 200 test values out of the test data of 10 lac. For all other records, both the functions' output don't conflict.
Even if the probability of class 2 is higher, predict function gives final class as 1.

What could be the reason for the output of model.predict() and model.predict_proba() not syncing with each other for those 200 records?

If you have two classes i.e you're doing binary classification, you should specify the target labels as 0 or 1 (neg/pos). The predict_proba() method returns estimated probabilities that a sample is in class 1.

"In the binary classification case, '1' is used to indicate positive samples, and '0' is used to indicate negative samples. We also support probability values in [0,1] as label, to indicate the probability of the instance being positive." - https://github.com/dmlc/xgboost/tree/master/demo/binary_classification


Split a string and create a news columns with parents let's say I have this dataframe, and I would like to split it into several after (.) and have it like that, with levels

I think you can achive the expected result by applying the following method. I've just create a fake dataframe to test.

import pandas as pd

df = pd.DataFrame(columns=["section"], data=[["level1.level2.level3.level4.level5"],["level1.level2.level3.level4.level5.level6"]])

columns = ['section', 'level', 'parent']

def split_parent(row):
    levels = row["section"].split(".")
    count_level = len(levels)
    sub_levels = '.'.join(levels[:(count_level-1)])

    #  create a new row for each level of section
    new_df = pd.DataFrame(columns=columns, data=[[row["section"], count_level, sub_levels]])
    for num_level in range(count_level-1, 0, -1):
        sec = ".".join(levels[:num_level])
        par = "" if num_level == 1 else ".".join(levels[:num_level-1])
        new_df = new_df.append({'section': sec, 'level': num_level, 'parent': par}, ignore_index=True)

    return new_df

result = pd.DataFrame(columns=columns)
for index, row in df.iterrows():
    result = result.append(split_parent(row), ignore_index=True)

print(result)

As result, you'll get:

                                      section level  \
0          level1.level2.level3.level4.level5     5
1                 level1.level2.level3.level4     4
2                        level1.level2.level3     3
3                               level1.level2     2
4                                      level1     1
5   level1.level2.level3.level4.level5.level6     6
6          level1.level2.level3.level4.level5     5
7                 level1.level2.level3.level4     4
8                        level1.level2.level3     3
9                               level1.level2     2
10                                     level1     1

                                parent
0          level1.level2.level3.level4
1                 level1.level2.level3
2                        level1.level2
3                               level1
4
5   level1.level2.level3.level4.level5
6          level1.level2.level3.level4
7                 level1.level2.level3
8                        level1.level2
9                               level1
10

If you need any further explanation, just write to me.

I hope I helped you.


Predict which user will buy with an offer - discount I have historical data from an e-shop transactions. I want to write a prediction model and check if a specific user will buy with or without a discount, so I can do some targeting offers.

The idea is:

If a user will buy the regular price, will not have an offer.
If a user will not buy the regular price, check if he/she will buy with an offer.

With this way, I will avoid to make an offer to someone who would buy with the regular price.

So, I am still in the brainstorming and trying to find a way for implementing the 1-2. Should I create two separate models to predict the 1) and then the 2) with the second model? Or should I join both in one prediction model?

You can use Decision trees for a single model prediction for both the set of users.

A good start would be to first read up on Decision Trees and their applications.

You can include the offer as a decision(as a boolean in this case).

buying with offer and buying without offer can be the decision criterion.

You can, in fact go ahead and put in the offer values also. For example,

offer>10% and offer <10%


Iterate over multiple dataframe rows at the same time I have 16 different dataframes with the same number of rows/columns and another 2 separate dataframes with that same shape that i'm using to compare with the 16 dataframe values.

I need to loop over all dataframes at the same time, and compare all row values with the separate dataframes, and then create another dataframe with the results like so:

comparison: sum(row_values_of_dataframe) - sum(row_values_of_reference). In the example below, the cell df_a_ref_a is equal to $(1 + 2 + 3 + 4) - (5 + 5 + 5 + 5) = -10$

Dataframe A (df_a)

col1 | col 2 | col 3 | col 4
1       2       3       4
2       4       6       8
[...]

Dataframe B (df_b)

col1 | col 2 | col 3 | col 4
10      5       2       1
4       4       6       2
[...]

Reference Dataframe 1 (ref_1)
col1 | col 2 | col 3 | col 4
5       5       5       5
5       5       5       5
[...]

Reference Dataframe 2 (ref_2)
col1 | col 2 | col 3 | col 4
3       3       3       3
3       3       3       3
[...]

Final dataframe should be:

df_a_ref_1 | df_a_ref_2 | df_b_ref_1 | df_b_ref_2 | ....
    -10          -2           -2           6        ....
     0           8            -4           4
[...]

This behaviour resembles zip() function in python.

Thanks in advance,

First: I think you want the product functionality, not zip, since you are checking every df with every ref. In zip, you would check df_a with ref_1 and df_b with ref_2 only.

Second: Your can look at the equation $(1+2+3+4)−(5+5+5+5)$ as $(1-5) + (2-5) + ...$ which is simply subtracting data frames and sum over columns.

With these two consideration, assuming you have defined your objects as follows:

df_a = {
    'name': 'df_a',
    'value': pd.DataFrame([[1, 2, 3, 4], [2, 4, 6, 8]])
}
df_b = {
    'name': 'df_b',
    'value': pd.DataFrame([[10, 5, 2, 1], [4, 4, 6, 2]])
}

ref_1 = {
    'name': 'ref_1',
    'value': pd.DataFrame([[5, 5, 5, 5], [5, 5, 5, 5]])
}
ref_2 = {
    'name': 'ref_b',
    'value': pd.DataFrame([[3, 3, 3, 3], [3, 3, 3, 3]])
}

I did this because I want to use the names in creating the name of the columns of your final df. Then your code would be:

from itertools import product

final_result = pd.DataFrame(
    {
        '{}_{}'.format(df['name'], ref['name']): (df['value']-ref['value']).sum(axis=1)
        for (df, ref) in product([df_a, df_b], [ref_1, ref_2])
    }
)

I have used dictionary comprehension to skip the ugly loop/append solution.
product function from itertools does your iteration. product on (ab, cd) gives you ac, ad, bc, bd
as for keys, df names are joined together with _, and as for values, I have subtracted two dfs and sum over columns (axis=1)

The result would then be as you expect:

   df_a_ref_1  df_a_ref_b  df_b_ref_1  df_b_ref_b
0         -10          -2          -2           6
1           0           8          -4           4

Still if you want to expand the dictionary comprehension or do not want to define dictionaries of names/values, of course you can imagine how you can write simple for loops with the same logic:

for (df, ref) in product([df_a, df_b], [ref_1, ref_2]):
    # your desired columns
    col = (df - ref).sum(axis=1)


Given rankings and the factors that determined the rankings, what is the best method to reverse engineer the rankings? I have a list of ranked items and the factors that the group used to create the rankings (like college rankings: 1, 2, 3...), and I'd like to reverse engineer their methodology.

Here's what I tried...

I took the standard score of each factor (factor value - mean /
standard deviation).
I assigned coefficients to each factor and added up all these products to create a score.
I used the scores to rank the items and compared these rankings to the real rankings, trying to minimize the difference.
Rinse and repeat by changing the coefficients.

This is a crude method and I am very new to this.  I was wondering what the best way is to figure out how this group weighted the factors to create the rankings.

Depending on the complexity of the ranking scheme, this might be more or less difficult, but if you have all the inputs and the outputs, it is very possible.

Start by plotting your data and try to see if you can see a pattern, or any obvious correlation, even though not necessarily linear. Do that for each variable.

You can also look at the distribution of your variables (what you call factors) for each of your rank, and see how they differ.

Once you have a better understanding on the relationship between the dependent variable (rank) and the independent variables (factors), then you can either:

Come up with the model right there
Apply a machine learning algorithm (linear/polynomial regression, decision trees, etc.) which will learn the model's parameters for you. Which one you chose will depend on your analysis.


Why do seaborn.dist and pyplot.hist generate two different looking histograms on the same data? I'm looking at telecom customers data. Two of the variables I'm looking at currently are:

Monthly Charges - The total amount charged to the customer monthly.
Is Senior Citizen - Whether the customer is a senior citizen.

I'm trying to plot two histograms to see if the distributions for non-senior and senior citizens is different.

If I use seaborn's distplot then I get the following result

And if I use pyplot hist then I get the following result

In the first plot the blue one towers above the orange ones in the range ~70-120 whereas in the second image the blue one always stays below the orange one.

What is the difference between these two?

The first returns a probability density of the distributions. As you can see, they integrate to 1, i.e. they cover the same area (because they are probabilities, not the raw data).

The second returns actual frequencies, and that's why you have the actual scale of the data. Different histograms having different scales.


When do I have to use aucPR instead of auROC? (and vice versa) I'm wondering if sometimes, to validate a model, it's not better to use aucPR instead of aucROC? Do these cases only depend on the "domain  & business understanding" ?

Especially, I'm thinking about the "unbalanced class problem" where, it seems more logical to use the aucPR because recall and precision are well-used metrics for this problem.

Yes, you are correct that the dominant difference between the area under the curve of a receiver operator characteristic curve (ROC-AUC) and the area under the curve of a Precision-Recall curve (PR-AUC) lies in its tractability for unbalanced classes.  They are very similar and have been shown to contain essentially the same information, however PR curves are slightly more finicky, but a well drawn curve gives a more complete picture.  The issue with PR-AUC is that its difficult to interpolate between points in the PR curve and thus numerical integration to achieve an area under the curve becomes more difficult.

Check out this discussion of the differences and similarities.

Quoting Davis' 2006 abstract:

  Receiver Operator Characteristic (ROC)
  curves are commonly used to present results
  for binary decision problems in machine
  learning. However, when dealing
  with highly skewed datasets, Precision-Recall
  (PR) curves give a more informative picture
  of an algorithm’s performance. We show that
  a deep connection exists between ROC space
  and PR space, such that a curve dominates
  in ROC space if and only if it dominates
  in PR space. A corollary is the notion of
  an achievable PR curve, which has properties
  much like the convex hull in ROC space;
  we show an efficient algorithm for computing
  this curve. Finally, we also note differences
  in the two types of curves are significant for
  algorithm design. For example, in PR space
  it is incorrect to linearly interpolate between
  points. Furthermore, algorithms that optimize
  the area under the ROC curve are not
  guaranteed to optimize the area under the
  PR curve.

This was also discussed on Kaggle recently.

There is also some useful discussion on Cross Validated.


Document similarity: Vector embedding versus BoW performance? I have a collection of documents, where each document is rapidly growing with time. The task is to find similar documents at any fixed time. I have two potential approaches:

A vector embedding (word2vec, GloVe or fasttext), averaging over word vectors in a document, and using cosine similarity.
tf-idf or its variations such as BM25.

Will one of these yield a significantly better result? Has someone done a quantitative comparison of tf-idf versus averaging word2vec for document similarity?

Is there another approach, that allows to dynamically refine the document's vectors as more text is added?

Here some points on which we can focus -

1.) Averaging the words vectors lose the order of words, making it very similar to the concept of Bag of Words. That is why with less Data Bag of words approach perform better than word2vec.
it has been found that training word2vec with potentially large data , outperform Bag of Words models. Google's results are based on word vectors that were learned out of more than a billion-word corpus.

2.) Published literature, distributed word vector techniques have been shown to outperform Bag of Words models. In this paper, an algorithm called Paragraph Vector is used on the IMDB dataset to produce some of the most state-of-the-art results to date.

3.) In your case Document size increase with time. So a good approach would be not to cluster each word rather than clustering documents. Word2vec algorithm has power to learn embedding for each document , which we call Doc2vec. it will cluster similar documents. and using cosine similarity you can find similar documents.


Understanding advantage functions The paper explaining 'Advantage Updating' as a method to improve Q-learning uses the following as its motivation.

  Q-learning requires relatively little computation per update, but it is useful to consider how the number of updates required scales with noise or with the duration of a time step, At. An important consideration is the
  relationship between Q values for the same state, and between Q values for the same action. The Q values Q(x,u1) and Q(x,u2) represent the long-term
  reinforcement received when starting in state x and performing action u1 or u2 respectively, followed by optimal actions thereafter. In a typical reinforcement learning problem with continuous states and actions, it
  is frequently the case that performing one wrong action in a long sequence of optimal actions will have little effect on the total reinforcement. In such a case, Q(x,u1) and Q(x,u2) will have relatively close values. On the other hand, the values of widely separated states will typically not be close to each other. Therefore Q(x1,u) and Q(x2.u) may differ greatly for some choices of x1 and x2. Therefore, if the network representing the Q function makes even small errors, the policy derived from it will have large errors. As the time step duration dt approaches zero, the penalty for one wrong action in a sequence decreases, the Q values for different actions in a given state become closer, and the implied policy becomes even more sensitive to noise or function approximation error. In the limit, for continuous time, the Q function contains no information about the policy.
  Therefore, Q-learning would be expected to learn slowly when the time steps are of short duration, due to the sensitivity to errors, and it is incapable of learning in continuous time. This problem is not a property of
  any particular function approximation system; rather, it is inherent in the definition of Q values.

How do I mathematically prove that this effect occurs?

1. Why are the q-values of different actions very close to each other for a given state ?

I'm going to explain this with a small example.

Consider the game of "Catch". Fruits(circular) keep falling from the top of the screen (vertically) and the agent(square) needs to just align itself to the fruit to get the reward. There are three actions that it can take : move left , stay , move right. Let's say that a2 refers to not moving the paddle, a3 refers to moving right and a1 refers to moving left. The image on the screen is to be used as the state in the MDP formalism.

Let's say we take a sub-optimal action a3(move right) and move to the next state. Then the best action in that state would be to move left (a1) and then execute the optimal action. So, the only cost difference between the actions a2 and a3 will be the two steps wasted to go and come back.

If there is no negative reward for taking the sub-optimal action, then the agent has no incentive to choose the optimal action. So, the negative rewards for taking the sub-optimal action should be high enough that the agent is discouraged from doing it. I've tried to put this intuition mathematically here. This could explain why q-values are so close to each other.

Then, the optimal Q* function satisfies the following :

2. Is there a bound on this gap ?
The bound that I've shown is approximate. The technical term for this difference is "Action Gap". A much better analysis is available here : Increasing the Action Gap: New Operators for Reinforcement Learning

3. Why are the q-values of different states very close to each other for a given action ?

This should be fairly obvious. Consider two different cases. In the first one, the fruit falls from the top left corner and in the second one, the fruit falls from the top right corner. The agent in both cases is at the center. Then, consider the action of moving left. This action will give a high q-value in the first case and a low q-value in the second case.

4. How to solve the action gap problem ?

Let's define the average reward in a state to be $V$. $V$ is approximately the average of Q-values of all the actions in a given state. If we now subtract this $V$ from each of the $Q$ values, we get a quantity which describes how good the action is compared to the average of all the action. This is called as the advantage of the action. In some sense, this is approximately the action gap. So, instead of predicting the q-values, we are directly predicting the action gap itself.

Link : A3C paper


Is CNN permutation equivariant? If I use stacked CNN layers with 3x3 kernels, zero padding, and with no pooling layers, the output feature map will consist of feature vectors, each vector of which is related directly to the original 3x3 block of the input image, right?
Therefore, for example, I could send the output feature map vectors as timesteps to LSTM and it would supposedly learn dependencies between clusters of the image given the image clusters have a temporal dependency.

If you use a single layer CNN, then each vector in the resulting activation maps would be related to the original 3x3 block. However, if you stack multiple CNN layers, you increase the receptive field of each resulting vector, as shown in the image below (taken from here):

After the CNNs, you can certainly compute an LSTM. There are, however, some design decisions you would need to take: disregarding the batch dimension, the LSTM takes as input a sequence of vectors (2D tensor) but you have a 3D tensor (height $\times$ width $\times$ num.channels) so, how are you going to make the 3D tensor fit as input to the LSTM? You could compute the LSTM of columns/rows of the image independently, in forward or reverse direction. You could also "collapse" one dimension (e.g. by averaging or taking the maximum value).


Unsupervised binning of non-normal data For some $8000$ customer profiles, in addition to a data-set, I have two kinds of scores available:

Type 1 Score ranges from $0$ to $1$ and gives the prediction probability of that profile belonging to a class. The distribution of this score is severely skewed, as shown

Type 2 Score ranges from $-5$ to $5$ and is a risk score for the profile obtained using the PRIDIT method on the actual data. The distribution of this score is fairly normal

The objective is to bin the profiles into some $N$ bins using the aforementioned information. The bins should represent various degrees of risk associated with that profile.

My first thought was to cluster the profiles based on the two scores available to us. However no meaningful clusters were observed.

The next idea was to sort the data on Type 1 Score and then cut on the points where there's a sudden change in slope, which seemed like a good idea at first but here's what the plot for Type 1 Score looks like ...

What I am thinking now is to transform Type 1 Score and Type 2 Score together to give some pseudo-normal type of distribution, hoping that some kind of discretization might give the correct bins.

My questions are:

How can I transform the two scores into a pseudo-normal distribution?
What would be a better way to approach the above problem?

It often is appropriate to use quantile bins.

I.e. to get four bins, choose the smallest 25%, then up to the median, then the Q75, ...


Connection between Embedding and LSTM and Dense layer I am building a "predict next word" model using the following model architecture. The codes fine, but I have a few questions:

# define model
model = Sequential()
model.add(Embedding(vocab_size, 50, input_length=seq_length))
model.add(LSTM(100, return_sequences=True))
model.add(LSTM(100))
model.add(Dense(100, activation='relu'))
model.add(Dense(vocab_size, activation='softmax'))
print(model.summary())

Between the Embedding and LSTM layers, are there 50 connection weights, or 50 * 100 connection weights? Based on the documentation, this LSTM block contains 100 cells. So I am not sure how exactly does the Embedding layer connect to the LSTM layer?

Also, between LSTM(100) layer and the Dense(100, activation='relu') layer, are there 100 connection weights or 100 * 100 connection weights?

Thanks!

The embedding layer has an output shape of 50. The first LSTM layer has an output shape of 100. How many parameters are here? Take a look at this blog to understand different components of an LSTM layer.

Then you can get the number of parameters of an LSTM layer from the equations or from this post.

Now, between LSTM(100) layer and the Dense(100, activation='relu') layer, there should be 100*(100 + 1) parameters. The additional 1 is for the bias.


Sementic segmentation data and model compile in Keras I have multi-label data for semantic segmentation. For semantic segmentation for one class I get a high accuracy but I can't do it for multi-class segmentation.

I have 6 class labels so my Y train matrix is equal [78,480,480,6] ('channel last'), where 78 - number of images 480,480 -image size, 6-number of masks and X train matrix [78, 480, 480, 1]

The last lines of my CNN model:

l = Conv2D(filters=64, kernel_size=(1,1), activation='relu')(l)
output_layer = Conv2D(filters=6, kernel_size=(1,1), activation='sigmoid')(l)

model = Model(input_layer, output_layer)
model.compile(optimizer=Adam(2e-4), loss='categorical_crossentropy', metrics=['accuracy'])

I don't know what I hove done wrong that my multi-label semantic segmentation model doesn't have proper results.

Image and masks:

Two things that stand out is that you are using sigmoid as activation which is used for binary classification (it just squash values between 0 and 1). The other is that your learning rate is 5x lower than the default value.

Also because you are classifying images, I think the title is misleading; semantics is associated with "meaning" i.e. with language.


Nested cross-validation generalization error for multiple models I am referring to this question:

Nested cross-validation and selecting the best regression model - is this the right SKLearn process?

In the answers it shows that nested cv can estimate the generalization error of hyperparameter optimization for different algorithms.
But in my opinion the choice between different algorithms is also an optimization process, which leads to generalization errors. Therefore, either the algorithm choice should be part of the inner cv or another third cv would have to be introduced to evaluate the error for the algorithm choice. Is this a correct assumption ?

In general you are right and in this answer it has been done as far as I see. The models are compared to each other while the best tuning of them is found, both inside the loop. It looks fine.

About your point, yes. But the point in Machine learning is that at some point we need to stop/limit our attempts as the number of algorithms which can do the task are very large. We usually try to evaluate different families of algorithms and then narrow the search from there but at the end we can never claim that the best answer we found is necessarily the best possible answer. In another POV, this is the main idea behind many research papers in ML. They just creatively find/modify an algorithm and show that it works better than previously applied algorithm through a benchmark dataset.


What is the procedure to realize zero mean and unit variance? The tag on feature scaling says:

  Popular feature scaling types include scaling the data to have zero
  mean and unit variance, and scaling the data between a given minimum
  and maximum value.

Are there any details that help us to generate zero mean and unit variance?

To generate zero mean and unit variance, you are looking at standardization. Where a feature $x$ becomes scaled feature $z$ like this:

$$z = \frac{x - \mu}{\sigma}$$

Where $\mu$ is the mean of that feature and $\sigma$ the standard deviation.


Questions about a multivariate timeseries forecasting model - keras I have trouble understanding the model I'm trying to create.
I have few questions so I'll explain my model first and what I'm trying to do:

I have created sequences of data (input and ouput of the model) 7 timesteps each so the input would be the values of the days of a certain week and the output is the values of the days the following week (**so input1 of my model has an imput shape of (7,1) same as the output**).
I also have prepared another input list that has some extra features like holiday flag and weather condition for the **following week** so the model's second input2 has the shape (7,7) each example.

This is the full model summary:

After the lstm layers and a fully connected NN, I tried to concatenate the two inputs together, basically, I want to concatenate input2 to the output of the layer dense_1.
So I'm concatenating (7,1) shape with the second input 'input_2' of shape (7,7)
My questions:
1- The outputs of my dense layers confuse me I thought they would be in the shape (None, number of units) but they seem to be (None, 7 , number of units) there is always the 2nd dimension "7" which i don't understand.

2- For the concatenation part, since i'm adding a 7x7 input to a 7x1 input i was expecting to have 49+7=56 units connected to the next dense layer but the number of parameters tells me it's not the case being just 14*120+120=1800

3- I was thinking of having 7 units in the last layer ( 1 for each day to predict ) but had to make it 1 so it would output (7,1) and for it to work, there is something i'm clearly missing.

This is the code for model definition:
input1 = tf.keras.layers.Input(shape=(7,1),name="input_1")
x = tf.keras.layers.Conv1D(120, kernel_size=5,strides=1,activation="relu",
                           padding="causal",input_shape=[7, 1])(input1)
x = tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(210, return_sequences=True),name="LSTM_1")(x)
x = tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(210, return_sequences=True,name="LSTM_2"))(x)
x = tf.keras.layers.Dense(60,activation="relu",name="dense_1_1")(x)
x = tf.keras.layers.Dense(30,activation="relu",name="dense_1_2")(x)
x = tf.keras.layers.Dense(7,name="dense_1_3")(x)

input2 = tf.keras.layers.Input(shape=(7,7),name="input_2")
concat = tf.keras.layers.concatenate([x, input2],name="concat_1")
x = tf.keras.layers.Dense(120,activation="selu",name="dense_2_1")(concat)
x = tf.keras.layers.Dense(90,activation="selu",name="dense_2_2")(x)
x = tf.keras.layers.Dense(60,activation="selu",name="dense_2_3")(x)
output = tf.keras.layers.Dense(1)(x)

model = tf.keras.Model([input1, input2], output)

These are the dimensions of the inputs (input1 & input2) and the output

From Keras docs:
https://keras.io/api/layers/core_layers/dense/

Input shape
N-D tensor with shape: (batch_size, ..., input_dim). The most common
situation would be a 2D input with shape (batch_size, input_dim).
Output shape
N-D tensor with shape: (batch_size, ..., units). For instance, for a
2D input with shape (batch_size, input_dim), the output would have
shape (batch_size, units).
If the input to the layer has a rank greater than 2, then Dense
computes the dot product between the inputs and the kernel along the
last axis of the inputs and axis 1 of the kernel (using tf.tensordot).
For example, if input has dimensions (batch_size, d0, d1), then we
create a kernel with shape (d1, units), and the kernel operates along
axis 2 of the input, on every sub-tensor of shape (1, 1, d1) (there
are batch_size * d0 such sub-tensors). The output in this case will
have shape (batch_size, d0, units).

The input to dense_1_1 is (None, 7, 420). The Dense layer will be applied to every [:,i,:] for in 1..7 slice of the input producing an output of size (None, 7, 60)
If you're expecting an output with shape (None, 60) you can reshape the input to (None, 420*7) but that would not make sense in this case since you would want to apply the Dense layer to each time slice of your input.


What if the votes for 2 classes are equal in an ensemble learning technique? Suppose in ensemble learning technique, if the number of models that predict class 1 is equal to the number of models that predict class 0. Then, which class will be decided as output?

Depending on the implementation, this problem never occurs. Most of the implementations build an odd number of trees or models to ensure one class's dominance.
However, some implementations allow an even number of trees and models. In such a case, some algorithms simply just throw an error by mentioning the tie has occurred.
Especially in R, some of the libraries solved this issue by randomly picking one of the classes.
The last and best (I think) solution is choosing the class that is more frequent than another. In other words, the class that is more frequent in the training data is chosen arbitrarily by the algorithm to break the tie.


How to determine layers structures appropriate for a specific application? I'm playing around with Keras. I went through their example of MNIST data, cats vs dog etc.
The structure of the network they suggest is different in each case. The MNIST stucture:
# Build a simple model
inputs = keras.Input(shape=(28, 28))
x = layers.experimental.preprocessing.Rescaling(1.0 / 255)(inputs)
x = layers.Flatten()(x)
x = layers.Dense(128, activation="relu")(x)
x = layers.Dense(128, activation="relu")(x)
outputs = layers.Dense(10, activation="softmax")(x)
model = keras.Model(inputs, outputs)

So basically

preprocessing
flattening the data
2 intermediate layers
output layers.

The cats vs dogs classifier has some preprocessing as well, but then many more intermediate layers of different types. I've played around with each, adding/remove layers and changing characteristics in each. Seems to influence loss/accuracy, but then many different variations also seems to have pretty similar results.
My question is - for some lambda application, what is a good process to determine the basic architecture of the networks (how many layers, what types)? I understand that Keras has a tuner to set the parameters, however my understanding is that this is for the model parameters. I'm talking about one step back from that - how many layers? Which types? Are there good rules of thumbs for that, good resources of "standard" structure as a function of types of inputs?

From the structure you have outlined it looks like you're going for a dense fully-connected neural network. A general rule of thumb for practitioners on this topic is that increasing the size of a single hidden layer is more likely to yield a more performant network than adding successive layers. This is obviously not true in all cases - but as you are experimenting with architectures it is more useful to start by increasing the size of your one hidden layer before you jump to two of even three. See this for more details.
I would recommend removing one of those hidden layers and increasing the number of nodes on the remaining.
Something else to note: if you are looking at image data you might consider using convolutional layers. These are much more appropriate for problems with data that are spatially related (images etc..). If you do decide to go with this approach the previous advice does not apply as it relates only to simple dense multilayer perceptrons. Instead review this.


How to control for Co-variate shift in test data set compared to train data for regression task? I am working on a regression project. But I am facing the problem of covariate shift in features due to time delay.Test data was collected a year later due to which there has been some change in distribution. The research paper that I am working on also mentioned this shift in feature values but didn't mention anything about rectifying it.
I found this but it talks about changing training set distribution.
What I need is a way to make test data distribution closer to training data or some other way in which I could control for it.
I am using sklearn and Elastic-net for regression.

There are several options:

Change the training dataset

Use some of the test data as training data. This the best option since it better models the problem you are trying to solve.

Since it happens over time, take only the most recent data for training.

Manually engineer features. If you have knowledge of how the test data feature values are different, then explicitly put that knowledge in the model.

Increase regularization. For scikit-learn's ElasticNet, increase alpha hyperparameter. Unless your validation dataset is similar to your test data, you'll have no evidence that this will improve the modeling.

Switch algorithms to one that is better at generalization. Possibly, Random Forest could be better than regularized linear regression.


Feature selection/reduction technique for combination of features in image processing I have a combination of features extracted from 3 descriptors. Namely, GLCM based features (correlation, homogeneity, energy, and contrast ), Local binary patterns (256), and discrete wavelet transform coefficients,
and a dataset of 40 cases. I am using SVM with RBF kernel.
Would there be a need for the feature selection? If so, what do you recommend?

I'd recommend assessing the performance of the model "as-is" and then decide for feature selection or not.

If you have good performance for both train / test sets, then there's no need to perform feature selection.

If you end up overfitting your data (low train error, high test error) you may want to reduce the number of features using feature selection / regularization (the cost function is penalized when the model uses a large number of parameters)

If you have low train performance, this means that you have high bias (the model does not learn properly from those features). This usually happens if the model is not complex enough to extract structure from the data. You may want to add features in this case, rather than selecting them out.

Hope this helps.


Converting a Pandas Series string of multiple attributes into individual attributes? I'm working on an Airbnb dataset in order to predict the nightly price--where each example is one Airbnb listing (ie a listing for someone's house, room, apartment, etc). Each feature is a characteristic of the listing (ie # bedrooms, # bathrooms, average review score etc).

One of the features is 'amenities', which are the amenities offered for the listing (ie wifi, tv, kitchen, etc). Each example contains a string of its own amenities like so:

{TV,"Cable TV",Wifi,"Air conditioning",Kitchen,Elevator,Heating,Washer,Dryer,"Smoke detector","Fire extinguisher",Essentials,Shampoo,Hangers,"Hair dryer",Iron,"Laptop friendly workspace","Self check-in",Keypad,"Private living room","Hot water","Bed linens",Microwave,"Coffee maker",Refrigerator,Dishwasher,"Dishes and silverware","Cooking basics",Oven,Stove,"Long term stays allowed"}

Dtype is a string ('O' in pandas).

I plan to remove the {} on the ends and the " quote signs using pd.Series.str.replace() for each character--unless those are useful to keep.

But past that I'm not sure of the best way to go about handling the feature itself.

My questions is:

Should I create individual binary columns for each amenity (ie 1 in column "TV" if TV is an amenity, 0 in column "wifi" if wifi isn't offered)? I'm hesitant because that would add a few dozen columns. If I should do this, how can I convert the current format to binary columns with Pandas?

This might Pandas convert a column of list to dummies.
You can try using the MultiLabelBinarizer class from sklearn.


How to get xgbregressor feature importance by column name? xgb = XGBRegressor(n_estimators=100, learning_rate=0.08, gamma=0,
  subsample=0.75, colsample_bytree=1, max_depth=7)

xgb.get_booster().get_score(importance_type='weight')
xgb.feature_importances_

This will return the feature importance of the xgb with weight, but how to return it with column name?

If you have X_train Dataframe then you can take columns from there and loop it with feature_importances_

for col,score in zip(X_train.columns,model.feature_importances_):
    print(col,score)


BPE vs WordPiece Tokenization - when to use / which? What's the general tradeoff between choosing BPE vs WordPiece Tokenization? When is one preferable to the other? Are there any differences in model performance between the two? I'm looking for a general overall answer, backed up with specific examples.

(This answer was originally a comment)
You can find the algorithmic difference here. In practical terms, their main difference is that BPE places the @@ at the end of tokens while wordpieces place the ## at the beginning. The main performance difference usually comes not from the algorithm, but the specific implementation, e.g. sentencepiece offers a very fast C++ implementation of BPE. You can find fast Rust implementations of both in Hugginface's tokenizers.


Dimension change of convolutional layer after applying the feature map In this lecture I don't understand how the output has 1 layer after applying the feature map to the 3 layer input.

In the video the speaker says, "center the filter on each pixel and perform a dot product which gives one value per position."
So I'm guessing it is a dot product between (L1, L2, L3) dot (F1, F2, F3) where L's are pixel values in the input and F's are whatever values are in the filter.


How fit_transform, transform and TfidfVectorizer works I'm a machine learning beginner and I tried to use the cosine similarity on fuzzy matching purpose. In the following example I want to compare 'data_dirty' with 'data_clean' :

When I have to vectorize my data I do not really understand what is the purpose of fit_transform and WHY 'dirty_idf_matrix' has ONLY transform argument with SAME vectorizer than 'clean_idf_matrix' which has saved the value with fit if I understood well.

Col_clean = 'fruits_normalized'
Col_dirty = 'fruits'

#read table
data_dirty={f'{Col_dirty}':['I am an apple', 'You are an apple', 'Aple', 'Appls', 'Apples']}
data_clean= {f'{Col_clean}':['apple', 'pear', 'banana', 'apricot', 'pineapple']}

df_clean = pd.DataFrame(data_clean)
df_dirty = pd.DataFrame(data_dirty)

Name_clean = df_clean[f'{Col_clean}'].unique()
Name_dirty= df_dirty[f'{Col_dirty}'].unique()

vectorizer = TfidfVectorizer(min_df=1, analyzer=ngrams)
clean_idf_matrix = vectorizer.fit_transform(Name_clean)
dirty_idf_matrix = vectorizer.transform(Name_dirty)

thank you for your help !

I'm not really sure what you're asking, but in general, you need to fit an Estimator to data so it can learn what it has to do, then you transform data with it. fit_transform just does fit and then transform. Here you fit the transformer to Name_clean, and then apply it to both in turn. That's pretty normal.


Which type auto encoder gives best results for text I did I couple of examples for auto encoders for images and they worked fine. Now I want to do an auto encoder for text that takes as input a sentence and returns the same sentence. But when I try to use the same auto encoders as the ones I used for the images I get bad results.

I guess the reason for this is that my text is sparse and I have a big vocabulary size of 500K words.

Do you have a link of a working example of an auto encoder for text in Keras?
I saw that in most papers they use cross-entropy as a loss function. How does cross-entropy calculate the loss exactly ? Does it make sense to use cross-entropy even if I do a character by character auto encoder?

A working example of a Variational Autoencoder for Text Generation in Keras can be found here.

Cross-entropy loss, aka log loss, measures the performance of a model whose output is a probability value between 0 and 1 for classification. Cross-entropy loss goes up as the predicted probability diverges from the actual label. In the case of character-by-character autoencoder, each character in the vocabulary would be a label.

Cross-entropy works if the input and output are the same size, that is the case in character-by-character autoencoder. Often times in text analysis, the input and output sequences are different lengths so a second term, encoder loss, is added to the objective function.


Features of word vectors in Word2Vec I am trying to do sentiment analysis. In order to convert the words to word vectors, I am using Word2Vec model. Suppose I have all the sentences in a list named 'sentences' and I am passing these sentences to word2vec as follows:
model = word2vec.Word2Vec(sentences, workers=4 , min_count=40, size=300,   window=5, sample=1e-3)

Since I am noob to word vectors, I have two doubts:
1- Setting the number of features to 300 defines the features of a word vector. But what these features signify? If each word in this model is represented by a 1x300 numpy array, then what do these 300 features signify for that word?
2- What does downsampling as represented by 'sample' parameter in the above model do in actual?

1- The number of features: In terms of neural network model it represents the number of neurons in the projection(hidden) layer. As the projection layer is built upon distributional hypothesis, numerical vector for each word signifies it's relation with its context words.

These features are learnt by the neural network as this is unsupervised method. Each vector has several set of semantic characteristics.
For instance, let's take the classical example, V(King) -V(man) + V(Women) ~ V(Queen) and each word represented by 300-d vector. V(King) will have semantic characteristics of Royality, kingdom, masculinity, human in the vector in a certain order. V(man) will have masculinity, human, work in a certain order. Thus when V(King)-V(Man) is done, masculinity,human characteristics will get nullified and when added with V(Women) which having femininity, human characteristics will be added thus resulting in a vector much similar to V(Queen). The interesting thing is, these characteristics are encoded in the vector in a certain order so that numerical computations such as addition, subtraction works perfectly. This is due to the nature of unsupervised learning method in neural network.

2- There are two approximation algorithms. Hierarchical softmax and negative sampling. When the sample parameter is given, it takes negative sampling. In case of hierarchical softmax, for each word vector its context words are given positive outputs and all other words in vocabulary are given negative outputs. The issue of time complexity is resolved by negative sampling. As in negative sampling, rather than the whole vocabulary, only a sampled part of vocabulary is given negative outputs and the vectors are trained which is so much faster than former method.


Dealing with highly variable feature set size I'm trying to use machine learning for security event classification. My goal is to predict the outcome (true positive or false positive) of a specific event. An event has a set of variables in it, called observables. These can be urls, ip addresses, file hashes etc. (8 types altogether). However, one event could contain a small number of observables or a huge number as well. Since I want to predict the outcome based on these observables, my feature size varies in length - between 1 and 2500.
This in an example of the data set:
['user1', '1.1.1.1', 'explorer.exe', NULL, NULL, NULL ...]
['google.com', 'msword.exe', NULL, NULL, NULL ...]
['user3', '1.1.1.9', 'explorer.exe', 'e0d123e5f316bef78bfdf5a008837577', 'http://google.com, NULL ...]

How can I handle this scenario? I'd like to try out a classification as well as neural networks as well for comparison.
Edit
I ended up using the Bag of Words approach, as the "observables" I mentioned could be interpreted as words in a document. From there my case is a relatively known text classification problem and I achieved good results with Naive Bayes algorithms and hash vectorization.

Before thinking of what type of algorithm you could use, I would think of how to properly preprocess your data.
Depending on how many possible values you might have for each of your 8 possible types (if I understood correctly), you could construct a 0's and 1's dataset, it is, indicating the presence or absence of each possible value in each event.
This would lead you to having a sparse matrix, but that is something you can deal with with some tools, a possibly nice example is in this link
Conclusion:

I would first try to identify if there are fixed possible values for each one of your types (by creating groups of ips by region? are there a fixed number of possible .exe, urls...?)
preprocess your data, so that http://google.com and google.com are the same value (i.e., by deleting the http:// from urls)
if you think the number of possible values is not ridiculously big, you might try to go for constructing the sparse matrix

After this, you could think of wich kind of algorithm to apply, and not going crazy for a neural network from the beginning.


How can I extract the top words from a string in my dataframe column? I have a column in my dataframe which is in a string providing description of a product. For example : This is a shirt. It is blue in color. The sizes available are large, small. The shirt is tight fit.

I want to extract the keywords from this description like shirt, blue, large, small, sizes, tight etc.

I am using python. How can I achieve this ?

If the words are all short and like the one you mentioned, I think you're pretty done with just removing the stopwords. It is a simple text cleaning of the most frequent words that don't bring additional meaning to the text.
Let me know.


Discrete Values Matrix: Select Rows and Scattered Plot problem I ran in a somewhat basic problem with Orange 3.3.1... but wonder if this is a bug or something I don't understand:

I have a Matrix, all with discrete values.
I select Columns, that goes ok
Select Rows... and use the « is one of » filter with 3 values out of 15 in a particular column.
But when I Scatter Plot. Values matching the filter seems to have been selected, but the X Axis still show X names, all of them,  that are not in « is one of ». On top of it, values have shifted to to wrong X name...

Anyone has an idea???

Regards

I removed the paint widget. Works fine witout it. That was in the version i had the problem. The latest daily build i installed after worked fine with the color widget. Thanks.


Retraining a machine learning model Created a Machine Learning model on some data. Used this model for predicting test data. Model has learnt only from training data. How this model can be retrained as new data come in for prediction. I don't know whether it is true that retraining a model is necessary so that it can predict new data more accurately.

It depends on a lot of factors:

Does the function you're learning change overtime ? If you try to detect dogs in pictures, and you already have a good accuracy, then relearning is not necessary. But if you try to detect topics in news streams, then you'll need to update your model regularly.
Do you have initially enough data ? If you receive a stream of training data, then you should find an algorithm that can be trained in an online manner.
If you are looking for constantly improving the accuracy, then the more data you feed the better results you'll get. Regularly retraining a new model with the old data + the new ones might be sufficient.


how classification scores are interpreted? I would like to know how to interpret  classification scores (i am not sure about the word score or probability, please correct me). For example, for a binary classification positive values are labeled as 1, and -1 for negative ones. Now, is it fair to say that for a score 10 the instance is more likely to be successfully predicted than a score 5, despite the result that can be wrong.
Thanks.

If the scores have values that are higher than 1, I wouldn't call them probabilities. Probabilities should always be between 0 and 1.
And indeed, the higher the score, the more likely an example is to be positive, this is the most natural interpretation of a score.


Pick a model from multiple models using a decision tree Let us say, I have 4 classification models on a training data set of various examples. Now, I want to choose which 1 out 4 models (or what combination of the 4 models) should I choose to get the final probability for a particular test example. So, basically I want to create a meta model to get ultimate probabilities for all the test examples. I know I can combine all 4 models using regression but I am particularly interested in using a decision tree to combine all 4 models.

So, the meta decision tree model will have 4 features/variables (4 model probabilities) as per my current thoughts. Is this a right approach? Should I include more features in it?

Please explain how do I combine 4 models using a meta decision tree model to choose the best model out of 4 for a particular test example.

Well, you can use this approach for stacking (AFAR this is how the process you are trying to perform is called):

1) Prepare a dataset in the format:

Your original features which you used to fit the 4 estimators (remove the target variable from it)
Add new target variable in it - the categorical output which shows which model should be picked up for that specific input. To obtain it - train your 4 estimators on your input and than label each input with the number of the estimator which has the closest output to the target

2) Train your stacker - on that dataset from #1 - for example the decision tree. Or any other classifier which will be able to accurately predict the number of the estimator which should be used for that specific input.

3) When using in combat mode - first run the stacker trained in #2 on new input and obtain an id of an estimator which best suites this example. Than run that estimator against new input record and get your response.

Hope that makes sense.
I tried it and it worked nicely for one of my problems.

Also you could try to employ some kind a 'blending' of these 4 predictors. Here is an approach you could take (my usage of terminology here becomes a bit frivolous since I am an engineer and not a scientist, so if someone provides a more elaborate answer on blending - please do).

First of all, recall how ANNs for classification work - and would work for you example, if you will use ANN as a stacker. You have input layer which reads the input values, some hidden layers and an output layer with a set of neurons, each neuron is associated with a specific class. For our case it would be 4 neurons. After you feed input each of 4 output neurons will output some real value, for example:
Neuron 1 - 5.4
Neuron 2 - 0.3
Neuron 3 - 15.3
Neuron 4 - 3.3

That output means that ANN thinks that the input belongs to 'class 3' and that you should use 3rd predictor for that example to get best results.

But! Let's think about probabilities. What is the probability of input to belong to ANY of 4 classes? It is 1.0 (since you have only 4 classes). And by default ANN doesn't have the knowledge that this the case. So for us it would be good if the ANN will output probability of data belong to X class instead of just some real value. For example we would like to obtain such output:

Neuron 1 - 0.6
Neuron 2 - 0.2
Neuron 3 - 0.15
Neuron 4 - 0.05

Which will read as 'I am 60% sure that you should use estimator 1 for that input, 20% sure  - that estimator 2' etc. Such objective is called 'cross-entropy'. Modern libraries for building ANNs support it and you could tell network to train to output these probabilities.

By now I believe you already got what I suggest to do. Prepare the dataset as I described at the beginning of the answer, than train an ANN with cross-entropy objective to output the probabilities. When you run it in combat mode:

1) For new input run it through ANN and obtain probabilities, like these:
Neuron 1 - 0.6
Neuron 2 - 0.2
Neuron 3 - 0.15
Neuron 4 - 0.05

2) Run each predictor against input and obtain their predictions
3) Multiply each prediction by the associated probability and sum the results:

predictor1_result*0.6 + predictor2_result*0.2 + predictor3_result*0.15 + predictor4_result*0.05

And get you final result.

Why do that? First of all, we can't be sure that each time we realy pick up the best estimator, since our stacker (ANN in our case) could make mistakes. By using such approach we give the stacker a chance to compensate for possible mistakes - which minimizes the average prediction error.

Also, consider that you get such output from ANN:
Neuron 1 - 0.45
Neuron 2 - 0.45
Neuron 3 - 0.05
Neuron 4 - 0.05

In such case the best option for us is to almost average the predictions from predictor 1 and 2 - since ANN is not really sure which of predictors to use.


Rule engine vs machine learning We would like to implement rule engine in one of our products, but when I read on the Internet, many are suggesting that rule engines are an old way of doing things and it will be very hard to manage as we add more rules into it. To be successful in implementing a rule based system we need to know all the rules beforehand.

Instead, everyone seems to be suggesting that we use Machine Learning algorithms to predict the outcome, but I am new to Machine Learning.

So my Question is, should I learn Machine Learning or go on implementing Drools as rule engine in my application.

Or can you suggest any online resources achieving the same using Machine Learning?

I'll try and answer your question point-by-point from an architect's point-of-view:

  We would like to implement rule engine in one of our products, but
  when I read on the Internet, many are suggesting that rule engines are
  an old way of doing things

The 'people of the internet' don't know your scale and the use-case. Only you know. So, they might be suggesting ideas from a very high-scale point of view, when your system is not necessarily a high scale one. So, pick the right idea/tool for the job.

  suggesting that rule engines are an old way of doing things and it
  will be very hard to manage as we add more rules into it

True. Rule-based systems are not scalable, due to obvious reasons. And, that's why people consider ML as a highly scaled out rule-based engine, wherein the rules can no longer be understood by humans. Thus, the black-box name.

  So my Question is, should I learn Machine Learning or go on
  implementing Drools as rule engine in my application.

That is for you to decide. We don't know your scale. But, if you think the rules would be increasing at a huge rate in the future, then ML is the best way forward for your use-case.

  Or can you suggest any online resources achieving the same using
  Machine Learning?

For me, every recommender system is a complex rule-based engine under the hood. So, go through Amazon's and how they do their recommendation.


Sentiment Analysis: Creating dictionary from dataset I've collected around half a million of unmarked comments from a newspage. The newspage has anti-foreign background. Because of that, a relativley  high number of comments contain hate.

Do you have an idea how to start creating a dictionary out of the comments? The dictionary shall contain words, which are connected to hate. With the dictionary I want to perform a text analysis for hate, later.

Because there is no such dictionary for German language I tried some things. For instance, I checked how often words, which are connected to emotions, appear in the comments. But it turned up, that they seem to be contained as often as any other word. I also thought about a Bag-of-Words and some other things, but I don't know.

I'm doing the analysis with Python 3 and R.

Kind regards and thanks in advance!

When looking on the probability of word occurrence, you will get stop words and other popular words. You are interested in words that appear more in the comments (assumed hate related) than in normal use.

Get a neutral  resource (e.eg., a German newspaper, the German wikipedia, maybe Google ngrams for German). Compute the probability on the neutral source $P_{neutral}(word)$, the probability of the comments $P_{comments}(word)$ and look for the words of lift  $\frac{P_{comments}(word)}{P_{neutral}(word)} > 1$. These are the words that are more popular at the comments.

As @chi wrote, many repositories can both give you a head start and help you tune the needed lift threshold (you might want words that appear much more often in the comments).

After this phase you might need do do a finer analysis. For example, I guess that there will be politicians names that will appear more often in the comments. See here for a possible approach.


Delete all rows between two values in a dataframe that repeat multiple times in a column I have dataframe and let's say inside of it is a column_A. This column_A has 3 strings as values, call them 'new_records', 'deletions', 'changes'  that repeat across the dataframe multiple times in that order always with multiple rows in between. I want to delete all rows from the beginning of deletions to the end of changes, i.e. I want to leave only new_records in the dataframe. The dataframe looks like this:

column_A         column_B     column_C ....
NEW_RECORDS        val1         val2
string1_new        val3         val4
string2_new        val5         val6
  NaN              val9         val10
  NaN              val11        val12
string3_new
 ...
DELETIONS          val7         val8
string1_del         ...           ...
   NaN              ...           ...
string2_del         ...           ...
  ...
CHANGES             ...           ...
 str1_ch            ...
 str2_ch
  ...
NEW_RECORDS
 str200_new        ...
 str300_new           ...
  NaN
  NaN
  ...
DELETIONS
 NaN
 str100_del
 NaN
 str290_del        ...
  ...
CHANGES
 str1000
 str20000
  NaN
   ...           ...

I want to have at the end only chunks of rows between new_records and deletions values, without rows that belong to the deletions group and changes group. How can I do that?

UPDATE:

There are many rows after the 'new_records' and before the start of 'deletions' group and there are many rows after the start of deletions group and beginning of the 'changes' group. I need to extract only rows that belong to the new_records group. So all rows after the value 'new_records' and before the value of 'deletions' across all dataframe.

You can achieve this by forward filling the blank values, and then selecting only those with new_records:

df.fillna(method='ffill')

df = df[df['column_A'] == 'new_records']

Depending on the actual values in your data frame, you may need to first replace what appear to be empty/space strings with NaN's:

df['column_A'] = df['column_A'].replace(r'^\s*$', np.nan, regex=True)


Domain-specific data science programs I am looking for domain-specific data science programs (as a major, not a minor or specialization). I found programs in bio-statistics (and other disciplines in public health), MS in marketing analytics, MS in learning analytics and MS in urban informatics. Does anyone know any other domain-specific data science field of study?

Unsurprisingly most programs in data science focus mostly on business or marketing applications. I too would like to see more programs focused on Biotechnology, Gaming or Energy for example but I think that in the future those kind of programs will become more and more available.
However there are a few I know of that offer a somewhat different approach.

Regarding health informatics there are many offers where you can find anything between Public health to Computational Biology and Quantitative Genetics. Take a look at the full list.

http://www.mastersindatascience.org/specialties/health-informatics/

On that list there are a few certification programs but there also are full fledged programs that lead to a M.Sc.

If you are more interested in Marketing Analytics you can find the same list but for marketing related programs.

http://www.mastersindatascience.org/specialties/marketing-analytics/

other options like Information Systems are also present, just browse that site and you might find something that interests you.


Clustering Small Text Descriptions Im presented with a unique text classification problem.

Im given a list of descriptions each containing 3-8 words. I know that there are some descriptions that are nearly the same, but the majority of them are significantly different from each other. My objective is to group the descriptions that are roughly the same and to consider the rest of the descriptions as unique. This data set can be considered too long to manually label for supervised learning.

My thought process so far:

TF-IDF will come in handy as the frequency of words that match is extremely low (therefore any match is highly valuable).
Unsupervised clustering techniques like K Means might be useful, but there is an inherit, "K" selection issue with K Means in this space. For example, if we have 1,000 descriptions and only 10 descriptions (5 clusters of 2 descriptions) are supposed to be clustered with one another (leaving 9,990 unique descriptions), we would need 9,995 clusters to represent the structure accurately (9,990 clusters with 1 item and 5 clusters with 2 items each). This issue would lead to an extremely difficult model building phase as the number of clusters would be so high (perhaps this isn't an issue and this theory should be tested).

I am very new to the space of unsupervised learning being used with NLP. I have a few more thoughts that I could elaborate on, but I really need some advice on how others deal with clustering text in this way. I dont need to use K Means or any other specific idea if better alternatives fit the problem space better.

Edit:

I am starting to think that DBSCAN will be a much better clustering model. This documentation explains that each point can be clustered and outliers are not taken into consideration more so than real groups (under specific configuration). In more technical terms, I believe setting a low eps score with a minpts values of 2 will be a good theory to test. This essentially assumes that most data belongs to its own cluster (most of the data is unique), but some data (under a weak minimum points configuration) should indeed be grouped into a cluster. DBSCAN avoids the pitfalls of K Means by dealing with outliers in an algorithmic way. K Means would be heavily dependent on a genius guess of a "K" value.

On the top of my head:

with regular clustering techniques you could try to use text-specific distance/similarity measures instead of only considering distinct words as elements. There are hybrid string similarity measures such as SoftTFIDF which take into account character-based and word-based similarity.
use lemmas instead of words in order to facilitate matching of the same concept
for more specific NLP methods you could look at topic modeling and/or word sense induction techniques. The two follow similar ideas in different ways: the former is closer semantically to your case, the latter is meant to work with small context windows similar to your short descriptions size. I'm not sure what is the state of the art nowadays but Latent Semantic Analysis was the standard not so long ago. Afaik these techniques are meant for a quite high amount of data, but they might be worth trying.


Make a chatbot using slack How did anaek created chat bots using slack for HR Tools and how much time will it take as a fresher to make a similar product?

I don't know the specific tools that you mentioned. But generally speaking, to create a chatbot, you would need to connect an API from Slack to your own API layer that handles the NLP side of it. So, if you're just starting out, your focus should be on creating an NLP algorithm that handles the types of questions that you want your bot to answer and converse with. You need to create code that both recognizes new inputs and understands how those inputs fit to everything else (i.e. an ongoing conversation). From there, you can create an API that you consume in any number of programs an you can practice with it (even in a terminal screen) for your own testing & algorithm training purposes. At that point, you should be able to integrate that into any number of platforms, slack being only one option.


How to input data to LSTM? I have a labelled dataset (0 and 1) with two types of short length time series data (from day1-day5) as follows.

Type1 (Temperature at location1):

sensor1, 38, 38, 35, 33, 32
sensor2, 33, 32, 35, 36, 32
and so on ....

Type2 (Temperature at location2):

sensor1, 18, 18, 12, 11, 09
sensor2, 13, 12, 15, 16, 12
and so on ....

In summary my dataset looks as follows.

ID, time-series1, time-series2, label
sensor1, [38, 38, 35, 33, 32], [18, 18, 12, 11, 09], 0
sensor2, [33, 32, 35, 36, 32], [13, 12, 15, 16, 12], 1
and so on ....

I would like to use my dataset in LSTM (or a equivalent deep learning model).

However, I am still not clear how I can use my dataset in a deep learning model such as LSTM.

I am happy to provide more details if needed.

RNN input data must follow this pattern:

  ( Number of observations , Number of input series , Window size )

Keep the number of observations to None when you define the LSTM input_shape. The number of input series in this case is 2, since you are using two time series to make predictions. Window size is the length of your time series row. Your input shape would then be:

input_shape = (2, window_size)


Comparing Efficiency of Machine Learning Algorithms I know that the algorithmic complexity of CNNs and other methods of deep learning can't be fully expressed in simple terms, like big oh complexity. That said, how would one go about comparing the efficiency/complexity of CNNs to standard machine learning methods (say decision trees, LDAs, Naive Bayes, etc.)? Deep learning methods are expensive, but how do we know how expensive they are comparatively?

One deep learning method can apply to multiple problems (for example computer vision, natural language processing etc), the performance of the method may be different depends on the problem that the method handles, but there is "popular dataset" that usually use by deep learning researchers to benchmark their method. Let say you have deep learning method, if you want to know how efficient your method for handwritten classification you can benchmark your method by classifying mnist dataset and compute its accuracy. If you want to compare the performance of your method with another method you can search on the internet another method that uses the same dataset for benchmark, here is an example


Accuracy differs between MATLAB and scikit-learn for a decision tree Is there any possibility to vary the accuracy of same data set in matlab and jupyter notebook by using python code ?

For same data set, at first I applied it in matlab and get 96% accuracy for decision tree method, then I apply that same data set in jupyter notebook by using python code where I get 53% accuracy for C4.5 (decision tree) by using k-fold cross validation.

I didn't understand where's the problem for getting different accuracy for same dataset and same method.

My procedure in python code is given below:

import pandas as pd
import numpy as np
from sklearn import tree
from sklearn.model_selection import KFold

train=pd.read_csv('E://New.csv')
train.head()

# define X and y
feature_cols = ['Past','Family_History','Current','current or previous
               workplace','diagnosed with a mental health condition by a
               medical professional?','do you feel that it interferes with
               your work when being treated effectively?','Gender']
X = train[feature_cols]

# y is a vector, hence we use dot to access 'label'
y = train['Diagonised condition']

kfold = KFold(n_splits=10,random_state=None)
model = tree.DecisionTreeClassifier(criterion='gini')

results = cross_val_score(model, X, y, cv=kfold,scoring = 'accuracy')
result = results.mean()*100

std = results.std()*100
print (result)

It is hard to make a direct comparison between a white box implementation (scikit-learn) and a black box implementation (MATLAB).

One guess they are using different algorithms. scikit-learn uses an optimized version of the CART algorithm. Maybe MATLAB uses  ID3, C4.5, or something else. Another guess two implementations are using different hyperparameters (e.g., different splitting criteria, max depth, minimum node size, ...).

Since decision trees are white-box models, you can examine their internal structure. Plot both trained trees. See how they each are making the splits and how many splits are being made.


How to implement a convolutional autoencoder? I would like to implement a convolutional autoencoder in Tensorflow, but it is not clear how the decoder part should work.

Each layer of the encoding, is a convolutional layer with activation function and then a pooling layer.

But how will the decoding work? I know that I have to add padding in each layer, but what will be the reverse of the convolution? How will it reproduce the original data from much less variables and the padding?

Transposed Convolutions is what you are looking for, for more details take a look here:
https://medium.com/towards-data-science/types-of-convolutions-in-deep-learning-717013397f4d