Basic backpropagation question I'm attempting to create my own neural network optimization model from scratch. I'm getting hung up on backpropagation.

I have just a few basic questions:

When adjusting a given weight by the learning rate, $\alpha$, the weight is selected by changing the weight with the highest $\delta$. Are these deltas all calculated simultaneously and independently? Or is it calculated one layer at a time, or some alternative method?
How is the delta calculated? I am not following any of the calculations since it seems to leave delta unsolved. Maybe I'm misunderstanding. Is it common to use finite differencing to estimate deltas, or is this very slow/inaccurate?
When optimizing on time series training data, is the total error the sum of the output errors of all the observations? If so, is the delta the change in the total error? What is the most efficient way to approach this problem?

If this is inappropriate for this board, apologies in advance.

Edit: Clarifying my question:

Suppose we have 3 inputs, two layers of hidden neurons with Sigmoid activation functions, and 3 neurons per layer, and one output neuron.

I will denote weights as $w_{i,j,k}$ where $i =$ target layer, $j =$ target node, and $k =$ preceding node. Nodes will be denoted as $n_{x,y}$ where $x = $ layer and $y =$ node.

The error for the output is given as $E = \Sigma \cfrac{1}{2} (Target - Out)^2$.

I want to clarify that I am doing this properly. Calculating $\delta$ is simple if the weight targets the output layer. However, if I want to find $\cfrac{\partial E}{\partial w_{2,1,1}}$, is it expanded as:

$\cfrac{\partial E}{\partial Out} \cfrac{\partial Out}{\partial net_O}
\cfrac{\partial net_O}{\partial n_{2,1}} \cfrac{\partial n_{2,1}}{\partial net_{2,1}} \cfrac{\partial net_{2,1}}{\partial w_{2,1,1}}$ where $Out = \cfrac{1}{1+e^{-net_O}}$

$\cfrac{\partial E}{\partial Out} = (Out - Target)$

$\cfrac{\partial Out}{\partial net_O} = (Out)(1-Out)$

$\cfrac{\partial net_O}{\partial n_{2,1}} = w_{3,1,1}$

$\cfrac{\partial n_{2,1}}{\partial net_{2,1}} = n_{2,1}(1-n_{2,1})$

$\cfrac{\partial net_{2,1}}{\partial w_{2,1,1}} = n_{1,1}$

Putting it all together:

$\cfrac{\partial E}{\partial Out} \cfrac{\partial Out}{\partial net_O}
\cfrac{\partial net_O}{\partial n_{2,1}} \cfrac{\partial n_{2,1}}{\partial net_{2,1}} \cfrac{\partial net_{2,1}}{\partial w_{2,1,1}} = $

$(Out - Target)(Out)(1-Out)(w_{3,1,1})(n_{2,1})(1-n_{2,1})(n_{1,1})$

In essence, to calculate $\delta$ for each layer of weights further back in the back propagation, you just add two more terms, being the derivative of the sigmoid for that node, and the node previous, as well as another weight term? Is this correct?

To make our life easier, let's omit optimization methods here (e.g. Learning Rate, Momentum) and focus on vanilla Backpropagation. As you know, backprop consists of computing the partial derivative with respect to each weight in order to penalize each weight for its role in the global output error; such that:

\begin{equation}
    \frac{\partial E}{\partial W_{ij}} = e_{i}y_{j}
\end{equation}

With $E$ the global output error to minimize, $W$ the weight to update, $y$ the output value, and $e$ the local error. The local error $e$ is the unknown we want to compute! :)

Using the chain rule, the gradient can be expressed as:
\begin{equation}
    \frac{\partial E}{\partial W_{ij}} = \frac{\partial E}{\partial y_{i}} \frac{\partial y_{i}}{\partial x_{i}} \frac{\partial x_{i}}{\partial W_{ij}}
\end{equation}

With $x_{i}$ the input value to neuron $i$.

First, the partial derivative with respect to $W_{ij}$ can be computed as follows:

\begin{equation}
    \frac{\partial x_{i}}{\partial W_{ij}} = y_{j}
\end{equation}

Which is simply equal to the output value $y_{j}$!

Secondly, the partial derivative with respect to $x_{i}$ is:

\begin{equation}
    \frac{\partial y_{i}}{\partial x_{i}} = \frac{\partial \phi(x_i)}{\partial x_{i}}
\end{equation}

Which is simply $\frac{\partial \phi(x_i)}{\partial x_{i}}$, namely the derivative of the activation function (i.e. Sigmoid, Tanh, ReLU...).

Thirdly, the partial derivative with respect to $y_{i}$ can be computed as follows:

\begin{equation}
    \frac{\partial E}{\partial y_{i}} =
    \begin{cases}
        \begin{aligned}
            \frac{\partial}{\partial y_{i}} (T_i - y_i)
        \end{aligned} &
        \text{if $i \in$ output layer,}
        \\
        \begin{aligned}
            \frac{\partial}{\partial y_{i}} \left(\sum\limits^{n}_{j=1}W_{ij}\frac{\partial E}{\partial y_{j}}\right)
        \end{aligned} &
        \text{otherwise;}
    \end{cases}
\end{equation}

With $T$ the target expected output (i.e. the label). The local error $e$ is computed differently depending on the layer position. For the output layer, the error is proportional to the difference between the predicted value (i.e. $y_{i}$) and the expected output (i.e. $T_{i}$). Otherwise, the error in hidden layers is proportional to the weighted sum of errors from connected layers.

Fourthly, combining the partial derivatives allow the computation of the error at a given neuron $i$ such as:

\begin{equation}
    \frac{\partial E}{\partial y_{i}} \frac{\partial y_{i}}{\partial x_{i}} = e_{i} =
    \begin{cases}
        \begin{aligned}
            \frac{\partial \phi(x_i)}{\partial x_{i}} (T_i - y_i)
        \end{aligned} &
        \text{if $i \in$ output layer,}
        \\
        \begin{aligned}
            \frac{\partial \phi(x_i)}{\partial x_{i}} \left(\sum\limits^{n}_{j=1}W_{ij} e_{j}\right)
        \end{aligned} &
        \text{otherwise;}
    \end{cases}
\end{equation}

As you can see here, you need to recursively compute the local error $e$ all the way to the input layer, that's why it's called backprop! ;) I think that what you missed!

If we now simplify:

\begin{equation}
    \frac{\partial E}{\partial W_{ij}} = \frac{\partial E}{\partial y_{i}} \frac{\partial y_{i}}{\partial x_{i}} \frac{\partial x_{i}}{\partial W_{ij}} = e_{i}y_{j}
\end{equation}

We already know $y_{j}$, and we now know how to compute $e_{i}$ all the way to the input layer. So the weight can finally be updated from the gradient as follows:

\begin{equation}
    W_{ij} = W_{ij} - \,e_{i}y_{j}
\end{equation}

Sources:

https://www.youtube.com/watch?v=i94OvYb6noo
Backpropagation, Intuitions: http://cs231n.github.io/optimization-2/
Chapter 9.2 Classification by Backpropagation in Data mining: concepts and techniques: concepts and techniques. Jiawei Han, Micheline Kamber, and Jian Pei. 2011.


How to scrape a table from a webpage? I need to scrape a table off of a webpage and put it into a pandas data frame. But I am not being able to do it. Let me first give you a hint of how the table is encoded into html document.

United States Total**
15,069.0
14,575.0
100.0

Arizona

Pinal Energy, LLC
Maricopa, AZ
50.0
50.0
NA
2012-07-01
2014-03

Arizona Total
50.0
50.0
NA

The body of the table begins with
....
. Each
....
 is a row of the table.Within each row, that is within each pair of
....
, each column is given by 50.0	.

Here are my questions:

1) How do I scrape it ? I am using BeautifulSoup and requests for this purpose as well as pandas module. I tried the following:

r = requests.get(url)
bs = BeautifulSoup(r.text)
info = bs.findALL('tr','td')
  ....
  ....

But it is giving me this error:

TypeError                                 Traceback (most recent call last)
<ipython-input-24-32d9483e2c59> in <module>()
      1 bs = BeautifulSoup(r.text)
----> 2 info = bs.findALL('tr','td')
      3 #print bs

TypeError: 'NoneType' object is not callable

2) I need to skip some of the rows based on the text in it. For example I don't want to read in the row in which the word 'Total' appears (as inUnited States Total**	). How do I do that ? Although, it is not extremely important as I can get rid off it later, but skipping these rows while reading the data is ideally what I need.

I know it is a long post, but if someone can help me with it, i would greatly appreciate it. Please let me know if more information is needed.

Thanks much.

This will give you all the  values under <tr>:

bs=BeautifulSoup(data, "lxml")
table_body=bs.find('tbody')
rows = table_body.find_all('tr')
for row in rows:
    cols=row.find_all('td')
    cols=[x.text.strip() for x in cols]
    print cols


Dictionary based statistical NER learner If I have to paraphrase the current NER methodologies, it generally finds patterns in strings and creates its own "vocabulary", so to speak.

Naturally, it would perform like a charm with a mammoth dataset, curated carefully with hardwork and labeled entities.

But what if, the system is first introduced with a dictionary of named entities for respective categories and then given a sample literature or simple tweets for that category, for eg, and from them it "learns" how those named entites appear in a context.

The difference is subtle from regex based, in that in regex it would try to match strings and more the dictionary size, more the rules, then more its usefulness.

But in this system, it would actually learn how "eating an apple" and "eating at Apple" both can be accurately classified with little training set, the subtleness of grammar when mentioning about fruit and about a company.

Some intuition on its implementation in CRF++, CRFsuite, Stanford or any other.

General disclaimer
Not a data scientist. Just a passing thought from an ML enthusiast.

It sounds like you're describing using a gazetteer and training, which is not particularly unusual. See here for example:

  How do we go about identifying named entities? One option would be to look up each word in an appropriate list of names. For example, in the case of locations, we could use a gazetteer, or geographical dictionary, such as the Alexandria Gazetteer or the Getty Gazetteer.

Using a gazetteer has tradeoffs; see here for some discussion.

Here's an example of someone using it in CRF++ as a feature.


How to anonymize (de-identify) data in Python? I have tried a simple algorithm to anonymize the data using the de-identification technique. But the code doesn't work for me. I want to anonymize the data by slightly changing the values of strings and integers. The data sample is available here

This is what i have tried.

import pandas as pd
import uuid as u
import datetime as dt
 # generate a pseudo-identifier sequesnce using python random number generator library uudi.

    def uudi_generator(length):

    uudi_list= list()
    i=0
    while i < length:
        uudi_list.append(u.uuid4())
    i+=1
    return uudi_list

#import original originaL dataset
dataset = pd.read_csv('bankcredit-data.csv')

# pseudo identifier
sLength = len(dataset['housing'])
dataset.insert(0, 'uuid', pd.Series(uudi_generator(sLength), index=dataset.index))

# Transaction record attached to the original
dataset.insert(0, 'transaction_date', pd.Series([dt.datetime.now]*sLength, index=dataset.index))

#transcation record is attached to originaL data file
dataset.to_csv('bankcredit-data.csv')

#delete identifiabLe record from dataset
del dataset['firstnamme']
del dataset['lastname']

# export  de-identified dataset as csv to be shared with the user
dataset.to_csv('deidentified-data.csv')

I don't have access to input dataset, I created sample on my own and tried your code with little modification and it worked

input dataset:
housing      lastname   firstname
64403818    AA  AB
30893205    AC  AD
89883627    AE  AF
90302087    AG  AH

After I executed, input dataset appended with uid and transaction_date

transaction_date    uuid                                     housing    lastname
10/31/2019 20:35    809b4505-2269-48b0-8833-e7502fc2738a    64403818    AA
10/31/2019 20:35    7de91a91-0b58-4703-b62b-4278efe22b05    30893205    AC
10/31/2019 20:35    d6b8cfbd-a9c2-4ffd-b336-0a23547445ea    89883627    AE
10/31/2019 20:35    11db6b3a-9679-4422-b754-4c1b23aa4801    90302087    AG

firstname
AB
AD
AF
AH

and the output dataset becomes
transaction_date    uuid                                    housing
10/31/2019 20:35    809b4505-2269-48b0-8833-e7502fc2738a    64403818
10/31/2019 20:35    7de91a91-0b58-4703-b62b-4278efe22b05    30893205
10/31/2019 20:35    d6b8cfbd-a9c2-4ffd-b336-0a23547445ea    89883627
10/31/2019 20:35    11db6b3a-9679-4422-b754-4c1b23aa4801    90302087

import pandas as pd
import uuid as u
from datetime import datetime
import datetime as dt
 # generate a pseudo-identifier sequesnce using python random number generator library uudi.

def uudi_generator(length):
    uudi_list= list()
    i=0
    while i < length:
        uudi_list.append(u.uuid4())
        i+=1
    return uudi_list

#import original originaL dataset
dataset = pd.read_csv('C:\\mylocation\\input_credit_data.csv', index_col=False)

# pseudo identifier
sLength = len(dataset['housing'])
dataset.insert(0, 'uuid', pd.Series(uudi_generator(sLength), index=dataset.index))

# Transaction record attached to the original
dataset.insert(0, 'transaction_date', pd.Series([datetime.now()]*sLength, index=dataset.index))

#transcation record is attached to originaL data file
dataset.to_csv('C:\\mylocation\\input_credit_data.csv',index=False)

#delete identifiabLe record from dataset
del dataset['firstname']
del dataset['lastname']

# export  de-identified dataset as csv to be shared with the user
dataset.to_csv('C:\\mylocation\\output_bankcredit-data.csv',index=False)


Shapley values without intercept (or without `expected_value`) I have a model and I want to derive its interpretability by using feature contributions. In the end, I want to have some contribution per feature such that the sum of contributions equals the prediction of the model.
One approach may be to use shapley values. The issue with shapley values and its implementation in the shap python library is that it also derives and expected_value, which I'll call $E$. To obtain the prediction of the model, one must add all the shapley values and $E$. Is there a way to derive feature contributions without the need of $E$?

My solution
I've derived a solution but I'm not sure it makes sense. Let's say I have features $x_1, \dots, x_n$ with shapley values $\phi_1, \dots, \phi_n$ for the model $f$. Then, I have
$$
f(x) = \phi_1 + \dots + \phi_n + E \Rightarrow f(x) = (1 + \frac{E}{\sum \phi_i})(\phi_1 + \dots + \phi_n)
$$
(by using a simple math trick). Then, I claim that I have shapley values, $\hat{\phi_1}, \dots, \hat{\phi_n}$ where
$$
\hat{\phi_j}=(1 + \frac{E}{\sum \phi_i})\phi_j
$$
These values are suitable with
$$f(x) = \hat{\phi_1} + \dots + \hat{\phi_n}$$
But, does it make sense to call them shapley values?
Any criticism is more than welcome.

This is very similar to fitting a linear regression and not including an intercept, and I think they will face similar issues.
To be very concrete, consider an example with $f(x)=1,\ E=1, \ \phi_1=1, \ \phi_2=-1$.
Then your scaling factor is undefined, trying to divide by zero.  Well OK, but you won't often get such exact numbers.  Let's tweak them to
$$f(x)=1,\ E=1, \ \phi_1=1.01, \ \phi_2=-1.$$
Now the scaling factor is $1+1/.01=101,$ so that $\hat{\phi}_1=102.01$ and $\hat{\phi}_2=-101$.  This isn't locally too bad, because it still indicates the relative importance of the two variables in making this prediction.  But if you want to compare this to other points $x$ (where say $f(x)=2,\ \phi_1=1,\ \phi_2=0$, so that $\hat{\phi}_1=2,\ \hat{\phi}_2=0$) things will look quite strange, and aggregating importances over your entire sample may be very misleading.
For the connection to linear regression and more, see this excellent description of shapley values.


Prove two equations are equivalent I'm reading Neural Networks and Deep Learning and running into trouble with the math. One of the exercises says:

Write out $a'=\sigma (wa + b)$ in component form, and verify that it gives the same result as the rule
$$\frac{1}{1 + \exp(-\sum_{j}w_jx_j - b)}$$for computing the output of a sigmoid neuron.

Not even sure where to start here. Can anyone help me out? I'd really appreciate a detailed explanation.

In your first equation w is weight matrix $(w_1 + w_2 + \ldots + w_j)$ and a is the input vector $(x_1 + x_2 + \ldots + x_j)$. Writing wa in component form means taking the dot product of these two vectors, which is $\sum_{j}w_ja_j$. Now, you just apply the sigmoid function to get your final answer:$$\sigma (z) = \frac{1}{1 + \exp(-z)}$$ $$\Rightarrow\sigma (\textbf{wa} + b) = \frac{1}{1 + \exp(-\sum_{j}w_jx_j - b)}$$


How to measure the correlation of different algorithms In stacked generalization, several algorithms are trained on the training set (i.e. at layer 1) and their predictions are then stacked using a layer 2 model. In many documentations, it is said that it is better that the layer 1 algorithms should be of low correlation. How can one compute this correlation between algorithms ?

For regression tasks correlation will be simply the correlation between the predicted values, for binary classification it will be correlation between predicted probabilities. In multiclass classification you can find correlation between predicted factor variables using the hetcor package in R


Back-propagation through max pooling layers I have a small sub-question to this question.

I understand that when back-propagating through a max pooling layer the gradient is routed back in a way that the neuron in the previous layer which was selected as max gets all the gradient. What I'm not a 100% sure of is how the gradient in the next layer gets routed back to the pooling layer.

So the first question is if I have a pooling layer connected to a fully connected layer - like the image below.

When computing the gradient for the cyan "neuron" of the pooling layer do I sum all the gradients from the FC layer neurons? If this is correct then every "neuron" of the pooling layer has the same gradient?

For example if the first neuron of FC layer has a gradient of 2, second has a gradient of 3, and third a gradient of 6. What are the gradients of the blue and purple "neurons" in the pooling layer and why?

And the second question is when the pooling layer is connected to another convolution layer. How do I compute the gradient then? See the example below.

For the topmost rightmost "neuron" of the pooling layer (the outlined green one) I just take the gradient of the purple neuron in the next conv layer and route it back, right?

How about the filled green one? I need to multiply together the first column of neurons in the next layer because of the chain rule? Or do I need to add them?

Please do not post a bunch of equations and tell me that my answer is right in there because I've been trying to wrap my head around equations and I still don't understand it perfectly that's why I'm asking this question in a simple way.

If this is correct then every "neuron" of the pooling layer has the same gradient?

No. It depends on the weights and activation function. And most typically the weights are different from the first neuron of the pooling layer to the FC layer as from the second layer of the pooling layer to the FC layer.

So typically you will have a situation like:

$ FC_i = f(\sum_j W_{ij} P_j)$

Where $FC_i$ is the ith neuron in the fully connected layer, $P_j$ is the jth neuron in the pooling layer and $f$ is the activation function and $W$ the weigths.

This means that the gradient with respect to P_j is

$grad(P_j) = \sum_i grad(FC_i) f^\prime W_{ij}$.

Which is different for j=0 or j=1 because the W is different.

  And the second question is when the pooling layer is connected to another convolution layer. How do I compute the gradient then?

It does not make a difference what type of layer it is connected to. It is the same equation all the time. Sum of all the gradients on the next layer multiplied by how the output of those neurons is affected by the neuron on the previous layer. The difference between FC and Convolution is that in FC all neurons in the next layer will provide a contribution (even if maybe small) but in Convolution most neurons in the next layer are not at all affected by the neuron in the previous layer so their contribution is exactly zero.

  For the topmost rightmost "neuron" of the pooling layer (the outlined green one) I just take the gradient of the purple neuron in the next conv layer and route it back, right?

Right. Plus also the gradient of any other neurons on that convolution layer which take as input the topmost rightmost neuron of the pooling layer.

  How about the filled green one? I need to multiply together the first column of neurons in the next layer because of the chain rule? Or do I need to add them?

Add them. Because of the chain rule.

Max Pooling
Up to the this point, the fact that it was max pool was totally irrelevant as you can see. Max pooling is just the that the activation function on that layer is $max$. So this means that the gradients for the previous layers $grad(PR_j)$ are:

$grad(PR_j) = \sum_i grad(P_i) f^\prime W_{ij}$.

But now $f = id$ for the max neuron and $f = 0$ for all other neurons, so $f^\prime = 1$ for the max neuron in the previous layer and $f^\prime = 0$ for all other neurons. So:

$grad(PR_{max neuron}) = \sum_i grad(P_i) W_{i\ {max\ neuron}}$,

$grad(PR_{others}) = 0.$


Training and test split for time series analysis I need help with below code:

dataset.tail(4)

                Date    Close
    1274    1/24/2019   4871.96
    1275    1/25/2019   4925.82
    1276    1/28/2019   4888.58
    1277    1/29/2019   4928.18

I want to split my dataset to training and test using:

training_data=dataset[dataset.Date<datetime(2018, 1, 1)]

But I get the following error:

TypeError
Traceback (most recent call last)
<ipython-input-74-904bd8b45617> in <module>()
----> 1 training_data=dataset[dataset.Date<datetime(2018, 1, 1)]
      2 training_data.tail(4)

TypeError: 'module' object is not callable

Your code looks incomplete but you can definitely try the following to split your dataset-

X_train, X_test, y_train, y_test = train_test_split(dataset,
                                                    y,
                                                    test_size=0.3,
                                                    shuffle=False)

Note: y will be a series object for your dependent variable.

Because of this reason, using train_test_split with shuffle=True is not a good practice and should strictly be avoided.

Alternately, for time series you can use TimeSeriesSplit() and look for the implementation here.


Predicting t+1 from a set of sequences Say I have have an experiment where I release a single rat into a maze and wait for it to reach the end. Say I also track this rat's position in the maze at various times. Let's do this $n$ times. Now, I have a dataset of $n$ variable-length movement profiles, since some of the animals finish faster, some take longer, etc. Is there a way I can analyze this data to then predict the next move $(x_{t+1}, y_{t+1})$ given an $(x_t, y_t)$? What techniques could I use and how would they work? I was thinking LSTM or RNNs, but unsure what techniques are used for problems like this.

Yes you can use LSTM or RNN,but from my point of view can be other possibilities.
If you want to use the recurrent neural network you should use a supervised training mode:

(many - to - many)

The input x1 is the move made by the rat at time1 and y1 is the move of the rat at time 2, x2 is the move of the rat at time 2 and y2 is the move of the rat at time3 and so on...

But for your kind of problem a recurrent netowork could not be the best choice. What i recommend you is to use a decision tree, or a random forest algorithm, since the possible outcomes of the rat are just 4: right, left, up, down, and you have a finite state of positions, "the maze". The best you can do is build a random forest algorithm where the features can be, for instance, the walls around the rat, the distances of these walls from it,the distance from the exit and the number of times the rat has passed through that state, and the labels are: right, left, up, down.

Remember that a machine learning model works better then a deep learning model if the data are not consistent and a random forest is a very powerful machine learning tool


Training a Convnet on 300GB data I have a large training set of ~300GB (which is a subset of an even larger dataset ~15TB).

I am trying train a Convnet with Keras (Tensorflow backend) to do something similar to semantic segmentation.

I couldn't find any valuable resources to handle such large data. Any suggestions for best practices for such humungous data is appreciated.

Thanks!

You don't need to load the whole dataset into memory at once. The only data you need in memory are the samples in a single training batch. Use the fit_generator method rather than fit to pass in an iterator that feeds samples to your model from disk rather than loading all of that data at once. Here's a tutorial that discusses this more.


Why do people prefer Pandas to SQL? I've been using SQL since 1996, so I may be biased. I've used MySQL and SQLite 3 extensively, but have also used Microsoft SQL Server and Oracle.

The vast majority of the operations I've seen done with Pandas can be done more easily with SQL. This includes filtering a dataset, selecting specific columns for display, applying a function to a values, and so on.

SQL has the advantage of having an optimizer and data persistence. SQL also has error messages that are clear and understandable. Pandas has a somewhat cryptic API, in which sometimes it's appropriate to use a single [ stuff ], other times you need [[ stuff ]], and sometimes you need a .loc. Part of the complexity of Pandas arises from the fact that there is so much overloading going on.

So I'm trying to understand why Pandas is so popular.

The real first question is why are people more productive with DataFrame abstractions than pure SQL abstractions.
TLDR; SQL is not geared around the (human) development and debugging process, DataFrames are.
The main reason is that DataFrame abstractions allow you to construct SQL statements whilst avoiding verbose and illegible nesting. The pattern of writing nested routines, commenting them out to check them, and then uncommenting them is replaced by single lines of transformation. You can naturally run things line by line in a repl (even in Spark) and view the results.
Consider the example, of adding a new transformed (string mangled column) to a table, then grouping by it and doing some aggregations. The SQL gets pretty ugly. Pandas can solve this but is missing some things when it comes to truly big data or in particular partitions (perhaps improved recently).
DataFrames should be viewed as a high-level API to SQL routines, even if with pandas they are not at all rendered to some SQL planner.

You can probably have many technical discussions around this, but I'm considering the user perspective below.
One simple reason why you may see a lot more questions around Pandas data manipulation as opposed to SQL is that to use SQL, by definition, means using a database, and a lot of use-cases these days quite simply require bits of data for 'one-and-done' tasks (from .csv, web api, etc.). In these cases loading, storing, manipulating and extracting from a database is not viable.
However, considering cases where the use-case may justify using either Pandas or SQL, you're certainly not wrong.  If you want to do many, repetitive data manipulation tasks and persist the outputs, I'd always recommend trying to go via SQL first.  From what I've seen the reason why many users, even in these cases, don't go via SQL is two-fold.
Firstly, the major advantage pandas has over SQL is that it's part of the wider Python universe, which means in one fell swoop I can load, clean, manipulate, and visualize my data (I can even execute SQL through Pandas...).  The other is, quite simply, that all too many users don't know the extent of SQL's capabilities.  Every beginner learns the 'extraction syntax' of SQL (SELECT, FROM, WHERE, etc.) as a means to get your data from a DB to the next place.  Some may pick up some of the more advance grouping and iteration syntax.  But after that there tends to be a pretty significant gulf in knowledge, until you get to the experts (DBA, Data Engineers, etc.).
tl;dr: It's often down to the use-case, convenience, or a gap in knowledge around the extent of SQL's capabilities.


Machine learning testing data I am new to machine learning and it might be a bit of a stupid question.

I have implemented my model and its working. I have a question about running it on the testing data. Its a binary classification problem. If I know the proportions of classes in test data how could I use it to improve the performance of my model or prediction made by the model. So lets say 75% belong to class 1 and 25% to class 0

Any help is greatly appreciated

No, your model isn't supposed to know about your test data, if you include clues in your training about what's in your test data, you will do something called
Data Leakage.
Data leakage would lead to Over-fitting which will give you good results on that particular test set, but won't generalize to other data.
Lets say, you deploy this model in production and feed into it real-life data that it never encountered before, the predictions will be below the expectations you had in the training/testing phases because of those two phenomenons i mentioned.
I suggest you tweak a bit more your model during the training phase, maybe clean your data more, do something called OverSampling and UnderSampling if the target classes are imbalanced (You have for example 90% / 10% proportions in your training dataset), pick better features etc..
In conclusion: adjusting your model to have good predictions on your test data, in particular, is not good practice and will lead to creating a bad model on unseen data.


Why Extra-trees should only be used within ensemble methods? I was reading scikit-learn documentation for Extremely Randomized Trees and I found this warning:

  Warning: Extra-trees should only be used within ensemble methods.

Why is that?

In a random forest tree, a random subset of features is available for consideration at each split. Extra-trees takes this a step further by using a random threshold at each split. The idea is that a forest (an ensemble of trees) with a large number of trees that have learned something different from each other (due to the random nature of the features and thresholds at each split) will result in a better prediction than any single tree.

If one were to use a random threshold for each split (what extra-trees do) in a single tree, one would likely end up with a suboptimal tree that predicts poorly.


How should I sample my validation set if I randomly sample training data? I have:
training dataset of size 150k.
validation dataset of size 19k.
At each epoch I randomly sample without replacement 10k datapoints for training because I get Out of Mem Errors.
I need to downsample my validation set too. Which of the following methods seem most appropriate:

Randomly sampling validation set which is x% of 10k and use the same set across every epoch.
Randomly sampling validation set which is x% of 10k at every epoch.

Actually you should never use any sampling techniques on your testing/evaluation data because this could lead to wrong classification results.
If your dataset is imbalanced you could perform upsampling or downsampling techniques (like SMOTE) on your training data only.
If you want to benchmark your multi-class classification you need to rely on e.g. the confusion matrix, recall, precision and F1 measure. Please keep in mind that the accuracy measure cannot be interpreted if you have too imbalanced data.


Do Clustering algorithms need feature scaling in the pre-processing stage? Is feature scaling useful for clustering algorithms? What type of features, I mean numeric, categorical etc., are most efficient for clustering?

Clustering algorithms are certainly effected by the feature scaling.

Example:

Let's say that you have two features:

weight (in Lbs)
height (in Feet)

... and we are using these to predict whether a person needs a 'S' or 'L' size shirt.

We are using weight+height for that, and in our trained set let's say we have two people already in clusters:

Adam (175Lbs+5.9ft) in 'L'
Lucy (115Lbs+5.2ft) in 'S'.

We have a new person - Alan (140Lbs+6.1ft.), and your clustering algo will put it in the cluster which is nearest.  So, if we don't scale the features here, the height is not having much effect and Alan will be allotted in 'S' cluster.

So, we need to scale it. Scikit Learn provides many functions for scaling. One you can use is sklearn.preprocessing.MinMaxScaler.


How can the process of hypertuning of XGBoost parameters be automated? I'm using xgboost for training a model on a data with extreme class imbalance. After referring from here.

After performing grid search and some manual settings, I found that the following parameters work the best for me:

weight <- as.numeric(labels) * nrow(test) / length(labels)
upscale <- sum(weight * (labels == 1.0))

xgb_params = list(
  objective = 'binary:logistic',
  eta = 0.1,
  max_depth = 4,
  eval_metric = 'auc',
  max_delta_step = 10,
  scale_pos_weight = upscale
)

How can the process of setting optimal hyperparameters for xgboost be automated for best AUC? Please note that some of these parameters aren't supported by the caret implementation of xgboost but are very important for the model I have to design.

In general, if you want to automate fine tuning a model's hyper parameters, its best to use a well tested package such as caret or MLR.

I've used the caret package extensively. Here is a reference of the parameters supported by caret for tuning a xgboost model.

To automatically select parameters using caret, do the following:

First define a range of values of each parameter you would want caret to search. Define this in the tuning grid.
Start model training using caret after specifying a measure to optimize, e.g. accuracy or Kappa statistic, etc.
Plot or print the performance comparison for various parameter values, refine and repeat if required.

Refer to the caret guide here to get step-by-step instructions on using it.

For handling class imbalance, I've found from my experience that adjusting weights is not as helpful as under-sampling majority class and over-sampling the minority class, or a combination of the two. However, it all depends on the size of data available and the case at hand.

In case you need to tune some parameters which are not supported by caret, then, you could write your own iterative loop to train and test the model for different values of that parameter and then choose one that works best. I think most of the really relevant parameters have already been included in caret.

You would need to adjust these parameters in case the population itself changes over time. Or, the methods to gather data and their accuracy may change which could result in performance deterioration. You could run a simple check by comparing the performance of your model over the current dataset vs. a 6 month older dataset. If the performance is similar, then you may not need to update the model in the future.


Training a function that maps n-dim to n-dim As an example, say the input is an array of numbers representing an audio snippet and the output is a transformed/filtered version of it.

What would be the proper name for that? Which are examples of algorithms for the job?

EDIT:
More specifically, I want to train audio source separation. The input is a mixed sound (spectrogram) and the output is the sound with some energy removed in certain frequencies. The function needs to recognize some pattern in the input and decide what to remove.

I would call a mapping between N dimensional input and N dimensional output a regression problem.

If you add more constraints about the relation between the input and output it might be called different names: linear filtering, nonlinear filtering, etc...

some examples on common techniques for that would be: neural networks, regression trees, regularised regressions...


Can BERT do the next-word-predict task? As BERT is bidirectional (uses bi-directional transformer), is it possible to use it for the next-word-predict task? If yes, what needs to be tweaked?

BERT can't be used for next word prediction, at least not with the current state of the research on masked language modeling.
BERT is trained on a masked language modeling task and therefore you cannot "predict the next word". You can only mask a word and ask BERT to predict it given the rest of the sentence (both to the left and to the right of the masked word).
This way, with BERT you can't sample text like if it were a normal autoregressive language model. However, BERT can be seen as a Markov Random Field Language Model and be used for text generation as such. See article BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model for details. The authors released source code and a Google Colab notebook.
Update: the authors of the MRF article discovered their analysis was flawed and BERT is not a MRF, see this
Update 2: despite not being meant for next word prediction, there have been attempts at using BERT that way. Here you can find a project that does next word prediction with BERT, XLNet, RoBERTa, etc.


Normalizing a list with both positve and negative numbers to the range [-1 1] For example, I have a list [-518.8134, 480.1884,160.4761]
How to normalize this to [-1 1] range?

You can do this yourself or use a library such as scikit-learn which has a MinMaxScaler.
data = [-518.8134, 480.1884, 160.4761]

# manual scaling
scaled = [-1 + 2 * (x - min(data)) / (max(data) - min(data)) for x in data]
print(scaled)
# [-1.0, 1.0, 0.3599364886029235]

# using scikit-learn's MinMaxScaler
from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler(feature_range=(-1, 1))
scaled = scaler.fit_transform([[x] for x in data])
print(scaled)
# [[-1.        ]
#  [ 1.        ]
#  [ 0.35993649]]


RMSE is higher for bigger values of target variable - how to decrease I am solving a problem with machine learning and I have some data with two integer type independent variables and a continuous dependent variable. I am optimising to RMSE. I had fairly large RMSE value on my validation data. I learned that my model didn't do good on larger values of target; so, I have tried removing rows with larger values and that didn't help. So, now in the process of understanding the mistakes, I calculated RMSE for each ground truth value and it's prediction from validation set and plotted it to understand where big mistakes had happened. Apparently, my model still doesn't do good at larger values of target.
Here is the plot:

And here is the plot showing relationship between ground truth values and predictions:

As you can see, my model's predictions got worse as the values got large. How do I prevent this?
Some information about my data(only what I can reveal):

There is absolutely no linear relationship between independent variables and target. So, I have used a tree based model and random forest is giving me relatively good results.
I can even went to say that both of my independent variables can be called categorical variables with very high categories.
Also, there are a lot of values in independent variables, that occur only once.
all the variables are highly skewed to the right.(range of IV_1: 0 to 3,700; IV_2: 0 to 40; target variable: 0 to 39,000)

How do I decrease my RMSE or doing what will decrease it?

It looks like that your prediction is clamping at 750.
Be mindful of the fact that Tree can't predict a Regression value that is outside the range it has been trained on.
So, first of all, please assure that your data doesn't have a trend.


Is max_depth in scikit the equivalent of pruning in decision trees? I was analyzing the classifier created using a decision tree. There is a tuning parameter called max_depth in scikit's decision tree. Is this equivalent of pruning a decision tree? If not, how could I prune a decision tree using scikit?

dt_ap = tree.DecisionTreeClassifier(random_state=1, max_depth=13)
boosted_dt = AdaBoostClassifier(dt_ap, random_state=1)
boosted_dt.fit(X_train, Y_train)

Is this equivalent of pruning a decision tree?

Though they have similar goals (i.e. placing some restrictions to the model so that it doesn't grow very complex and overfit), max_depth isn't equivalent to pruning. The way pruning usually works is that go back through the tree and replace branches that do not help with leaf nodes.

  If not, how could I prune a decision tree using scikit?

You can't through scikit-learn (without altering the source code).
Quote taken from the Decision Tree documentation: Mechanisms such as pruning (not currently supported)

If you want to post-prune a tree you have to do it on your own:
You can read this excellent post detailing how to do so.


What is the best strategy to use on data with many classification labels? In general, what sort of supervised algorithms and techniques should I use on data that has the following charactersitcs:

2 potential classification labels?
3-5 potential classification labels?
6-10 potential classification labels?
10-50 potential classification labels?
50 or more potential classification labels?

My Primary Questions:

What algorithms learn most effectively at these different tiers of total possible class labels?
What algorithms generally make the best predictions with the smallest amount of data at each of these tiers?

I know at some point it makes more sense to use regression rather than a classifier. How many potential class labels would this be?

There are many factors which influence choice of classifier algorithm. Number of target classes does not generally have an influence, compared to the nature of input features.

As one example, if your input data is natural audio or image, then regardless of the number of classes, a deep convolutional neural network is very likely going to have the best performance.

  What algorithms generally make the best predictions with the smallest amount of data at each of these tiers?

There is no a priori best approach based on number of output classes. The "best predictions" vs "smallest amount of data" is also a trade-off, where simpler models will perform better than complex ones on small amounts of data, but the more complex models will cope better with larger amounts of data, and will then give you better predictions. At some point you might have enough data so that sampling more will not improve your trained models, but you need to establish that empirically.

Most algorithms allow you to explore the trade-off within them by varying hyper-parameters to make the model simpler for smaller data sets and more complex when there is more training data.

  I know at some point it makes more sense to use regression rather than a classifier. How many potential class labels would this be?

That is not strictly true. In general, the distinction between classifying and regression is a hard line. If you are classifying hand-written symbols into an alphabet for instance, it doesn't really matter if you are doing this for 10, 100 or 1000 classes, there is not a practical point at which the symbols turn from being a set of objects into a continuous space to perform regression over.

It could be true if your target class represents a range within some continuous variable (e.g. classifying an event by some of its properties into which year it occurred). But in that case the problem is inherently a regression problem to start with. In fact you may be better off training a regression algorithm in this case even for small number of target classes, and simply binning the predictions into relevant classes.

It could also be true that your target class represents a rank or sequence within an ordered set. In which case this does look more like a regression problem when you have a longer sequence. In general, if you can arrange your target classes into a meaningful sequence, then you might be able to perform some kind of ordinal regression which could be a better choice than using a classifier. However, classifying symbols/alphabets does not work this way, because the sequence in those is arbitrary.

Finally, you might be facing such a large number of classes that a single classifier model is overwhelmed and you need to approach the problem differently.

For an example of this last case, consider a classifier for images of pets. If it had three classes (cats, dogs, rabbits), then you'd clearly use standard classification approach. Even when classifying by breed - 100s of classes - then this approach still works well enough, as seen in ImageNet competitions. However, once you decide to try and detect the identity of each pet (still technically a class) you hit a problem using simple classifier techniques - in that case the structure of the solution needs more thought. One possible solution is a regression algorithm trained to extract biometric data from the image (nose length, distance between eyes, angle subtended between centre of jaw and ears) and move the classification stage into KNN based on a database of biometric data for observed individuals. This is how some face identifying algorithms work, by mapping images of faces into an easy-to-classify continuous space first (typically using a deep CNN), then using a simpler classifier that scales well across that space.


Alternatives to linear activation function in regression tasks to limit the output I want to know whether there is a way to limit the output of a regression deep model. Suppose that I want my model outputs values which are in a specified range and penalizes the outputs which are not inside the range while training. I've not seen any paper but I have two solutions for that right now but I don't know if there is any standard way or not.

My first suggested solution is to use an activation function like the following. It is linear in the specified range and has high slope out of the specified range. Although the gradient will oscillate if I use simple gradient descent, I guess it will perform well if I use second-order optimization algorithms like Adam. Any suggestions for this?

My second solution is inspired by L1/L2 regularization. I find the output of the model. If it exceeds the absolute value of the range, suppose it's symmetric, I'll add a constant big value which is far from the real output. Here We can replace it with a big constant or maybe multiply it by a constant. The second one has this property that it will be differentiable with a slope.

Does anyone have suggestions which are applicable or even these are ok or any other thing?

In regression tasks, it is customary to use linear activation function as the non-linearity of the final layer in order to estimate a function which outputs real value(s). The reason sigmoid function is used is that its output is limited to the range 0 to 1 which is a good range for specifying probability. Moreover, sigmoid is used more for classification tasks where the classes are mutually exclusive in the input o.w. softmax activation function is used. Tanh is used due to having zero mean which accelerates the training process. These are usually not applied in classification tasks these days because their differentiation saturates in their limits. Relu is a customary choice because its slope does not saturate.
Sigmoid-shape activation functions cannot be used as the last layer of the regression tasks. I want to use a kind of activation function like linear as the last layer's activation function in order to have outputs in a special range. While training I want to train the model in a way that it never tries to output a value which is outside a specified range. For doing so, I've put two suggestions that I have tried. I don't know whether there is a paper about that or not. Moreover, about the provided answer, sigmoid does not penalize the outputs, I mean does not increase them in a way that the error value increases, which are outside the range of the domain of the function. What it does is saturating the output to zero or one. What I want to do is to make a huge difference between the output of the network and the real output of the input in order to cause a high error. Consequently, the network will try to find weights which do not yield results out of the range.

Don't use that activation function shown in the question.  It doesn't do what you think it does.  Instead:

If you want the output to be in the range $[-1,1]$, you can use a sigmoid or tanh activation function in the final layer.

If you want the output to be in the range $[-100,100]$, do the same, then multiply the final output by 100 (a hardcoded constant).

If you want the output to be in the range $[-50,150]$, do the same, then multiply the final output by 100, then add 50.

Hopefully you can now see how to handle any desired range.

After some edits, I see some misconceptions in your question:

Contrary to what you write, there is no reason that the last layer of a regression model has to use a linear activation function.  It's not like there are "regression police" who are going to arrest you for using a different activation function.

You say it is customary to use a linear function at the output of a regression model.  That's not really because those models are doing regression; rather, that's more because they are solving a task where you want range of possible outputs to be $[-\infty,+\infty]$, so of course they're not going to use an activation function that restricts the range.  Now you have a new setting where you do want to restrict the output range -- so it is reasonable to use a different activation function.  You're have a non-customary regression task, so don't be surprised if this means you might benefit from an activation function that is non-customary.
Sigmoid and tanh have been used in other contexts because they do have other useful properties, but that doesn't mean they're not also useful here.  An activation function can have multiple useful properties, some of which are relevant in one setting and some of which are relevant in another.

Sigmoid (also called softmax) is often used in classification partly because in classification we want the output to be in the range $[0,1]$, so it is natural to pick an activation function that ensures that range; and partly because there are some statistical reasons to choose sigmoid (aka softmax).  The former is relevant here; the latter is not.  You're allowed to use the sigmoid function for your task for a different reason than why people use sigmoid for classification.
It's true that some sources have used tanh because of its derivatives, but that doesn't mean it's the only allowed reason to use tanh.  You're allowed to use tanh because of the way it restricts the output range rather than because of properties of its derivatives.

Contrary to what you wrote, sigmoid can be used at the last layer of regression tasks.

Ultimately, I think you are prematurely rejecting plausible approaches without a good reason for doing so.

It's also possible to experiment with approaches where you penalize outputs that are outside the range, instead of changing the last-layer activation function.  Indeed, as long as all outputs in the training set are in-range, a standard loss function (like squared loss) will already do that.  However, it's hard to say how well that will work, or whether it will work better than a clever choice of activation function for the last layer.  The only way to find out is to try it and see.


Inappropriate stemming in nltk.stem I was starting to learn stemming with nltk and a few words were quite inappropriately stemmed.
For example:- very was stemmed to 'veri', important to 'import', once to 'onc', poorly to 'poorli' , etc.

I was just thinking that during data analysis, does it cause any error or these errors can be ignored?
Or is there a better option for the same purpose

The renown algorithm for stemming is Porter stemming algorithm. Hence, you can use stemmer = nltk.stemmer.PorterStemmer() for the stemming. you can test it using stemmer.stem('poorly').

Moreover, you can see this post for more details.


has number of output layer of DNN any effect in speed of find the optimal answer of DNN? has number of output layer of DNN any effect in speed of find the optimal answer of DNN?
For instance the more episodes is needed to train a DNN when the number of outputs is more? Is it correct?

If you mean by the number of outputs is the number of classes,  then the answer is yes. Increasing the number of outputs will increase the number of parameters that you need to tune. The last set of layers in the dnn are fully connected and they contribute the most in the number of parameters, usually the sizes of these fully connected layers  depend on the number of classes. The more classes the larger these fully connected layers and thus the more parameters


How to understand image of Fourier transform? When image is transformed using 2d fourier transform we can visualize the output as:

I think I understand correctly that those lines that come from the center represent high frequencies in horizontal/vertical direction when the size value of frequency related the to length of the line.

What the those lines that seem to connect the ends of those lines comming from the center mean?
How is value of sine/cosine visible on this image?

The lines connecting the ends seem also parallel to existing lines with a similar pattern but faded a lot more.

I think they are the 2D equivalent of harmonics.

Maybe start with a generated straight line segment at a small angle, e.g. $\pi / 6$ rad.
Not end-to-end, and maybe not even centered.


Spark/Databricks: GPU does not appear to be utilized for ML regression (cross-validation, prediction) notebook I have created and attached a notebook to a GPU-enabled Databricks cluster (6.4 ML (includes Apache Spark 2.4.5, GPU, Scala 2.11), EC2 type: p2.xlarge).

I have started running the notebook that includes cells with PySpark/MLlib code for performing cross-validation and prediction using a Pipeline consisting of a VectorAssembler, MinMaxScaler, and GBTRegressor.

When I run this job it appears to be utilizing only CPU (Ganglia UI shows no GPU activity whatsoever, but plenty of CPU being used). Perhaps there is PyCpark code I need to add to my notebook and/or configuration settings for the cluster to allow for running this code with the help of my cluster's GPU?

I am new with Spark/MLlib, it's very possible that I am missing something obvious. Thanks in advance for any suggestions!

Spark itself does not use GPUs at all, so this is not surprising.

The operations it performs are at best L3 BLAS ops of moderate size, and most are small L1 operations, so generally a GPU isn't a win. It does use BLAS to accelerate those ops in hardware if a BLAS library like MKL or OpenBLAS is present.


combining trained neural nets in tensorflow I am using tensorflow to train two instances of the same neural network with two different datasets. the network itself is quite simple with an input and output layer and 6 hidden layers (each layer is a 20 meurons followed by a non-linear activation function).

I can train the network with two different datasets and that is fine. Now, what i want to do is basically create a new network which is a combination of these two trained networks. In particular, I want the input and the first 3 layers to be from one of the trained network and the last 3 layers and the output layer to be from the other network. I am very new to tensorflow and have not found a way to do this. Can someone point me to the API or some way to do this sort of hybrid networks?

It seems merging two neural networks does not make any sense. You may instead train one deep R-CNN or with just 2 different NN trained, You may classify as for a test case,if outputs of both NN agree, then output the common output, else the NN which outputs 1, its precision and NN which outputs -1, ratio of its correct negatives to total cases which it predicted negative(calculated in training), whichever is greater, output that NN's output.
Hope these articles will help you.
Build Your First Deep Learning Classifier using TensorFlow: Dog Breed Example
Deep Learning Part 3: Combining Deep Convolutional Neural Network with Recurrent Neural Network


value error of incompatible wrong dimension during training a model I was implementing a decision tree on a dataset. Before that, I wanted to transform a particular column with CountVectorizer. For this, I am using pipeline to make it simpler.
But there is an error of incompatible row dimensions.
code
# Imported the libraries....
from sklearn.feature_extraction.text import CountVectorizer as cv
from sklearn.preprocessing import OneHotEncoder as ohe
from sklearn.compose import ColumnTransformer as ct
from sklearn.pipeline import make_pipeline as mp
from sklearn.tree import DecisionTreeClassifier as dtc

transformer=ct(transformers=[('review_counts',cv(),['verified_reviews']),
                             ('variation_dummies', ohe(),['variation'])
                            ],remainder='passthrough')

pipe= mp(transformer,dtc(random_state=42))

x= data[['rating','variation','verified_reviews']].copy()
y= data.feedback

x_train,x_test,y_train,y_test= tts(x,y,test_size=0.3,random_state=42,stratify=y)
print(x_train.shape,y_train.shape)             # ((2205, 3), (2205,))

pipe.fit(x_train,y_train)                       # Error on this line

Error
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-79-a981c354b190> in <module>()
----> 1 pipe.fit(x_train,y_train)

7 frames
/usr/local/lib/python3.6/dist-packages/scipy/sparse/construct.py in bmat(blocks, format, dtype)
    584                                                     exp=brow_lengths[i],
    585                                                     got=A.shape[0]))
--> 586                     raise ValueError(msg)
    587
    588                 if bcol_lengths[j] == 0:

ValueError: blocks[0,:] has incompatible row dimensions. Got blocks[0,1].shape[0] == 2205, expected 1.

Questions

How is this error of incompatible row dimension forming?
How it can be solved?

As per the documentation, whenever the transformer expects a 1D array as input, the columns were specified as a string ("xxx"). For the transformers which expects 2D data, we need to specify the column as a list of strings (["xxx"]).

so the code below will work.

## Important: i have passed the columns a string to CV and list of columns to OHE

transformer=ct(transformers=[('review_counts',cv(),'verified_reviews'),
                             ('variation_dummies', ohe(),['variation'])
                            ],remainder='passthrough')

Credit goes to Another man who helped me on this.


Is it possible to use NEAT networks for solving video games? Sorry to start such an unspecific question, but I am slightly lost in the big topic.

My tutor proposed to chose neural networks for my final project, and we started by building a CNN for detecting hand written digits in python.

My own idea was to build another neural network that can play video games of ascending difficulty, for example pong -> snake -> megaman [with tools for extracting in game data] or driving in GTA San Andreas similar to what Sentdex did. (For GTA, I would try something different to Sentdex as I would train the neural network by reinforcement learning in artificial conditions, starting with staying in a straight lane and then moving on to curves, traffic etc.)
I made my own version of snake, which I could use for additional input and seems like a realistic minimal goal.

But now, I am not so sure which approach I should take for this. I took a look at Sethblings MarI/O and read the NEAT paper. Then, I read the book "Machine Learning in Python" explaining a lot of concepts, and realised that neural networks and especially GA's are only a small part of possible algorithms. Also, I am not sure how much Sethblings network actually thinks and how much it just randomly adapted to the level.

Do you think that the following idea is realistic?:
"I build my own neural network in python based on NEAT, then test it on video games / tasks of increasing difficulty and try to tune parameters for performance"

I could also imagine comparing NEAT to an implementation of Alexnet for the same tasks.
The whole goal is not to create the perfect NEAT but instead just make one and see how far it takes me. I have a lot of time and motivation at hand, but I don't want to exceed the boundaries of a high school final and my own possibilities.

Feedback welcome and thank you in advance

Implementing the NEAT algorithm for playing video games is quite realistic, especially considering that it is not novel, so there are extensive resources and tutorials to help you.  In fact, there are publications that will walk you through exactly what you want to do, such as this paper on using the NEAT algorithm for playing Pong.

Of course designing a self-driving car would be significantly more advanced than playing the 2D games you mentioned, but it appears that much of the work for developing AI in GTA V exists.

The person you mentioned, Sentdex, has a github page for developing GTA V code.  However, note that:

  We read frames directly from the desktop, rather than working with the game's code itself. This means it works with more games than just GTA V, and it will basically learn (well, attempt to learn...) whatever you put in front of it based on the frames as input and key presses as output.

This is a very primitive approach.

I am by no means an expert on self-driving cars, but after some Googling, deep drive appears to be the premier simulated self-driving car development platform.  If you are going to develop the GTA network, I suggest checking out it out.

As for comparing to an implementation of AlexNet:

Again, I am not an expert on self-driving cars, but I would expect an ensemble of networks to be used for this type of task, so I imagine an object detection network or semantic segmentation network to be used in conjunction with a network designed with the NEAT algorithm.

For instance, you can use a fully convolutional AlexNet model to detect objects such as pedestrians/cars/sidewalks/etc and feed that data into another network that also takes other inputs such as your current speed, wheel angle, road conditions etc that combines all this information and translates it into actions such as applying the brakes, turning etc.

AlexNet is a Convolutional Neural Network (CNN), which is primarily used in the field of vision processing.  The NEAT algorithm typically generates Multilayer Perceptrons (MLP), which are not used individually for machine vision challenges; they are different tools for different tasks.  It would be like asking, "which is better: a hammer or a screwdriver".  The answer depends on the task.

TL;DR:

Yes, using NEAT for video game AI development is a good introductory project for some of the more advanced topics in machine learning and deep learning.


NCF Recommender- The target encoded within the model input, why doesn't it overfit easily? In the recommender system NCF, the input is a batch of user-item interactions (one-hot encoded) and the output is a 0-1 score of whether the item has been bought or not:

This seems to indicate that the item input vector that the model is trained on already contains y. I understand the purpose of this, but doesn't that lead to a dangerously high chance of overfitting?

The input of the system is a given user and a given item (each respectively one-hot encoded). The output of the system is binary (picked or not).

The authors filtered the data to require each user to have picked at least 20 different items. Thus, enabling the system to learn the general preferences of a user.

The system could overfit. The system could memorize that users would only pick items they have already picked.

The system would fail on the chosen evaluation protocol - held-out last interaction. The model performs better than other recommendation systems on this evaluation protocol which is evidence that this system is not overfitting.


How deep should ones linear algebra knowledge be before starting data science? How important is linear algebra to being a data scientist? Are we talking college postgraduate level?

I think it truly depends on what you decide to specialize in. Data science is a very broad field, and you can actually work with data without knowing what eigenvalues and eigenvectors are. However, if you want to acquire a intermediate/advanced understanding of statistics or machine learning, you need at least an intermediate/advanced knowledge of linear algebra.

I suggest to take an introductory class on linear algebra on MOOC - just to have a more precise idea of what linear algebra is - and then study some other topics that you are interested in. Linear algebra is a useful tool, but it can be very boring, especially if you are an "applied" kind of guy. Moreover, I think that some concepts like eigenvalues or eigenvectors are easier to understand when seen in an applied context, e.g. principal component analysis.


Caret + RStudio: Error "Please make sure `y` is a factor or numeric value" when training I'm new to Caret and I've been trying a couple things to get the hang of things. But this error happened to me and I'm not sure why.

I've been trying to train a model with some data I got from "PimaIndiansDiabetes". These are the "x" and "y" I'm using:

  >str(pima.Datos.Train[pima.Vars.Entrada.Usadas])
  'data.frame':   615 obs. of  8 variables:
   \$ pregnant: num  6 1 8 1 3 10 2 8 4 10 ...
   \$ glucose : num  148 85 183 89 78 115 197 125 110 139 ...
   \$ pressure: num  72 66 64 66 50 0 70 96 92 80 ...
   \$ triceps : num  35 29 0 23 32 0 45 0 0 0 ...
   \$ insulin : num  0 0 0 94 88 0 543 0 0 0 ...
   \$ mass    : num  33.6 26.6 23.3 28.1 31 35.3 30.5 0 37.6 27.1 ...
   \$ pedigree: num  0.627 0.351 0.672 0.167 0.248 ...
   \$ age     : num  50 31 32 21 26 29 53 54 30 57 ...

  >str(pima.Datos.Train[pima.Var.Salida.Usada])
  'data.frame':   615 obs. of  1 variable:
   \$ diabetes: Factor w/ 2 levels "neg","pos": 2 1 2 1 2 1 2 2 1 1 ...

  >pima.modelo <- train(pima.Datos.Train[pima.Vars.Entrada.Usadas],
                        pima.Datos.Train[pima.Var.Salida.Usada],
                        method='mlp')

train returns an Error: Please make sure y is a factor or numeric value.
But as far as I know, "y" is a factor, so I'm not really sure where the Error comes from. ¿Any help with this?
Thanks in advance

Have you done any pre-processing or data manipulation before training the model?

Its hard for me to say what is the reason the error, but I have tried to do the same task and it worked whithout errors, here is my code:

library(mlbench)
library(caret)
df <- PimaIndiansDiabetes

#Using x, y arguments

modelo <- train(x = df[, 1:8] , y = df$diabetes, method='mlp')

#Using formula
modelo <- train(diabetes ~.,  data = df, method='mlp')

Both approaches valid, and produce result like this:

> modelo <- train(x = df[, 1:8] , y = df$diabetes, method='mlp')
> modelo
Multi-Layer Perceptron

768 samples
  8 predictor
  2 classes: 'neg', 'pos'

No pre-processing
Resampling: Bootstrapped (25 reps)
Summary of sample sizes: 768, 768, 768, 768, 768, 768, ...
Resampling results across tuning parameters:

  size  Accuracy   Kappa
  1     0.6545936   0.0000000000
  3     0.6416382  -0.0008031577
  5     0.6397277   0.0002662915

Accuracy was used to select the optimal model using the largest value.
The final value used for the model was size = 1.

Please note that your results might differ a little.


Change format of table How can I change the table format from this:

To This?:

Is there any way to do this in Python using pandas or NumPy?

A = np.array(['A','B','C','A','B','A','B','C','A','A','B','C','D','E'], dtype=object).reshape(-1,1)
B = np.array([1, np.nan,np.nan,2,np.nan,3,np.nan,np.nan,4,5,np.nan,np.nan,np.nan,np.nan], dtype=object).reshape(-1,1)
arr = np.hstack((A,B))

df = pd.DataFrame(arr, columns=['A','B'])
df.fillna(method='ffill',inplace=True)

cols = np.append(['num'],df['A'].unique())
data = pd.DataFrame(columns=cols)

data['num'] = pd.Series(df['B'].unique())

for i, row in data.iterrows():
    for j, column in row.iteritems():
        check = df[(df.B==i+1) & (df.A==j)]
        if len(check) != 0:
            data.loc[i,j] = j

data.fillna('',inplace=True)
data

Some engineering might be needed on boundary conditions if pattern changes with new data.


Does a CNN think things inside the filter are collocated aka dependent on each other? I am running a 1D CNN on tabular data. The rows are data that I have are not sequential, that is to say they are not part of a time series or ordered string, which is why I am not using an LSTM.
So when the filter looks at both row n and row n+1, does it learn from them as if they are spatially collocated or dependent upon each other? Is there a way to prevent this from happening?
Alternatively, is there a better way to look at 2 dimensional data (rows of features) as independent rows?

Yes, the filter will learn as if they are spatially co-located.
The main purpose of convolutions is to detect local features, where the notion of locality comes from the positions over which the filter is applied.
Some neural network building blocks that you could use are:

Position-wise dense layers: this applies a linear transformation to each of the elements. The result would be an element for each of the elements in the input. The dimensionality depends on the matrix used for the multiplication. This can be achieved also by applying a 1D convolution with a filter of length 1.

Max/avg. pooling over time: taking the maximum/average value over all elements in the input, separately for each input channel. This gives you a single element, as it collapses all the elements in one. A variation is to take the $k$ maximum elements.

Of course, you may want to exploit the relations between the elements in your input, but not in their original meaningless ordering: you could sort your inputs (from lowest to highest or vice versa). Then you could apply convolutional layers.


How to standarize feature vector with data in different scales? Let's suppose I have a dataset with numerical attributes of different types.

Let's suppose I want to employ a Neural Network for supervised classification with that dataset. For that, I need to extract feature vectors from that data.

Those feature vectors must be suitable for NNs. (should be normalized/standarized vectors...)

  As an example, our dataset consist of data from football games.

DATASET:

-------------------------------------------------------------------
| local_elo| vis_elo| local_pts | vis_pts | loc_goals | vis_goals |
-------------------------------------------------------------------
|   2820   |  3250  |     45    |    54   |    13     |     17    |
-------------------------------------------------------------------
|   4230   |  5125  |     87    |    81   |    67     |     65    |
-------------------------------------------------------------------

The feature vectors this two data points:

x_1 = [2820, 3250, 45, 54, 13, 17]
x_2 = [4230, 5125, 87, 81, 67, 65]

but they are not suitable for feed them into a Neural Network.

How could this dataset be preprocessed in order to extract feature vectors suitables for feeding them into a Neural Network?

we've to use standardization(substitute by z-scores or subtract mean divide by standard deviation) in such cases:

#find mean of each feature
mean = data.mean(axis = 0)
#subtract mean
data - = mean
#find standard deviation of each feature
std = data.std(axis = 0)
#divide by it
data /= std


Not sure how to visual customer retention vs product line count I created a customer retention grid that tracks customer counts by territory by year. The sales managers have loved it. The numbers were terrible, but now they know what they didn't know and can address it.
The follow up question that the report generated was calculating customer retention vs product line count. We have five major product lines and the sales managers want to see if there's a correlation between the number of product line purchases and retention, which I think is a great question. The assumption is that if they only buy one from one product line, the retention is low, but if we get the whole system in place and they buy 4 or 5, the retention is high.
What I'm stuck on is how to group and display the data.

Customer A buys 1 product line in year 1, then 3 in year 2, and then 2 in year 3, and then 0 in year 4.
Customer B buys 1 product line in year 1, then 1 in year 2, and then 0 in year 3, and then 0 in year 4.
Customer C buys 2 product lines in year 1, then 2 in year 2, and then 2 in year 3, and then 1 in year 4.
Customer D buys 2 product lines in year 1, zero in the rest

My initial thought is to average the counts and then graph based on that.
Is there a better / fancier / cooler way to calculate what I'm trying to achieve?

There are probably a good few options, as usual I would start with something simple (possibly similar to what you have in mind).

Simple year1 vs year2 or year2 vs year3 global comparison: proportion of customer who go from number X in year N to number Y in year N+1. A heatmap would probably work well with this kind of data, and if you want some fancy visualization to impress your boss you could try a chord diagram or a Sankey diagram. Of course you can vary the input with average of the past 2 years, etc.
Since you have data by territory and it appears to show different patterns, it might be interesting to look at something by territory. I'm not sure which value should be used as a "summary" of the trend for a territory, I can think of a few options:

simple difference of the mean number of products between year N and year N+1 (i.e. shows positive or negative trend)
some measure of how different the distributions are, i.e. how much variation from year N to year N+1
Simple Pearson correlation over all the customers between number of products in year N vs year N+1.


What is the difference between Inception v2 and Inception v3? The paper Going deeper with convolutions describes GoogleNet which contains the original inception modules:

The change to inception v2 was that they replaced the 5x5 convolutions by two successive 3x3 convolutions and applied pooling:

What is the difference between Inception v2 and Inception v3?

In the paper Batch Normalization,Sergey et al,2015. proposed Inception-v1 architecture which is a variant of the GoogleNet in the paper Going deeper with convolutions, and in the meanwhile they introduced Batch Normalization to Inception(BN-Inception).

  The main difference to the network described in (Szegedy et al.,2014)
  is that the 5x5 convolutional layers are replaced by two consecutive
  layer of 3x3 convolutions with up to 128 filters.

And in the paper Rethinking the Inception Architecture for Computer Vision, the authors proposed Inception-v2 and Inception-v3.

In the Inception-v2, they introduced Factorization(factorize convolutions into smaller convolutions) and some minor change into Inception-v1.

  Note that we have factorized the traditional 7x7 convolution into
  three 3x3 convolutions

As for Inception-v3, it is a variant of Inception-v2 which adds BN-auxiliary.

  BN auxiliary refers to the version in which the fully connected layer of the auxiliary classifier is also-normalized, not just convolutions. We are refering to the model [Inception-v2 + BN auxiliary] as Inception-v3.


Tool for labeling audio I have few thousand audio signals to label into 2 different classes and save them to numpy array for further training of models. MATLAB recently released Signal Labeler for their Signal Analyzer, that could help to label time series, but for certain reasons, I can't use it.

Is there any specific tool for analysis and labeling of time series for Python? It is not necessary to save data and labels into numpy arrays, .csv format or anything similar is suitable as well.

Will appreciate any help!

Audacity is a free and open source audio editing software, available for all common desktop operating systems. It can be used to annotate audio, by using Label Tracks. You can have a label at a certain position, or covering a selection of time.

Annotating

Open or import your audio into Audacity
Select the audio track you want to annotate.
Click Track -> Edit Labels to add a new Label track
Select where in the track you want to annotate
Insert new label using keyboard shortcut Ctrl + B

Repeat until done with the track.

Export annotations

Export the labels using File -> Export -> Export Labels

Loading annotations

To load the labels into Python, you can use this snippet of code:

import pandas

path = 'PATH/TO/FILE'
labels = pandas.read_csv(path, sep='\t', header=None,
                        names=['start', 'end', 'annotation'],
                        dtype=dict(start=float,end=float,annotation=str))


Vectorize One line text data How to vectorize one-line text data? I have used tf-idf including bigrams and trigrams but I am not able to get good results.
I have purchase order descriptions which are one-liners and I need to classify.
It is a multi-class imbalanced data and I have a small dataset to train around 700 PO descriptions. The number of classes is 7 and the class distribution is similar to exponential. One class is dominating.
My take is that TF IDF should not work since the term frequency and the IDF frequency will be very small.
Also, can we make some user-defined functions to create vectors? If yes, what should be it?

Please suggest some alternative approaches as  well.

Check my answer to this question. Nowadays there're many pretrained embedders to choose from. They'll give you fixed-size numerical vector of features. You don't even have to go DNN way, xgboost will work just fine.


How to know if my Decision tree model is good or bad? I built a decision tree model and am not sure if it is good or bad. Could you help to evaluate my model?

My code:

from sklearn.tree import DecisionTreeRegressor
from sklearn.preprocessing import OneHotEncoder

encoder = OneHotEncoder()
X_new = encoder.fit_transform(X)
#Decision tree model
model = DecisionTreeRegressor(random_state=1)

# Fit model
model.fit(X_new, y)
print(model.predict(X_new))

Result that I got:

[ 2.86       12.83333333  4.5        ...  2.9         2.925
  2.5       ]"

It appears that you are training your model and generating predictions on the same dataset (X_new). You should not attempt to evaluate your model's performance using this output - because you are applying the model to the same data you trained it on, your evaluation will be over-optimistic. You need to set a portion of your dataset aside as test data, train the model on the remainder, and then apply the model to the independent test data. You can then calculate performance metrics like MSE or MAE as mentioned in the other answers, which will give you an unbiased estimate of how your model performs on unseen data. Cross-validation is another way to get an unbiased estimate of performance, which is essentially repeating train-test splits in a way to leverage all the available data.


Do I need to engineer lagged features when creating an LSTM for time series forecasting? Long short-term memory networks are fairly complicated and I haven't completely wrapped my head around them.
It seems to me like the big gain in LSTMs for time series forecasting is the lacking necessity for lagged features: it determines on its own which lagged information is actually significant and remembers it for the next tiestep(s).
Should one still create lagged features as inputs for timesteps when training an LSTM? Like the output of the last timestep, or means and medians of the foregoing timesteps, means and medians for a specific class, distances, differences, etc.?

TL;DR

No, you don't have to include lagged variables when using an LSTM.

Long Answer

In an LSTM architecture, your neurons have not only an input gate and an output gate, but they also have a forget gate which causes them to remember values over multiple time intervals. So, you don't have to include any lagged variables in your feature set, since you can expect the LSTM to discover relevant time-depended relationships on its own.
This is not only the case for LSTM's but also for other RNN-architectures.


Positive semidefinite kernel matrix from Gower distance I have a dataframe with continuous and categorical variables and I want to obtain a kernel matrix for classification. The kernel matrix must be symmetric and positive semidefinite, so that no eigenvalue is negative. I started with Gower distance matrix for mixed data, which is not positive semidefinite. I tried to transform the Gower distance matrix into a positive semidefinite and symmetric kernel with the function D2Ksof MiRVpackage in R, with no success. I tried also to apply the approach of page 799 in Zhao, Ni et al. “Testing in Microbiome-Profiling Studies with MiRKAT, the Microbiome Regression-Based Kernel Association Test” American journal of human genetics vol. 96,5 (2015): 797-807. with no success, as well. I always obtain a indefinite kernel matrix with positive and negative eigenvalues. Any suggestion?

You may use nearPD in R to convert a matrix to its Nearest Positive Definite counterpart.


Keras - Error when using HDF5Matrix to fit the model So I define my data like this:

x_train = keras.utils.io_utils.HDF5Matrix('dataset.h5', 'x_train')
y_train = keras.utils.io_utils.HDF5Matrix('dataset.h5', 'y_train')
x_test = keras.utils.io_utils.HDF5Matrix('dataset.h5', 'x_test')
y_test = keras.utils.io_utils.HDF5Matrix('dataset.h5', 'y_test')

But then when I try to fit the model like this:

model.fit(x_train, y_train, epochs=epochs,
          validation_data=(x_test, y_test),
          callbacks=[tensorboard, modelcheckpoint],
          batch_size=batch_size, shuffle=False)

I get this error:

File "G:\CryptoForecast\cryptomodel.py", line 34, in train_model
    model.fit(x_train, y_train, epochs=epochs, validation_data=(x_test, y_test), callbacks=[tensorboard, modelcheckpoint], batch_size=batch_size, shuffle=False)
  File "C:\ProgramData\Anaconda3\envs\TradingBot\lib\site-packages\keras\engine\training.py", line 952, in fit
    batch_size=batch_size)
  File "C:\ProgramData\Anaconda3\envs\TradingBot\lib\site-packages\keras\engine\training.py", line 670, in _standardize_user_data
    'You passed: x=' + str(x))
ValueError: Please provide as model inputs either a single array or a list of arrays. You passed: x=<keras.utils.io_utils.HDF5Matrix object at 0x000002342575E4E0>

But it should be working, no?

This might be a bug in Keras. However, you might have some luck if the "validation_split" parameter is set to 0.001 as mentioned on the corresponding Github page.


What is a reasonable way to compare the improvement in accuracy? Suppose you have a classification task and accuracy is what you care about. Now an old system $s_1$ has an accuracy of $a(s_1) = 0.9$ and a new system $s_2$ has an accuracy of $a(s_2) = 0.99$. This is an absolute improvement of $a(s_2) - a(s_1) = 0.09$ percentage points and a relative improvement of $\frac{a(s_2)-a(s_1)}{a(s_1)} = \frac{0.09}{0.9} = 0.1$.

However, when you now try to get a system of $a(s_3) = 0.999$ this seems to be much more difficult, although it is only an absolute improvement of $0.009$ and a relative improvement of $0.00\overline{90}$. So neither the absolute nor the relative difference in accuracy seems to capture this well.

Is there a common other way to quantify how much better the system is?

In high accuracy regimes (>0.9), I look at the error rate -- the complement of the accuracy -- or its reciprocal; the mean time between errors. This captures the intuition that an accuracy of 0.99 is ten times as "good" as an accuracy of 0.9.


Convert Word to Semantic Prime Does anyone know of any libraries I could use to convert a word into a semantic prime, given a list of approved primes?

In other words, given a sentence or a word, I would like to find the combination of "prime" words that best match the input semantically.

A good example of this would be a function that takes in a Wikipedia article and outputs something like Simple Wikipedia - it has the same semantic content, but restricts word usage to a number of prime words (in this case, simpler words).

Another example: say my primes are ["person", "move", "tool", "nice", "building"].

Then I would like to have some function, library, or NLP algorithm that converts "hammer" into ["building", "tool"], "car" into ["moving", "tool"], and  "friend" into ["nice", "person"].

My first intuition was to use word2vec: loop through every combination of the primes and find the combination of primes who's vector most closely matches the original word. However, I suspect this will not work out very well (partially because vector addition doesn't seem to be accurate enough). Is there a more robust way to do this?

Finding a robust model for what you are looking or trying to build is quite difficult at this point of time and as per I know there are no any such existing algorithms to do this.

One approach that you can follow is make a Knowledge base for your primes containing info/word it can be related with.
To handle most of the relation you will need a word dictionary which contains all the possible related words. Making such an exhaustive dictionary is not completely possible/feasible. But if you have some domain specific work you can make a dictionary which can handle 80% of you cases. You can use wordnet/word2vec to find the most similar words of your base primes and can extrapolate those words.

Apart from these you would need a set of possible relations that the word can have. For this you can use models like Open IE to extract the Subject-Object-Relation and you can associate these relations with your primes. But for this you will require a dataset of your domain from which you can get the possible relations.


What are advantages of oversampling over changing threshold for unbalanced classes? Let's say that I have unbalanced data set that has two classes, and I am using Random Forest to make my predictions. Random forest will be biased towards the majority class, which will cause low recall and high precision on minority class predictions that I am interested in, and it might not be desirable.

One approach is to plot recall, precision and f1 score on the y-axis and threshold on the axis, and use this plot to select the proper threshold.

Another approach is to use oversampling, let's say SMOTE, for the minority class.

My questions is: What advantages would oversampling approach offer?

So far I only see disadvantages:

1) We have to be very careful, that our minority class is in one cluster, otherwise SMOTE will put points between the clusters, which is not desired at all. To avoid that, we should create different labels for different clusters, which is much more complicated.

2) We are creating extra data, that increases run time and memory usage.

3) Even after we have done oversampling, we still might not have ideal recall, threshold balance, and it feels redundant to still have to adjust the threshold.

I know that oversampling is a commonly used technique, what am I missing?

I would suggest to not use oversampling because of the disadvantages you listed above. Important one is that it increase noise, may contain biased data & may have outliers that demands another pre-processing step. I would suggest go for undersampling the majority class. But for your answer here is the link which will help you further if you are opting for oversampling


Alternatives for MultiLabelBinarizer There a lot of information on how to handle categorical variables when preprocessing data for ML classification. However, I cannot find any feedback on how to handle categorical variables, where each sample can belong to more than one label.

I'm working on bug detection classifier. I've got many features like who contributed to the source code. There are about 200 unique labels and creating so many dummy variables makes my model overfit.

So are there any alternatives for this method. Something like target-based encoders (ex. CatBoost)

Just having 200 unique labels and using MultiLabelBinarizer does not automatically mean overfitting. Overfitting is an empirical question. The number features and observations relative to the complexity of the algorithm effect the chance of overfitting.


Modelling in python and scoring in MATLAB? I have model objects either pickled object or H2O POJO. Is it possible to call those objects and do the scoring in MATLAB?

"Data.p" #yours pickle file
[data]=pickle.load(open("Data.p","rb"))


How to compute f1 in TensorFlow I have a code that computes the accuracy, but now I would like to compute the F1 score.
accuracy_1 = tf.reduce_mean(tf.cast(tf.equal(
    tf.argmax(output_1, axis=-1),
    tf.argmax(y_1, axis=-1)), tf.float32), name="accuracy_1")
accuracy_2 = tf.reduce_mean(tf.cast(tf.equal(
    tf.argmax(output_2, axis=-1),
    tf.argmax(y_2, axis=-1)), tf.float32), name="accuracy_2")

How can I compute F1 equivalent for the above code? I'm finding it difficult as I am very new to TensorFlow.

TF Addons computes the F1 score and more generally the FBeta Score


When should I NOT scale features Feature scaling can be crucially necessary when using distance-, variance- or gradient-based methods (KNN, PCA, neural networks...), because depending on the case, it can improve the quality of results or the computational effort.

In some cases (tree-based models in particular), scaling has no impact on the performance.

There are many discussions out there about when one should scale their features, and why they should do it. Apart from interpretability (which is not a problem as long as the scaling can be reverted), I'm wondering about the opposite: are there cases when scaling is a bad idea, i.e. can have a negative impact on model quality? or less importantly, on computation time?

Scaling often assumes you know the min/max or mean/standard deviation, so directly scaling features where these information is not really known, can be a bad idea.

For example, clipped signals may hide this info, so scaling them can have a negative result because you may distort its true values.

Below is an image of 1) a signal that can be scaled, and 2) a clipped signal that scaling should not be done.


Categorization of Natural Language Processing Tasks Problem

I am currently learning basics of natural language processing. I see many tasks in this area is assigning labels to each individual words in the sentence, including POS tagging, chunking, named entity recognition and semantic role labeling.

My question is

Are there other tasks that do not involve assign labels to each word in the sentence. One thing I could think of is sentiment analysis, which is assigning label to entire sentence/document or different aspects of sentence/document.
Is there some survey that is recommended to read that summarize the general tasks in natural language processing.

Basically, you need to know about the unsupervised learning tasks in NLP. For this, we mostly vectorize the input sentences using an embedding matrix.

Text Summarization:

Text Summarization not based on Neural Networks is used in a number of systems. These systems vectorize the input and then rank the sentences using a ranking algorithm ( like TextRank ) and cosine similarity based on their importance. The most important sentences are then given as output. You can read more here.

Similarity between two documents:

If you have a powerful document vectorizer like Doc2Vec, then the vectors of similar documents are similar to each other with some score. This score can be derived using cosine similarity.

  A technology blog will have a higher score of similarity with a computer science based blog rather than a holiday/tourism blog.

Such systems are useful in many use cases. Clustering algorithms have also been applied here.

Sentiment Analysis is not an unsupervised learning task. It is a classification task:

For training a sentiment analysis model, you need a dataset which consists of text and its corresponding sentiment ( categorical or binary ). Such a model could not be achieved through clustering or ranking methods. But, yes, these models use word embedding like the other models.

Some links which describe the basic tasks categorized under NLP:

https://natural-language-understanding.fandom.com/wiki/List_of_natural_language_processing_tasks
https://www.analyticsvidhya.com/blog/2017/10/essential-nlp-guide-data-scientists-top-10-nlp-tasks/
“Deep Learning for NLP: An Overview of Recent Trends” by Elvis https://link.medium.com/iItIayc0NW
https://blog.algorithmia.com/introduction-natural-language-processing-nlp/


How does a 3D, 4D,.. hyperplane look like (visualization)? I am looking at support vector machine classification algorithm.

It finds the optimal hyperplane. In linear algebra, hyperplane is a space that is one dimension lower than the ambient plane. For example, in a 2D space, the hyperplane is a 1D line. In a 3D space, the hyperplane is a 2D plane. The following image shows such examples.

I am interested to see the visualization of the hyperplane beyond 2D. For example: 3D, 4D...

As you can see from your examples a hyperplane of dimension $n$ is visualized in $n+1$ dimensional space. This goes basically back to the definition of hyperplane:

  In geometry, a hyperplane is a subspace whose dimension is one less than that of its ambient space. If a space is 3-dimensional then its hyperplanes are the 2-dimensional planes, while if the space is 2-dimensional, its hyperplanes are the 1-dimensional lines. This notion can be used in any general space in which the concept of the dimension of a subspace is defined.

Source: Wikipedia

Since we exist in 3D space (or 4D spacetime if you include time) one cannot visualize anything directly in 4D or higher. So you can't visualize a hyperplane with more than 2 dimensions directly either.


RNN: Multiple inputs per time step with categorical variables I am trying to a build RNN model to forecast daily sales for several different cities and different product segments (categorical features and multiple inputs for each day) along with numerical features such as traffic , temperature etc (single input for each day). How do I go about building a model using these features. I tried to label encode (Labelencoder) the categorical features and tried to successively train/update a RNN model feeding it with the time series of each city's product segment data but I ended up with huge prediction errors. Is there a way to one hot encode the categorical features and if I do use one hot encoding how do I go about it (what will be my output dimensions)

I would recommend using embedding in this case so that each unique category within a categorical feature is represented as a vector of floats. Then, you can concatenate all vectors coming from different categorical features, and append this long vector along with all other numerical feature values at each timestep.
Besides, if your categorical features are time-invariant (meaning not change over time, e.g. gender), then it might have some issues incorporating static feature values along with other numerical values since RNN is trying to capture the temporal information within the given sequence. But, there is a very helpful link regarding how to add non-temporal features into RNN.
Hope this helps.