Preprocessing text before use RNN I'm going to use ( RNN+Logisitic Regression ) to make sentiment analysis.

Should I do preprocessing for text like remove stop words, punctuation and extract keywords by found nouns ?

Welcome to the Data Science forum.

Yes, data preprocessing is an important aspect of sentiment analysis for better results. What sort of preprocessing to be done largely depends on the quality of your data. You'll have to explore your corpus to understand the types of variables, their functions, permissible values, and so on. Some formats including html and xml contain tags and other data structures that provide more metadata.

At a high level the sentiment analysis (using bag of words) will involve 4 steps:

Step 1: Data Assembly
Step 2: Data Processing
Step 3: Data Exploration or Visualization
Step 4: Model Building & Validation (train & test)

Lets understand different possible data preprocessing activities:

Convert text to lowercase – This is to avoid distinguish between words simply on case.
Remove Number – Numbers may or may not be relevant to our analyses. Usually it does not carry any importance in sentiment analysis
Remove Punctuation – Punctuation can provide grammatical context which supports understanding. For bag of words based sentiment analysis punctuation does not add value.
Remove English stop words – Stop words are common words found in a language. Words like for, of, are etc are common stop words.
Remove Own stop words(if required) – Along with English stop words, we could instead or in addition remove our own stop words. The choice of own stop word might depend on the domain of discourse, and might not become apparent until we’ve done some analysis.
Strip white space – Eliminate extra white spaces.
Stemming – Transforms to root word. Stemming uses an algorithm that removes common word endings for English words, such as “es”, “ed” and “’s”. For example i.e., 1) “computer” & “computers” become “comput”
Lemmatisation – transform to dictionary base form i.e., “produce” & “produced” become “produce”
Sparse terms – We are often not interested in infrequent terms in our documents. Such “sparse” terms should be removed from the document term matrix.

To give you more insight onto the steps involved, here are some example sentiment analysis using logistic regressions codes https://github.com/srom/sentiment

https://github.com/jadianes/data-science-your-way/blob/master/04-sentiment-analysis/README.md

Hope this helps.


NMT, What if we do not pass input for decoder? For transformer-based neural machine translation (NMT), take English-Chinese for example, we pass English for encoder and use decoder input(Chinese) attend to encoder output, then final output.

What if we do not pass input for decoder and consider it as a 'memory' model for translation.
Is it possible and what will happen?

It seems decoder could be removed and there only exist encoder.

Could I do translation task like text generation?
See:

https://github.com/salesforce/ctrl/blob/master/generation.py

https://einstein.ai/presentations/ctrl.pdf

The currently dominant NMT paradigm is based on an encoder-decoder architecture where the translation is generated autoregressively. This means that each token in the translation is generated conditioning not only on the source sentence but also on the previously generated tokens. The encoder generates a representation of the source sentence in one go and then the decoder takes this representation and generates the target side tokens one by one. This way, the decoder is responsible for the autoregressive part.

When you have an architecture where there is no decoder, you have non-autoregressive (NAR) NMT. It is possible to build a NAR model, but most attempts to train models directly in such a way have failed. They usually repeat the same words once and again, utterly generating garbage instead of actual translations.

Nevertheless, if NAR NMT were possible, it would lead to huge speedups ($O(1)$ instead of $O(n)$ complexity). This is currently a very active line of research. Most of the proposed attempts rely on creating some sort of intermediate (latent) representation to feed to the decoder, which then decodes non-autoregressively.

These are some of the latest articles from that area:

Non-Autoregressive Neural Machine Translation
Non-Autoregressive Machine Translation with Auxiliary Regularization
Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference using a Delta Posterior


Use of Standardizer to handle outliers? I have a dataset with 60 columns and 5K records. There are few columns which has outliers. I understand that there are multiple approach to handle outliers.

Actually I don't wish to drop the data as it is an outlier because am not sure whether it is really an outlier or a meaningful value (like how income could be skewed).

So what I did is just standardize the columns using standard scaler.

Is it okay to do this? Am assuming that by doing this standard scaling, I have retained the data as well as got rid of outliers.

I did refer couple of posts (post1, post2) but couldn't get an idea

Is this what standardization does? Can anyone suggest me is there anyother way to handle outliers without dropping the records?

Can you help?

Standardising may not be the best option.

Because they will still not be bounded (like when normalised) between -1 and 1 but be distribution dependent. What I mean is if they are outliers their standard deviation will be big for these values. In any case its not that you should rescale the values to combat these outliers. Outliers are not in the first place problematic because they are big in values, but mostly because these values lie in the cluster thats not possible/representative of future data and you will learn on it.

One solution for without dropping them is for example binning, where you say all outliers that have standard deviation bigger than two should have value "X" where X is some 99,5% quantile for example.


What does the classification report interpret? Class 1 indicates abnormal data How to interpret the report and How is precision, recall values are calculated for individual class labels. What is the significance of macro avg ? Does this report signify a good predictions by the model?

Classification report gives a perspective of your model performance. The 1st row shows the scores for class 0. The column 'support' displays how many object of class 0 were in the test set. The 2nd row provides info on the model performance for class 1.

Recall for each class is calculated as follows:
True Positives/(True Positives + False Negatives)
E.g., recall for class 1: 15/(15+27) = 0.56

Precision for each class is calculated as follows:
True Positives/(True Positives + False Positives)
E.g., precision for class 1: 15/(15+22) = 0.41

This particular classification report shows that the performance of the model is poor. Accuracy as a metric may be misleading. Although the model shows 99% accuracy, it hardly detects the object of class 1 (I suppose these are the object of interest, the abnormal data you want to detect).

One of the reasons for that may be the imbalance of the data, i.e. there are many objects of class 0 and few samples of class 1. As a result, the classifier is overfitting to class 0 to minimize error during training. Consider reading about some techniques to overcome the imbalance problem.


Importing C4.5 file to R dataframe I want to convert the input data available as C4.5 file format into a R data frame.

The data set can be downloaded from the following link
https://archive.ics.uci.edu/ml/machine-learning-databases/communities/

and is avaialble as .NAMES and .DATA files.

My approach was to convert it to a csv file using WEKA and then import the csv file into R. Could not do it as WEKA detected a error while parsing the C4.5 file.

Can some one point me in the right direction.

Thanks

URL <- 'https://archive.ics.uci.edu/ml/machine-learning-databases/communities/communities.data'
data <- read.table(URL, sep = ',',  na.strings = '?')


What does discriminator of a GAN should do? A Generative Adversarial Network (GAN) consists of two sub-networks: (1) generator and (2) discriminator.

What does a discriminator should be able to do? Or more specifically, should it be able to distinguish (classify) a real object (for example a vector) from a generated one or should it be able to distinguish a set of generated vectors from a set of real vectors?

I tend to think that the second option is correct. However, if it is the case, how do we build a neural network that classifies a set of vectors instead of a vector?

The discriminator must classify individual elements as being fake (i.e. created by the generator) or real (i.e. taken from the training dataset). The discriminator generates labels (real/fake) for each element in the batch. The loss functions are computed based on those labels.

Elements are fed to the discriminator in batches of the same type (i.e. all elements in the batch are real or all elements in the batch are fake). This is because the fake data batch is directly generated by the generator as part of the same computational graph (i.e. the output of the generator is
 directly connected to the input of the discriminator). This is so to be able to propagate gradients of the generator parameters through the discriminator.


Identify Resume Structure I am trying to build a resume parser (from PDF to JSON). After extracting text from a pdf as one long string, how would you split the string into different sections like the red lines show. Resumes have different formats and people use different labels for these sections. Is there any machine learning technique that I could look into? Thanks! .

This is one of the famous implementations for your task. It works well mostly. If you just need such a tool you can use it. However, if you want to develop your own tools you might want to analyze its structure.
It is also able to look for specific skills in the resumes as mentioned here.
As in your requirement, it accepts pdf and also doc and returns json.


Reinforcement Learning - What's the formula for the value function I'm trying to implement a value-iteration algorithm to solve a grid-world problem (I'm new to the field). The usual formula that I encounter about the value function V(s) is:
$$V(s) = R(s) + max_{a \in A} \sum_{s' \in S} T(s, a, s') V(s')$$
where $S$ is the set of states, $A$ the set of actions, $T$ the transition model

$$T(s, a, s') = P(s_{t+1} = s' | s_t = s, a_t = a)$$
and $R$ the reward function.

Since I'm working on a model-based problem, $T$ and $R$ should be known; the problem is that I don't know how to define (or compute knowing the details of the problem) $R$. If I'm in a state $s$ and take an action $a$ that would make the agent hit a boundary of the grid world, the new state $s'$ will be $s$, and the reward, as defined by the problem, is -5.

On the other hand, if I had arrived in state $s$ through another state, the reward will be -1. So basically, $R(s)$ depends on the previous state and the previous action. How do I represent that in the $V(s)$ formula?

The formula you have quoted is a bit unwieldy, precisely because the R function as defined needs to "look ahead" to all possible outcomes and their probabilities, but how to do that is not included explicitly.

There are actually a few variants of the Bellman equation that express more  or less detail. A good place to start for a truly generic version is Sutton & Barto (2nd Edition):

$$v^*(s) = \text{max}_{a \in \mathcal{A}}[\sum_{r,s'}p(r,s'|s,a)(r + v^*(s')]$$

Where $\sum_{r,s'}$ is over all possible reward and next state pairs.

The above equation changes your transition function that only handles next state, to a similar function that handles successor state and reward:

$$p(r,s'|s,a) = Pr\{ R_{t+1} = r, S_{t+1} = s' | S_{t} = s, A_{t} = a \}$$

Usually this does not increase the number of items to sum over, or add much complexity, because reward will most often associated with the transition.

The benefit is that this approach removes the need for a reward function that works with expected reward, and just works with specific rewards. Other variants are possible too, such as an expected reward function based on $(s,a)$ or $(s,a,s')$ - the difference is just a little bit of juggling with the expression so that it remains effectively the same given the subtle differences in definition for $r$.


Is there a Feature selection process for ARIMA model? I have a dataset representing sales per day for certain products. It contains 30000 observations and 6 features (target included). Since my task is to make a prediction about the number of pieces sold, I decided to use an ARIMA model (following this tutorial here).

I realized that the only preprocessing step for auto ARIMA is removing all the columns except target. (since auto ARIMA trains on the previous values).

So, my question is: is it common to not perform any feature selection and only keep the Target variable in ARIMA?

Thank you.

In some sense, it is common to do feature selection before you fit the ARIMA model, or at the very least, it is natural (in my opinion).

The problem is that there seems to be little development in automatic feature selection techniques for statistical time series models that can use exogenous variables (like ARIMA). Thus, it is not clear as to how we can do feature selection. To make things worse, auto.arima doesn't do any feature selection on exogenous variables, it just uses AICc to find the most optimal order of your model (in a stepwise fashion in its default setting). If you include exogenous variables in your model, they will always be included in all models in the selection process.

Basically, one way to do variable selection would be to try all possible combinations of exogenous variables, use auto.arima to find the "best" orders based on AICc, record this model's AICc (recall that AICc penalizes models that have large amounts of fitted parameters that do not increase the model's likelihood by a justifiable amount), and then pick the absolute best model out of all combinations of exogenous variables. Kind of a pain, and possibly very time consuming.

I hope this helps.


Does discretization of continuous features also lose information about distance? During discretization, it "squashes" nearby values into one bin, losing a little bit of information along the way.

But doesn't it also lose information about distances of features? For example, if we have height as continuous feature, we can e. g. create bins very small, small, medium, large and very large. Isn't it problem, that once we have these categories, we lose information that very small and small are closer than small and very large?

This would be even worse for predicted feature - I would assume that regression that tries to predict height number would be much more successfull than classifier that tries to predict category of height discretized, because cost function of regression can account for a distance from correct height, but cost function of classifier could only answer "correct" or "not correct".

Yet I haven't found any mention about this when I've searched about discretization.

Are my assumptions incorrect?

You are absolutely correct. Also, sometime the binning could be arbitrary as well, but sometimes it depends on how you use the insight.

Using marketing as an example, we often see age grouped into buckets like 18-24, 25-34, etc.

Would a 34 year and 35-year old behave dramatically differently? Probably not. But then, it might still make sense to use such grouping, because many ad platform targeting tools use the same bin definition. It doesn't make sense logically but practically.

Of course, it really depends on the situation and the kind of analyses you need to do. For some distribution, you might want to do a log scale before you bin, etc.

Back to your example, if you feel that height should be a continuous feature and most models can handle it, is there a particular reason why you would want to discretize it?


Nested IF Else in R - SAT/ACT test I have the following data set

df <- data.frame(student=c(1,2,3,4,5,6,7,8,9), sat=c(365,0,545,630,385,410,0,655,0), act=c(28,20,0,0,16,17,35,29,21))

student     sat    act
    1       365    28
    2         0    20
    3       545     0
    4       630     0
    5       385    16
    6       410    17
    7         0    35
    8       655    29
    9         0    21

and I'd like to create a new field with the following conditions

If there is an SAT score > 0 use SAT score
If SAT=0, then convert the ACT to an SAT score using the rubric here. (When there was a range in the SAT score, I just used the median.

ACT SAT
8   200
9   210
10  220
11  225
12  250
13  285
14  325
15  360
16  385
17  410
18  440
19  465
20  485
21  505
22  525
23  545
24  560
25  575
26  595
27  615
28  635
29  655
30  675
31  700
32  725
33  750
34  775
35  790
36  800

This is one heck of an ifelse statement. I've tried this:

df$newgrade=-ifelse(ACT=8,200, ifelse (ACT=9,210, ifelse(ACT=10,220, ifelse (ACT=11,225, ACT=12,250, ifelse(ACT=13,285, ifelse (ACT=14,325, ACT=15,D, ifelse(ACT=16,C, ifelse (ACT=17,B, ACT=18,D, ifelse(ACT=19,C, ifelse (ACT=20,B, ACT=21,D, ifelse(ACT=22,C, ifelse (ACT=23,B, ACT=24,D, ifelse(ACT=25,C, ifelse (ACT=26,B, ACT=27,D, ifelse(ACT=28,C, ifelse (ACT=29,B, ACT=30,D, ifelse(ACT=31,C, ifelse (ACT=32,B, ACT=33,D, ifelse(ACT=34,C, ifelse (ACT=35,B, ACT=36,D))))))))))))))))))))

I tried to follow the example at the bottom of this page but it didn't work.

Someone else on another board suggested:

df$newgrade<-ifelse(df$sat == 0, conversion$SAT[match(df$act, conversion$ACT)], df$sat)

but then a new issue presented itself:
If there is neither an ACT nor a SAT score. How can it put a 0 in for that group??

Thank you for any assistance you may bring.

You can indeed use the conversion table:

conversion <- read.table(text = "ACT SAT
8   200
9   210
10  220
11  225
12  250
13  285
14  325
15  360
16  385
17  410
18  440
19  465
20  485
21  505
22  525
23  545
24  560
25  575
26  595
27  615
28  635
29  655
30  675
31  700
32  725
33  750
34  775
35  790
36  800", header = TRUE)

With the help of this table and mathematical/logical operators, you can create the values:

transform(df, newgrade = (sat | act) *
            (conversion$SAT[match(df$act, conversion$ACT)] ^
               as.logical(act) * !sat) + sat)

The result:

   student sat act newgrade
1        1 365  28      365
2        2   0  20      485
3        3 545   0      545
4        4 630   0      630
5        5 385  16      385
6        6 410  17      410
7        7   0  35      790
8        8 655  29      655
9        9   0  21      505

The value in newgrade will also be 0 if both sat and act are 0.


Evaluating correlations between dichotomous/binary variables? What test? I have 4 variables, each are dichotomous, 1 = in this group & 0 = not in this group (for each of the 4 groups).
I have read mixed accounts of whether Pearson's correlation can be use used for binary variables. If not, what is a good alternative?
Would the Cramer V test work with a twoway-chi 2?

You want to be using a chisq test instead of looking at correlation. Correlation isn't well defined for categorical variables, since it relies on variance and covariance, which doesn't really make intuitive sense in the categorical (binary) case


Binning data in one of the columns of a dataframe(Using R) I have a data frame that contains random values between 0-60 (inclusive 0 & 60). These values denote months. I want to bin the data into three categories
(x<=6, 6< x <=12, x>12) and generate a new single columns which will be a factor containing 3 values (0,1,2) denoting the respective bins.

I am able to generate 3 columns one-hot encoded style but I am unable to think of a way to generate single column having 3 factors.

Use cut:

> df = data.frame(v=sample(1:60,1000,TRUE))
> df$cat = cut(df$v,c(-Inf,6,12,Inf))
> table(df$cat)

 (-Inf,6]    (6,12] (12, Inf]
       97        92       811

Also, simple R questions are better asked on StackOverflow.


Massive variation in results with tensorflow and keras I'm new to Tensorflow and Keras and I some background knowledge of how CNN's work.
I'm using a basic sequential model based on the code by https://pythonprogramming.net/convolutional-neural-network-deep-learning-python-tensorflow-keras/

I have a problem where my results variation is very big. The first time I ran the model today I got around 90% accuracy. But the runs after that were around 25% which is as good as guessing since I have four classes.

Here's my code:

tf.reset_default_graph()

batch_size = 32

logdir = os.path.join("logs", datetime.datetime.now().strftime("%Y%m%d-%H%M%S"))
tensorboard_callback = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)

X = X/255.0

model = Sequential()

model.add(Conv2D(64, (3, 3), input_shape=X.shape[1:]))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(64, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Flatten())  # this converts our 3D feature maps to 1D feature vectors
model.add(Dense(64))
model.add(Activation('relu'))

model.add(Dense(4))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy',
          optimizer='adam',
          metrics=['accuracy'])

with tf.Session() as sess:
model.fit(train_X, train_y, batch_size=batch_size, epochs=3,
validation_split=0.1, callbacks=[tensorboard_callback], validation_data=(test_X, test_y))

Am I doing something completely wrong with the model? I do have quite a small dataset, just 1640 images. But why do some runs perform so good then?

I commented previously, now some idea as an answer:

1) I think that the validation data may include "lucky shots" which match the trained patterns well in some cases. Also a small validation set can be a problem. As far as I know, validation data is the last X% of the data (in Keras). You can set shuffle=True to mix validation data or False to not mix it (as I understand). I would try this.
https://github.com/keras-team/keras/issues/597

2) An option to get a good idea of how the model performs is cross validation. If you can afford it, train 3 or 5 or 10 models with different validation data and have a look at the average accuracy. Should give you also an idea of how volatile the results are.

3) You have a small sample. Thus, adding more validation data can be a problem. However, I would also try with 20%.

4) Since you have a small sample, the NN might have trouble getting all the relevant features. Instead of training a convnet from scratch, you can use a pretrained model. This has already learned a lot of features. Given a small sample, this could be THE way to go. Note that there are a number of different pretrained models which you can try.
https://keras.io/applications/

5) Get more data (if possible).

I recently used a pretrained model in a multiclass setting. Here is my code, which is a slightly modified version of a tutorial code by F. Chollet (https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/5.3-using-a-pretrained-convnet.ipynb).

from keras.applications import VGG16
import os, datetime, statistics
import numpy as np
from keras.preprocessing.image import ImageDataGenerator
from keras.utils import to_categorical
from keras import models
from keras import layers
from keras import optimizers
from keras.layers.core import Dense, Dropout, Activation
from PIL import ImageFile
ImageFile.LOAD_TRUNCATED_IMAGES = True

###############################################
# DIR with training images
base_dir = 'C:/kerasimages'
# Number training images
ntrain = 2000
# Number validation images
nval  = 500
# Batch size
batch_size = 20
# Epochs fine tuning
ep = 600
# Epochs first step
ep_first = 50
# Number of classes (for training, output layer)
nclasses = 22
###############################################

conv_base = VGG16(weights='imagenet', include_top=False, input_shape=(150, 150, 3))
train_dir = os.path.join(base_dir, 'train')
validation_dir = os.path.join(base_dir, 'val')

datagen = ImageDataGenerator(rescale=1./255)

def extract_features(directory, sample_count):
    features = np.zeros(shape=(sample_count, 4, 4, 512))
    labels = np.zeros(shape=(sample_count))
    generator = datagen.flow_from_directory(
        directory,
        target_size=(150, 150),
        batch_size=batch_size,
        class_mode='binary')
    i = 0
    for inputs_batch, labels_batch in generator:
        features_batch = conv_base.predict(inputs_batch)
        features[i * batch_size : (i + 1) * batch_size] = features_batch
        labels[i * batch_size : (i + 1) * batch_size] = labels_batch
        i += 1
        if i * batch_size >= sample_count:
            break
    return features, labels

# Lables and features
train_features, train_labels = extract_features(train_dir, ntrain)
validation_features, validation_labels = extract_features(validation_dir, nval)
train_labels = to_categorical(train_labels)
validation_labels = to_categorical(validation_labels)
train_features = np.reshape(train_features, (ntrain, 4 * 4 * 512))
validation_features = np.reshape(validation_features, (nval, 4 * 4 * 512))

#######################################
# Model
model = models.Sequential()
model.add(conv_base)
model.add(layers.Flatten())
model.add(layers.Dense(2048, activation='relu'))#256
model.add(Dropout(0.20))
model.add(layers.Dense(1024, activation='relu'))#256
model.add(Dropout(0.20))
model.add(layers.Dense(512, activation='relu'))#256
model.add(Dropout(0.20))
model.add(layers.Dense(nclasses, activation='softmax'))
conv_base.trainable = False

#######################################
# Data generators
train_datagen = ImageDataGenerator(
      rescale=1./255,
      rotation_range=40,
      width_shift_range=0.2,
      height_shift_range=0.2,
      shear_range=0.2,
      zoom_range=0.2,
      horizontal_flip=True,
      fill_mode='nearest')

test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
        train_dir,
        target_size=(150, 150),
        batch_size=batch_size,
        class_mode='categorical')

validation_generator = test_datagen.flow_from_directory(
        validation_dir,
        target_size=(150, 150),
        batch_size=batch_size,
        class_mode='categorical')

# Model compile / fit
model.compile(loss='categorical_crossentropy',
              optimizer=optimizers.RMSprop(lr=2e-5),
              metrics=['acc'])

history = model.fit_generator(
      train_generator,
      steps_per_epoch=100,
      epochs=ep_first,
      validation_data=validation_generator,
      validation_steps=50,
      verbose=2)

#######################################
# Fine tuning
conv_base.trainable = True

set_trainable = False
for layer in conv_base.layers:
    if layer.name == 'block5_conv1':
        set_trainable = True
    if set_trainable:
        layer.trainable = True
    else:
        layer.trainable = False

model.compile(loss='categorical_crossentropy',
              optimizer=optimizers.RMSprop(lr=1e-5),
              metrics=['acc'])

history = model.fit_generator(
      train_generator,
      steps_per_epoch=100,
      epochs=ep,
      validation_data=validation_generator,
      validation_steps=50)


Combine Pandas DataFrame Rows Based on Matching Data and Boolean I have a Pandas DataFrame with sales data and columns for year, ISO week, price, quantity, and organic [boolean]. Because each row represents a different location, dates are repeated. I would like to combine rows with matching year, ISO week, and organic. Ideally, the combined row would have the average price and sum of total quantity. Any help is much appreciated!

I believe what you need is agg from pandas. You can pass a dictionary of the different aggregations you need for each column:

import pandas as pd

df = pd.DataFrame({'year':['2017','2018','2019','2019'],
                   'ISO Week':[1,2,3,3],
                  'Price':[5,10,15,20],
                   'quantity':[1,2,3,4],
                  'organic':[True, False, True, True]})

   ISO Week  Price  organic  quantity  year
0         1      5     True         1  2017
1         2     10    False         2  2018
2         3     15     True         3  2019   #<------ combine
3         3     20     True         4  2019   #<------ combine

df.groupby(['year','ISO Week','organic'], as_index=False).agg({'Price':'mean', 'quantity':'sum'})

   year  ISO Week  organic  Price  quantity
0  2017         1     True    5.0         1
1  2018         2    False   10.0         2
2  2019         3     True   17.5         7


Can we use one hot encoding instead of for loops? I have an anomaly detection model, that I run per store with a bunch of features.
I intend to run this code, everyday, per store. Now, lets say I have 8000 stores, I would imagine, I should write a for loop and iterate and create the model per store:

final is the dataframe that consists of all the stores

Something like:

for store in final['StoreNbr'].unique():
   run the model.

My input features include: Store number, Cashier, etcc (many other features). Can I use one hot encoding for the StoreNbr column and then run the model once as opposed to a for loop and running the model per storenbr?

If you want to train the data each day and have a new model per store and per day, you should not use One Hot Encoding. If you do One Hot Encoding you will get one single trained model which will learn from the data you have in all the stores. I suppose you do not want one trained model but one model for each store.


Performance metric in recommender systems with implicit feedback This paper describes a technique for making recommendations when the feedback is implicit, that is, $r_{ui}$ is only a guess. The recommendation problems boils down to the following optimisation problem:
$$
\min_{x^*, y^*}\ \ \sum_{u,i}c_{ui}(p_{ui}-x_i^Ty_i)^2+\lambda \left(\sum_u\|x_u\|^2+\|y_u\|^2 \right)
$$
where $c_{ui} = 1+\alpha r_{ui}$.

I have the following concerns:

The algorithm above approximates the binary indicator function $p_{ui}$, weighted by the term $c_{ui}$. However, most people use RMSE between $r_{ui}$ and $x_i^Ty_i$ for assessing performance. I am not sure why that would make sense.
Why using RMSE at all? With implicit feedback, it would make more sense to me to use rank-based metrics. I would imagine RMSE is more suitable in machine learning competition where the objective metric is usually RMSE.

I think your post is missing a clear question.

I'll answer your concerns anyway:

For a implicit feedback problem like the one described RMSE, indeed, doesn't make much sense. First of all it is not clear what the difference between the real value (number of interactions, e.g. watched movie 4 times) and the estimation would mean. Second, in recommender systems you usuallly want to give the user 10-20 relevant recommendations and you don't care for the rest. Hence, it makes sense to optimize your algorithm to get these top recommendations right while disregarding your performance further down in the ranking.
See my first answer. Also, the authors of your cited paper argue for a rank-based metric.


Adaboost for 3D Input data I want to do a text classification problem for which I want to train use Adaboost Classifier from sklearn using a Keras estimator. I do know how to use the Keras wrappers for using sklearn functions. So that is not the problem. I have a 3D input data (number_sentences, number_words, features) and when I checked the source code of Adaboost, it shows that X can be only 1D or 2D.

Why is that so?
Is there any other way for me to use Adaboost for 3D    input data?

Adaboost doesn't care about the order of the features, so no matter how many dimensions your samples have, you can just flatten them. Flattening them gives you a 2D X with shape (n_samples, n_features) which you can feed to the Adaboost classifier.


Choice of ML algorithm for problem Working on a school project where we have to match some users based on common interests. Assuming I have a list of inputs like this:

Name Interest1 Interest2 Interest3 Interest4 Interest5
Name Interest4 Interest6 Interest7 Interest8 Interest9
Name Interest1 Interest2 Interest4 Interest3 Interest5
Name Interest7 Interest8 Interest9 Interest11 Interest12

And another user comes

Name Interest4 Interest6 Interest13 Interest12 Interest7

Closest match for him would be user 2. If I want to take a ML approach in solving this what algorithm would fit this kind of problem?

This would be an example of Recommendation based system.
You can build a Popularity based recommendation system in this scenario. This type of algorithm works with trend, it basically uses the items which are in trend right now.

Else you can go for Collaborative recommendation system or Content based recommendation system


Why do my target labels need to begin at 0 for sparse categorical cross entropy to work? I'm following a guide here to implement image segmentation in Keras.
One thing I'm confused about are these lines:
# Ground truth labels are 1, 2, 3. Subtract one to make them 0, 1, 2:
y[j] -= 1

The ground truth targets are .png files with either 1,2 or 3 in a particular pixel position to indicate the following:
Pixel Annotations: 1: Foreground 2:Background 3: Not classified

When I remove this -1, my sparse_categorical_crossentropy values come out as nan during training.
Epoch 1/15
 25/196 [==>...........................] - ETA: 27s - loss: nan - accuracy: 0.0348 - sparse_categorical_crossentropy: nan

Why is this the case? If the possible integer values are 1, 2, 3, why would I need to alter them to begin at 0 to be correctly used?
If I include the -1, the training looks correct:
Epoch 1/15
196/196 [==============================] - 331s 2s/step - loss: 2.0959 - accuracy: 0.6280 - sparse_categorical_crossentropy: 2.0959 - val_loss: 1.9682 - val_accuracy: 0.5749 - val_sparse_categorical_crossentropy: 1.9682

Have a look at this stackoverflow answer, it seems to be caused by that the fact that your labels need to be zero indexed as the argmax function also return the index based on a zero indexed array.


User-based nearest neighbour implementation in R? I am just starting to learn to use R and am not sure how to find the best packages yet. I am looking for a package that will allow me to calculate user-based nearest neighbours as an input for a recommendation.

My original data is approximately as below:

User    Item    Rating
Anna    Apple   2
Bob Apple   4
Carlos  Apple   1
Ethan   Apple   1
Gene    Apple   2
Anna    Banana  4
Dana    Banana  3
Ethan   Banana  5
Frank   Banana  5
Bob Carrot  1
Ethan   Carrot  2
Gene    Carrot  4

I have used the package reshape2 and dcast method to generate a different datastructure where users are rows, items are columns and the rating is in the value section (similar to an Excel pivot).

User    Apple   Banana  Carrot
Anna    2   4   n/a
Bob 4   n/a 1
Carlos  1   n/a n/a
Dana    n/a 3   n/a
Ethan   1   5   2
Frank   n/a 5   n/a
Gene    2   n/a 4

Is there any R package that will allow me to compute the similarity between users based on the user vectors? Preferably I would like to use the Jaccard coefficient as my similarity metric. I tried to google for R packages with user-based nearest neighbour implementations but came up empty.

Any help of alternative suggestions will be much appreciated.

Thanks!

Maybe the best answer for Your question is recommenderlab package for R available from CRAN. Here's the link to vignette. It has CF algorithms implemented and Jaccard coef. for 0/1 user feedback case.

Only similarity(distance) computation in base stats R package there is dist function. But it's limited to few distance measures. I know that when I search for distance measures in R package called vegan claimed to have many methods implemented. But I have never used it.

IMHO most basic recommender systems algorithms are really simple. It worth spending few minutes and implement it to get a better understanding and intuition how it works.
Nevertheless, below my quick and dirty user-based kNN implementation for Your case. The code can be rewritten in better way (Adv.R is really good resource) - but this is only example for learning purpose and I do not take responsibilities for any damages :-)

require(reshape2)

ratings <- read.csv(con <- textConnection('
User,Item,Rating
Anna,Apple,2
Bob,Apple,4
Carlos,Apple,1
Ethan,Apple,1
Gene,Apple,2
Anna,Banana,4
Dana,Banana,3
Ethan,Banana,5
Frank,Banana,5
Bob,Carrot,1
Ethan,Carrot,2
Gene,Carrot,4
'), header = T)

R <- ratings %>% dcast(Item ~ User)
ProductNames <- R[,1]
R <- R[,-1] # only ratings

# tanimoto similarity (~Jaccard coefficiet)
tanimoto <- function(v1,v2) length(intersect(v1,v2))/length(union(v1,v2))

# return top-k similar users
getKNN <- function(R, i, k, sim = tanimoto) {
  similarity <- array(0,length(R))
  for(j in 1:length(R)) {
    if(i!=j) similarity[j] = sim(which(!is.na(R[,i])),which(!is.na(R[,j])))
  }
  idx <- order(similarity, decreasing = T)[1:k]
  data.frame(idx = idx, similarity = similarity[idx])
}

kNNRecommender <- function(R, k) {
  reco <- data.frame()
  for(u in 1:length(R)) {
    userItems <- which(!is.na(R[,u])) #items rated by user
    nn <- getKNN(R,u, k) # get user neighbours

    # rating prediction - wiegted by similarity
    r <- array(0, c(dim(R)[1], k))
    for(i in 1:k) {
      r[,i] = R[,nn[i,]$idx] * nn[i,]$similarity
    }
    r[is.na(r)] <- 0
    userReco <- rowSums(r) / sum(nn$similarity)
    userRecoIdx <- order(userReco, decreasing = T)

    # remove items already rated by user
    userRecoIdx <- userRecoIdx[-userItems]

    # add to recommendations result
    reco <- rbind(reco, data.frame(
      User=u,
      Item=(if(length(userRecoIdx)==0) NA else userRecoIdx),
      Prediction=(if(length(userRecoIdx)==0) NA else userReco[userRecoIdx])
    ))
  }
  reco
}

kNNRecommender(R, 2)


Predictive modeling when output affects future input Assume I have a model  which predicts the outcome of number of icecream sold in a store.
The model is trained on data for the last 5 years while keep the last year as a validation set and has produced very good results.
We now put the model into production such that the CFO can create an estimate for the upcomming year's budget. The CFO now look at the prediction for May, say 2000 ice creams, and thinks "Ooh... I was hoping for some more sale in May. I'll go 4000" thus he orders some more advertising, introduces new flavours etc. and reaches the 4000 sold ice cream at the end of May as he was hoping for.
The first of June we talk to the CFO to evalutate the model after the first 6 months, and we see that our prediction in May is off by 100%!
This spike can be explained with the increased advertising etc., and all the other days the model have done really well, but if the CFO starts tweaking the advertising, flavours etc. each day to hit the budget, how will we ever be able to test, if our model is indeed good in production/real-world? And how will we be able to re-train the model, since the first 5 years sale is without any "human influence" whereas after a year, the sale has been influenced by advertising etc., thus the spike in May is not "natural" but is due to some exogenous variable we are not able to incorporate (e.g we don't know the CFO's budget).

I am afraid that such situations are fundamentally inherent in predicting/forecasting contexts; quoting from the very recent paper by Taleb et al., On single point forecasts for fat-tailed variables (open access, para 3.7):

3.7. Forecasts can result in adjustments that make forecasts less accurate
It is obvious that if forecasts lead to adjustments, and responses
that affect the studied phenomenon, then one can no longer judge these
forecasts on their subsequent accuracy.

So, other than  communicating this clearly beforehand and reaching an agreement on how the predictions will be assessed in the presence of such adjustments, there is not much else you can do from a modelling or methodology perspective. The advice suggested further in the same paragraph quoted above:

In that sense a forecast can be a warning of the style “if you do not act, these are the costs”.

can form the basis of such a communication and agreement.


Combine K-nearest neighbor with categorical embedding I've tried a few ways to do my multi-class classification. For categorical data, I used the embedding technique with Tensorflow, which moves the entity closer with its similarity. This technique provides me with approximately 25%-30%, which is low.
For the numerical data, I used the KNN algorithm that gave me roughly 40% accuracy. I am wondering is there any way to "combine" these two techniques together to achieve a better result. For example, perhaps using the probability given by the KNN algorithm to form a layer concatenated with the embedding layer. Then, use the Dense layer to further train these data.
I've searched on the Internet. It's not the technique of ensembling, which averages the accuracy of each model. It's more like concatenating the layer together.
Any help is highly appreciated.

If I understand well, you label encode categorical variables and fed them to a neural network. If this is the case, you can try the following:

add the numerical variables
create and train an autoencoder
use the encoder part to map input to a vector space and perform k-nearest neighbor to it.

You can read the second method described in https://towardsdatascience.com/detecting-credit-card-fraud-with-autoencoders-in-python-98391cace8a3 It uses a dataset with numerical variables only, but since you label encode categorical variables it applies in your case too.


Unnormalized Log Probability - RNN I am going through the deep learning book by Goodfellow. In the RNN section I am stuck with the following:

RNN is defined like following:

And the equations are :

Now the $O^{(t)}$ above is considered as unnormalized log probability. But if this is true, then the value of $O^{(t)}$ must be negative because,

Probability is always defined as a number between 0 and 1, i.e, $P\in[0,1]$, where brackets denote closed interval. And $log(P) \le 0$ on this interval. But in the equations above, nowhere this condition that $O^{(t)} \le 0$ is explicitly enforced.

What am I missing!

You are right in a sense that it is better to be called log of unnormalized probability. This way, the quantity could be positive or negative. For example, $\text{log}(0.5) < 0$ and $\text{log}(12) > 0$ are both valid log of unnormalized probabilities. Here, in more detail:

Probability: $P(i) = e^{o_i}/\sum_{k=1}^{K}e^{o_k}$ (using softmax as mentioned in Figure 10.3 caption, and assuming $\mathbf{o}=(o_1,..,o_K)$ is the output of layer before softmax),
Unnormalized probability: $\tilde{P}(i) = e^{o_i}$, which can be larger than 1,
Log of unnormalized probability: $\text{log}\tilde{P}(i) = o_i$, which can be positive or negative.


Unigram tokenizer: how does it work? I have been trying to understand how the unigram tokenizer works since it is used in the sentencePiece tokenizer that I am planning on using, but I cannot wrap my head around it.
I tried to read the original paper, which contains so little details that it feels like it's been written explicitely not to be understood. I also read several blog posts about it but none really clarified it (one straight up admitted not undertanding it completely).
Can somebody explain it to me? I am familiar with the EM algorithm but I cannot see how it related to the loss function in order to find the subwords probabilities...

The explanation in the documentation of the Huggingface Transformers library seems more approachable:

Unigram is a subword tokenization algorithm introduced in Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (Kudo, 2018). In contrast to BPE or WordPiece, Unigram initializes its base vocabulary to a large number of symbols and progressively trims down each symbol to obtain a smaller vocabulary. The base vocabulary could for instance correspond to all pre-tokenized words and the most common substrings. Unigram is not used directly for any of the models in the transformers, but it’s used in conjunction with SentencePiece.
At each training step, the Unigram algorithm defines a loss (often defined as the log-likelihood) over the training data given the current vocabulary and a unigram language model. Then, for each symbol in the vocabulary, the algorithm computes how much the overall loss would increase if the symbol was to be removed from the vocabulary. Unigram then removes p (with p usually being 10% or 20%) percent of the symbols whose loss increase is the lowest, i.e. those symbols that least affect the overall loss over the training data. This process is repeated until the vocabulary has reached the desired size. The Unigram algorithm always keeps the base characters so that any word can be tokenized.
Because Unigram is not based on merge rules (in contrast to BPE and WordPiece), the algorithm has several ways of tokenizing new text after training. As an example, if a trained Unigram tokenizer exhibits the vocabulary:
["b", "g", "h", "n", "p", "s", "u", "ug", "un", "hug"],
"hugs" could be tokenized both as ["hug", "s"], ["h", "ug", "s"] or ["h", "u", "g", "s"]. So which one to choose? Unigram saves the probability of each token in the training corpus on top of saving the vocabulary so that the probability of each possible tokenization can be computed after training. The algorithm simply picks the most likely tokenization in practice, but also offers the possibility to sample a possible tokenization according to their probabilities.
Those probabilities are defined by the loss the tokenizer is trained on. Assuming that the training data consists of the words 1,…, and that the set of all possible tokenizations for a word  is defined as (), then the overall loss is defined as
$\mathcal{L} = -\sum_{i=1}^{N} \log \left ( \sum_{x \in S(x_{i})} p(x) \right )$

There are some parts that are not very detailed, though, for instance, how it initializes the base (seed) vocabulary to a large number of symbols".
This part is more clearly explained in the original article by the end of section 3.2:

There are several ways to prepare the seed vocabulary. The natural choice is to use the union of all characters and the most frequent substrings in the corpus. Frequent substrings can be enumerated in $O(T)$ time and $O(20T)$ space with the Enhanced Suffix Array algorithm (Nong et al., 2009), where T is the size of the corpus.

About the details of the expectation maximization algorithm used to compute probabilities, this is what happens:

[Expectation] Estimate each subword probability by the corresponding frequency counts in the vocabulary
[Maximization] Use the Viterbi algorithm to segment the corpus, returning the optimal segments.

You can check the details, together with practical examples, in this tutorial.


Using classifier's probabilities as independent variables to predict Y Suppose you have a classification task y~X with (n_samples,m_features). A colleague told me that it is correct to run r different classifiers to predict y based on X and then use the probabilities given for each classifier as a new matrix Xnew (n_samples_probabilities,r_columns) to train a new classifier y~Xnew

My questions are:

1) Is this something reasonable to apply?

2) If so, is there any mathematical support on this method?

This is called "stacked ensembling," or just "stacking."

Wikipedia

I'm not aware of any theoretical mathematical support, but intuitively it may smooth out the results of the $r$ initial models, possibly (depending on the final model) promoting models on data segments where they perform best.  Stacking consistently wins Kaggle competitions, though the added complexity might make it less attractive for certain applications.


How to find out if two datasets are close to each other? I have the following three datasets.
data_a=[0.21,0.24,0.36,0.56,0.67,0.72,0.74,0.83,0.84,0.87,0.91,0.94,0.97]
data_b=[0.13,0.21,0.27,0.34,0.36,0.45,0.49,0.65,0.66,0.90]
data_c=[0.14,0.18,0.19,0.33,0.45,0.47,0.55,0.75,0.78,0.82]

data_a is real data and the other two are the simulated ones. Here I am trying to check which one (data_b or data_c) is closest or closely resembles to data_a.
Currently I am doing it visually and with ks_2samp test (python).
Visually
I graphed the cdf of real data vs cdf of simulated data and try to see visually that which one is the closest.

Above is the cdf of data_a vs cdf of data_b

Above is the cdf of data_a vs cdf of data_c
So by visually seeing it one might can say that data_c is more closer to data_a then data_b but it is still not accurate.
Kolmogorov-Smirnov (KS) Test
Second method is the KS-test where I tested data_a with data_b as well as data_a with data_c.
>>> stats.ks_2samp(data_a,data_b)
Ks_2sampResult(statistic=0.5923076923076923, pvalue=0.02134674813035231)
>>> stats.ks_2samp(data_a,data_c)
Ks_2sampResult(statistic=0.4692307692307692, pvalue=0.11575018162481227)

From above we can see that statistic is lower in when we tested data_a with data_c so data_c should be closer data_a than data_b. I didn't consider the pvalue as it would not be appropriate to think of it as a hypotheses test and use the p-value obtained because the test is designed with the null hypothesis predetermined.
So my question here is that if I am doing this correctly and also is there any other better way to do it? Thank You

You could take an Information Theory approach by finding the lowest Kullback–Leibler divergence between the distributions. There is a KL divergence option within SciPy's entropy function.

>>> from scipy.stats import entropy

>>> p = [0.21,0.24,0.36,0.56,0.67,0.72,0.74,0.83,0.84,0.87] # Data removed to make equal sizes: [0.91,0.94,0.97]
>>> q_1 = [0.13,0.21,0.27,0.34,0.36,0.45,0.49,0.65,0.66,0.90]
>>> print(entropy(p, q_1))
0.019822015024454846

>>> q_2 =[0.14,0.18,0.19,0.33,0.45,0.47,0.55,0.75,0.78,0.82]
>>> print(entropy(p, q_2))
0.01737229446663193

The second simulated distribution is closer than the first simulated distribution to the real distribution.

If you are interested in inference, you could run many simulations and compute p-values. That process is a variation of permutation testing.


Why and how to match variables in logistic regression? I have a dataset of ~4.7K records focused on binary classification with 60 features. class 1 is of 1554 records and class 2 is of 3558 records.

Now I would like to find the risk factors that influences the outcome which is disease present or not. This is a supervised learning problem

I understand that people do matching to ensure that both the classes have similar distribution, so that the comparison results are reliable.

1) I see people usually do matching based on demographics like Age etc. Is it to infer what factors really influence the outcome if we keep Age constant. Am I right to understand this way?

2) If I put all the variables in logistic regression model, doesn't that account for confounding? Why do I have to do matching?

3) Out of 60 features, I would like to do matching based on 4 variables. How do I do this for my full dataset? Is there any python package to do this?

Can someone help me on how to do this?

It appears (if I understand you correctly) that you want to model the causal effect of some confounders $X$ on some outcome $y$ by using Logit. If this is correct, and if you know the outcome $y$, you should be fine with using just Logit, because in this case (and if you are able to contol for all relevant confounders $X$), you can identify the marginal effect of $x$ on $y$.

I guess you refer to "Propensity Score Matching" (PSM) in your question. This technique is used to "predict" cases in which you do not know for sure if some $i$ would have received treatment ($y=1$ or $y=0$) and you try to "predict" this outcome. In other words: The propensity score is the conditional (predicted) probability of receiving treatment given some $x$.

However, in case you observe $y$, and in case you don't have rasons to believ that there is a bias in $y$, you should be fine with a normal Logit. Here it would be really important for you to clarify what your problem actually is (this is not clear from your question). So if treatment ($y$) is non-random, you may think about "correcting" this bias using PSM. If not, you can go with normal Logit.

PSM is used in econometrics and related fields. So there is no off the shelf Python implementation to my best knowledge. However, you may have a look at this post. Or if you don't mind using R, have a look at this.

Anyway: PSM has it's pros and cons. It also is a little outdated since matching has made huge progress in recent years. So the most important thing for you to do is - as it appears to me - to get a good idea if you really need to "adjust" the treated/non-treated $y$ (so if you have reasons to believe that the data generating process is biased or non-random).


ML algorithm for determining CSV file header names based on content I have a large amount of CSV files, an example of which (for job titles) is listed below. The data is noisy (there are misspellings, difference in capitalisation, missing values, and they are not well-formed (some files have headers, some have not, and if headers are present, they don't always agree on name). I have gazetteers available.

...,IT Manager,...
...,Senior IT Manager,...
...,it manager,...
...,IT managre,...
...,junior IT managre,...
...,NULL,...
...,Business Consultant,...
...,Business consultent,...

I have a finite set of entity types (First Name, Last Name, Location etc.), and the task is: Given the content of this column, which entity type, if any, does it represent? (It is the first step in a processing pipeline.)

It is a form of named entity recognition [supervised learning classification] task, but all the papers I have read about named entity recognition uses conditional random fields or maximum entropy for natural language tasks, and I don't think this count as natural language.

What would be the appropriate approach/ML algorithm for such a task? How should my training data be formatted?

Yes, it might not be exactly natural Language Understanding but CRF is an excellent algorithm to train Named Entity Recognition tasks and is the stamdard model used by Stanford NLP group. You can try out their NER tagger here. If you want something that accounts for language understanding then there are certain papers who have trained recurrent neural network architectures such as LSTM and Bi-directional RNNs. Look at this paper. I must tell you that Named Entity Recognition is an incredibly hard problem and if you want to use Deep Learning architectures, it will require tremendous amount of data. I will suggest try out the Stanford NER tagger, since your data doesn't have much of a sequential nature, I am certain it will perform well. In the end here is part 1 of an excellent blog post that goes in detail about training your own NER model using the Stanford NER tool.


TensorFlow - Resume training in middle of an epoch? I have a general question regarding TensorFlow's saver function.

The saver class allows us to save a session via:

saver.save(sess, "checkpoints.ckpt")

And allows us to restore the session:

saver.restore(sess, tf.train.latest_checkpoint("checkpoints.ckpt"))

Inside the TensorFlow documentation, there is an example code (with an added epoch loop, and restore):

# Create a saver.
saver = tf.train.Saver(...variables...)
# Launch the graph and train, saving the model every 1,000 steps.
sess = tf.Session()
saver.restore(sess, tf.train.latest_checkpoint("checkpoints.ckpt"))
for epoch in xrange(25):
    for step in xrange(1000000):
        sess.run(..training_op..)
        if step % 1000 == 0:
            # Append the step number to the checkpoint name:
            saver.save(sess, 'my-model', global_step=step)

The problem is that if we stopped the training loop at epoch=15, and execute again, then if we would start at epoch=0 again, but the model is trained up to epoch=15.

Is there a way to resume from epoch=15?

The network doesn't store its training progress with respect to training data - this is not part of its state, because at any point you could decide to change what data set to feed it. You could maybe modify it so that it knew about the training data and progress, stored in some tensor somewhere, but that would be unusual. So, in order to do this, you will need to save and make use of additional data outside of the TensorFlow framework.

Probably the simplest thing to do is add the epoch number to the filename. You are already adding the current step within the epoch, so just add in the epoch multiplied:

saver.save(sess, 'my-model', global_step=epoch*1000000+step)

When you load the file, you can parse the filename to discover what epoch and step you were on and use those as the start point for the xrange functions. To make this easier to re-start from any given checkpoint, you could use argparse to allow your script to take the name of the checkpoint file you want to use.

In brief, it might look like this:

# Near top of script
import argparse
import re

# Before main logic
parser = argparse.ArgumentParser()
parser.add_argument('checkpoint')
args = parser.parse_args()

start_epoch = 0
start_step = 0
if args.checkpoint:
    saver.restore(sess, tf.train.latest_checkpoint(args.checkpoint))
    found_num = re.search(r'\d+', args.checkpoint)
    if found_num:
        checkpoint_id = int(found_num.group(0))
        start_epoch = checkpoint_id // 1000000
        start_step = checkpoint_id % 1000000

# Change to xrange:
for epoch in xrange(start_epoch, 25):
    for step in xrange(start_step, 1000000):
        sess.run(..training_op..) # etc

    # At end of epoch loop, you need to re-set steps:
    start_step = 0

You may want to reduce the number of checkpoints you are creating - as it stands you would have 25,000 checkpoint files generated by your code.

Another option would be use a single checkpoint file, and  to save and restore a Python pickle of a simple dict containing the state at the time you made the checkpoint, with a similar name.


Are pre-trained models vor CIFAR-10 / CIFAR-100 / SVHN available? I know that there are various pre-trained models available for ImageNet (e.g. VGG 16, Inception v3, Resnet 50, Xception). Is there something similar for the tiny datasets (CIFAR-10, CIFAR-100, SVHN)?

Yes, google the various "Model Zoo" (e.g. one for caffe here: https://github.com/BVLC/caffe/wiki/Model-Zoo), they host a bunch of models including  "tiny" ones.


Cost function and minimization of error In linear regression model, how can we define cost function. also after defining cost function how to minimize the error term?

Statistical programs, such as R, typicall use Least Squares estimation. It's a deterministic algorithm that makes a linear model find its optimal tuple of parameters. Because of this, you don't have to worry about the choice of a loss function.

In case you wanted to train your linear regression with a gradient descent algorithm, instead, you'd have to specify a loss function to run it. Classical loss functions for regression are: Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE).


How does a query into a huge database return with negligible latency? For example, when searching something in Google, results return nigh-instantly.

I understand that Google sorts and indexes pages with algorithms etc., but I imagine it infeasible for the results of every single possible query to be indexed (and results are personalized, which renders this even more infeasible)?

Moreover, wouldn't the hardware latency in Google's hardware be huge? Even if the data in Google were all stored in TB/s SSDs, I imagine the hardware latency to be huge, given the sheer amount of data to process.

Does MapReduce help solve this problem?

EDIT: Okay, so I understand that popular searches can be cached in memory. But what about unpopular searches? Even for the most obscure search I have conducted, I don't think the search has ever been reported to be larger than 5 seconds. How is this possible?

Well, I'm not sure if it is MapReduce that solves the problem, but it surely wouldn't be MapReduce alone to solve all these questions you raised. But here are important things to take into account, and that make it feasible to have such low latency on queries from all these TBs of data in different machines:

distributed computing: by being distributed does not mean that the indexes are simply distributed in different machines, they are actually replicated along different clusters, which allows for lots of users performing different queries with low retrieval time (yes, huge companies can afford for that much of machines);
caching: caches tremendously reduce execution time, be it for the crawling step, for the retrieval of pages, or for the ranking and exihibition of results;
lots of tweaking: all the above and very efficient algorithms/solutions can only be effective if the implementation is also efficient. There are tons of (hard coded) optimizations, such as locality of reference, compression, caching; all of them usually appliable to different parts of the processing.

Considering that, lets try to address your questions:

  but I imagine it infeasible for the results of every single possible query to be indexed

Yes, it would be, and actually is infeasible to have results for every single possible query. There is an infinite number of terms in the world (even if you assume that only terms properly spelled will be entered), and there is an exponential number of queries from these n -> inf terms (2^n). So what is done? Caching. But if there are so many queries/results, which ones to cache? Caching policies. The most frequent/popular/relevant-for-the-user queries are the ones cached.

  wouldn't the hardware latency in Google's hardware be huge? Even if the data in Google were all stored in TB/s SSDs

Nowdays, with such highly developed processors, people tend to think that every possible task that must finish within a second (or less), and that deals with so much data, must be processed by extremely powerful processors with multiple cores and lots of memory. However, the one thing ruling market is money, and the investors are not interested in wasting it. So what is done?

The preference is actually for having lots of machines, each using simple/accessible (in terms of cost) processors, which lowers the price of building up the multitude of clusters there are. And yes, it does work. The main bottleneck always boils down to disk, if you consider simple measurements of performance. But once there are so many machines, one can afford to load things up to main memory, instead of working on hard disks.

Memory cards are expensive for us, mere human beings, but they are very cheap for enterprises that buy lots of such cards at once. Since it's not costly, having much memory as needed to load indexes and keep caches at hand is not a problem. And since there are so many machines, there is no need for super fast processors, as you can direct queries to different places, and have clusters of machines responsible for attending specific geographical regions, which allows for more specialized data caching, and even better response times.

  Does MapReduce help solve this problem?

Although I don't think that using or not MapReduce is restricted information inside Google, I'm not conversant about this point. However, Google's implementation of MapReduce (which is surely not Hadoop) must have lots of optimizations, many involving the aspects discussed above. So, the architecture of MapReduce probably helps guiding how the computations are physically distributed, but there are many other points to be considered to justify such speed in querying time.

  Okay, so I understand that popular searches can be cached in memory. But what about unpopular searches?

The graph below presents a curve of how the kinds of queries occur. You can see that there are three main kinds of searches, each of them holding approximately 1/3 of the volume of queries (area below curve). The plot shows power law, and reinforces the fact that smaller queries are the most popular. The second third of queries are still feasible to process, since they hold few words. But the set of so-called obscure queries, which usually consist of non-experienced users' queries, are not a negligible part of the queries.

And there lies space for novel solutions. Since it's not just one or two queries (but one third of them), they must have relevant results. If you type in something much too obscure in a Google search, it shan't take longer to return a list of results, but will most probably show you something it inferred you'd like to say. Or it may simply state that there was no document with such terms -- or even cut down your search to 32 words (which just happened to me in a random test here).

There are dozens of appliable heuristics, which may be either to ignore some words, or to try to break the query into smaller ones, and gather the most popular results. And all these solutions can be tailored and tweaked to respect feasible waiting times of, say, less then a second? :D


Data Scientist Prerequisites: What are the essential prerequisite statistics topics? I have may years of experience with database application development mainly focused on SQL Server and Oracle.

I like to move into Data Science field and I my understanding is that I need a solid knowledge of Statistics and background - I wonder if that is correct?

I wonder what are the important statistic topics that I must know as data science prerequisites?

I am reading the following book:

Statistics in a Nutshell 2nd Edition (http://amzn.com/1449316824 )

Which of the following topics are important for data science or is there any other important topic that I must learn before I start Data Science?

Chapter 3 Inferential Statistics
Chapter 4 Descriptive Statistics and Graphic Displays
Chapter 6 The t-Test
Chapter 7 The Pearson Correlation Coefficient
Chapter 8 Introduction to Regression and ANOVA
Analysis of Variance (ANOVA)
Chapter 10 Multiple Linear Regression
Chapter 11 Logistic, Multinomial, and Polynomial Regression
Chapter 12 Factor Analysis, Cluster Analysis, and Discriminant Function Analysis

Thank you

Everything on that list is is important material you should know. The sequence the topics are introduced looks to be chosen so that the material builds on itself as you move through (this effect is never totally perfect but seems generally right in this case), so I'd pretty much read the chapters as they appear.
The material listed is the most basic of basic statistics, so it is hard to make a case to omit something.

However, while knowledge of these topics is important, it is very common (probably 'more common') for data scientists to involved in machine learning, which is a different approach to analysing data which gives better results for prediction. An overlapping term (not completely equivalent) term that is used is 'data mining' - searching on Amazon for these terms will give good results that can be used to pick a book that matches your background.

Note that statistics and machine learning are both heavily dependent on maths, especially calculus, linear algebra and probability. The book you mention has been written for people with little background in these, but to go deeper you'll need to brush up or get new knowledge in these areas.


Determining a correct ML approach I've little idea about choosing a ML approach for the following problem. It is a classification problem and there are 2 classes that are positive and negative. There are about 100k samples and samples are structured like this:
Period = 1min   Pattern = M1>S1>T1>B2>M2>S2>S3>T3>M3>B3
Period = 5min   Pattern = S1>M1>T1>B2>S2>M2>S3>T3>M3>B3
Period = 10min  Pattern = M1>T1>S1>M2>B2>S2>S3>T3>M3>B3
Period = 15min  Pattern = M1>T1>S1>B2>S3>M2>S2>T3>M3>B3
Period = 20min  Pattern = S1>M1>S3>T1>B2>M2>S2>T3>M3>B3
Period = 30min  Pattern = S1>S3>B2>M1>T1>S2>M2>T3>M3>B3
Period = 60min  Pattern = S1>B2>M1>T1>S2>M2>S3>T3>B3>M3
Period = 120min Pattern = S1>M1>T1>B2>S2>M2>T3>S3>M3>B3

This sample is classified as negative. A sample is composed of 8 periods. Within each period there is a pattern such as M1>S1>T1>B2>M2>S2>S3>T3>M3>B3. Each pattern has 10 elements and their positions are changing along samples and periods. We need to come up with a solution that could tell which period or lineup of elements are responsible for classification.
Let's say we have p1, p2, p3 positive examples and n1, n2, n3 negative examples with 1min Periods like this:
p1: M1>S1>T1>B2>M2>S2>S3>T3>M3>B3
p2: M1>S1>T1>B2>S2>M2>S3>T3>M3>B3
p3: M1>S1>T1>B2>M2>S2>S3>T3>M3>B3

n1: M1>S1>T1>B2>S2>M2>S3>T3>B3>M3
n2: M1>S1>T1>B2>M2>S2>S3>T3>B3>M3
n3: M1>S1>T1>B2>M2>S2>S3>T3>B3>M3

It could be inferred that first 4 elements M1,S1,T1, B2 are irrelevant for classification since they are all same across all samples. 5th and 6th elements are also irrelevant since they don't show same pattern along same class of samples. However, elements B3, M3 is a solid positive since M3>B3 for positive samples and B3>M3 for negative samples.
Thanks.

I think all you need is to build proper features.
For each period and element, I would build a categorical feature. That is 80 categorical features. It looks like there aren't many possible values by feature, let's say there are 3 or 4 possible values for each feature, by doing one-hot encoding you would end up with 240-320 features.
Then you can do some kind of feature selection, like Lasso, and train your model with the selected features.


How does for loops and elif work with python pandas? the following code:
df ['weight'] = 1
for feature in df.fclass:
    if feature == 'bus_stop':
        df.weight = 150
    elif feature == 'taxi':
        df.weight = 100
    else:
        df.weight = 50
df.head()

results in the following dataframe:

I expected that the value of the 'weight' column to be 100 in the second column because df.fclass == taxi. Is there something wrong with the for loop or with my 'elif' request?

Instead of just setting values for specific rows (i.e. rows where fclass is equal to the feature), you are setting the whole column (df.weight) with the value. Given that the last feature of df.class is probably bus_stop, which has a weight of 150 based on your logic, the final output will have a weight column filled with only values of 150.
There are multiple ways of getting to the output you want: (1) you can use .apply with a function containing the logic you want, (2) use the .loc method to set values for just the rows you want (so just the rows where the fclass column has a specific value) in combination with a loop, or even better, (3) use numpy.where/numpy.select to set all values at once without looping over the different features.
# method 1
def weight_function(row):
    if row["fclass"] == "bus_stop":
        return 150
    elif row["fclass"] == "taxi":
        return 100
    else:
        return 50

df["weight"] = df.apply(weight_function, axis=1)

# method 2
for feature in df["fclass"].unique():
    if feature == "bus_stop":
        df.loc[df["fclass"] == feature, "weight"] = 150
    elif feature == "taxi":
        df.loc[df["fclass"] == feature, "weight"] = 100
    else:
        df.loc[df["fclass"] == feature, "weight"] = 50

# method 3
import numpy as np

conditions = [df["fclass"] == "bus_stop", df["fclass"] == "taxi", ~df["fclass"].isin(["bus_stop", "taxi"])]
choices = [150, 100, 50]
df["weight"] = np.select(conditions, choices)


What are features for state-action pairs in RL? I read this answer:  What are features in the context of reinforcement learning?

But it only describes features for the state only in the context of cartpole, ie.   Cart Position, Cart Velocity, Pole Angle, Pole Velocity At Tip

On slide 18 here:  http://www.cs.cmu.edu/~rsalakhu/10703/Lecture_VFA.pdf

It states:

But does not give examples.  I started reading from p. 198 in Sutton's book for Value Function Approximation but also did not see examples for "features of state-action pairs" .

My best guess is for example in Cartpole-V1 (discrete action space) would be to add one more number to the tuple describing the state-action pair, ie. (Cart Position, Cart Velocity, Pole Angle, Pole Velocity At Tip, push_right) .
In the case of Cartpole I guess each state action pair could be described with a feature vector of length 3 where the final input for the tuple is either "push_left", "do_nothing", "push_right".

Would the immediate reward from taking one of the actions also be included in the tuples that form the state-action feature vector?

In the cartpole example, a state-action feature could be

$$\begin{bmatrix}
\text{Cart Position}\\
\text{Cart Velocity}\\
\text{Pole Angle}\\
\text{Pole Tip Velocity}\\
\text{Action}
\end{bmatrix}$$

where Action is either left, right, or do nothing. The reward is not part of the feature vector because reward does not describe the state of the agent; it is not an input. It is a (possibly stochastic) signal received from the environment that the agent is trying to predict/control with the use of feature vectors.


Signature verification - machine learning I have to do a project on signature verification.
My goal is having a program with two images as inupt (one is a genuine signature of a selected subject and the other one is the one that I want to test).
I have a dataset to train the algorithm with 3000 signatures: half of them are genuine and the other ones are forgery, and they are taken from 150 subjects.

My doubt regards the confrontation between the two images I use as input (the genuine one and the one I want to test).
How can I deal with it?

I think a Siamese Network is the solution you are looking for. Arxiv has a paper which precisely addresses your problem.


PyTorchs ConvTranspose2d padding parameter Im confused about what PyTorchs padding parameter does when using torch.nn.ConvTranspose2d.
The docs say that:

"The padding argument effectively adds dilation * (kernel_size - 1) -
padding amount of zero padding to both sizes of the input".

So my guess was that the dimensions of the feature maps increase when applying padding. However running test they decrease:
inp = torch.ones((1, 1, 2, 2))
conv_no_pad = nn.ConvTranspose2d(1, 1, kernel_size=(3, 3), stride=2, padding=0)
conv_pad = nn.ConvTranspose2d(1, 1, kernel_size=(3, 3), stride=2, padding=1)
print(conv_no_pad(inp).shape)
# => (1, 1, 5, 5)
print(conv_pad(inp).shape)
# => (1, 1, 3, 3)

Can somebody explain how the padding works?

As you quoted The padding argument effectively adds dilation * (kernel_size - 1) - padding, so you see the padding value is subtracted, the resulting shape becomes lower. It's a reverse (in some sense) operation to Conv2d, which means the arguments work the opposite way here. And I think this behavior is introduced to make it easier to design neural nets with symmetric architecture (like autoencoders) -- you just copy the parameters of kernel size, stride and padding from the corresponding Conv2d layer and get an operation which preserves the input shape of an image.


How to combine two different machine learning models, to get the combined result? To further explain my question, I will explain my use-case. Say I have a model which is trained for how good/bad a food is for obesity based on its nutrition facts. And another model for, say hypertension. I wish to combine these models to be able to predict food which are good for a person suffering from both obesity and hypertension.
I do not wish to retrain a new model for both cases as eventually I will add more diseases and do not wish to train for each combination of diseases.
Ideally I would like to be able to combine different models (eg Logistic Regression, SVM) but as a stepping stone, I was looking to combine two models of the same type.
I am not sure if Stacking would be appropriate in this case.

Train another XGB model on second layer. Your first layer contains all the individual model and in the second layer you will train another model on top of all of them. XGB is good for this task.


Is there an R tutorial of using LSTM for multivariate time series forecasting? There is a great blog post about how to use keras stateful LSTM in R to forecast sunspots. I applied it to financial ts data sets, and the problem I am encountering is that some of the forecasting numbers are way off to a degree that are unreasonable.

Therefore, I am wondering if there is an R tutorial of using LSTM for multivariate times series forecasting? I'd like to include variables like opening and closing price because I think that will "normalize" the forecasting values. I found a few tutorials in Python but I have limited experience with it. Maybe it's a time to pick up Python? Any advice or suggestions will be appreciated!

I like the Jena data example. In multivariate settings, you only need to generate lookbacks over all X.

https://blogs.rstudio.com/tensorflow/posts/2017-12-20-time-series-forecasting-with-recurrent-neural-networks/


how to adjusting already built ML predictive model How can I continue machine learning model after predicting results?
What I mean by that is that I built a model for my 1 million records dataset, this model took around 1 day to get built.
I extracted the model results using Python and now I have a (function) that I can feed it with my features and it gives me a prediction results
but with time my dataset has become 1.5 million records.
I do not want to redo the whole thing all over again from scratch.
Is there any way I continue of top of thf first model I built ( the one with 1 million records) so the new model take less time to adjust it based on the new 0.5 million records compare to re building everything from scratch for 1.5 million records.
P.S. I am asking for all algorithms, if there is anyway to do this for any algorithm that would be good to know which ones are these

This depends on your model type:
Classical using ensemble/stacked models:
If you are using classical machine learning, you could use your old model built on the previous 1 million records, and create a new model on the most recent 500k records and then combine the predictions in an ensemble or stacked approach.
References for Ensemble and Stacking:
https://machinelearningmastery.com/stacking-ensemble-machine-learning-with-python/
Video Reference: https://www.youtube.com/watch?v=Un9zObFjBH0
AI/NN using transfer learning:
If your are using a NN (neural network) model, you can use the idea of transfer learning. Save your model built on the first 1 million records, then add it as an initial layer to a new NN for analyzing the new data. You can then save the new NN and use it in the next round.
Reference: https://machinelearningmastery.com/transfer-learning-for-deep-learning/
Video Reference: https://www.youtube.com/watch?v=yofjFQddwHE
General guidelines:
If you need to do this updating process many times, you can create a new model on n number of records, drop the oldest data/model off once your new dataset reaches a minimum, and predict only on the last x number of models. n and x are adjusted based on your data, flexibility and need for real-time predictions. If the data is changing over time, then it would be better to only use the latest data, or weight the older data lower and the newer data higher.
Here is a good definition of transfer learning: "Transfer learning is a machine learning method where a model developed for a task is reused as the starting point for a model on a second task."


Why is learning rate causing my neural network's weights to skyrocket? I am using tensorflow to write simple neural networks for a bit of research and I have had many problems with 'nan' weights while training. I tried many different solutions like changing the optimizer, changing the loss, the data size, etc. but with no avail. Finally, I noticed that a change in the learning rate made an unbelievable difference in my weights.

Using a learning rate of .001 (which I thought was pretty conservative), the minimize function would actually exponentially raise the loss. After one epoch the loss could jump from a number in the thousands to a trillion and then to infinity ('nan'). When I lowered the learning rate to .0001, everything worked fine.

1) Why does a single order of magnitude have such an effect?

2) Why does the minimize function literally perform the opposite of its function and maximize the loss? It seems to me that that shouldn't occur, no matter the learning rate.

You might find Chapter 8 of Deep Learning helpful. In it, the authors discuss training of neural network models. It's very intricate, so I'm not surprised you're having difficulties.

One possibility (besides user error) is that your problem is highly ill-conditioned. Gradient descent methods use only the first derivative (gradient) information when computing an update. This can cause problems when the second derivative (the Hessian) is ill-conditioned.

Quoting from the authors:

  Some challenges arise even when optimizing convex functions. Of these, the most prominent is ill-conditioning of the Hessian matrix $H$. This is a very general problem in most numerical optimization, convex or otherwise, and is described in more detail in section 4.3.1.

  The ill-conditioning problem is generally believed to be present in neural network training problems. Ill-conditioning can manifest by causing SGD to get “stuck” in the sense that even very small steps increase the cost function. [my emphasis added]

The authors provide a simple derivation to show that this can be the case. Using gradient descent, the cost function should change (to second order) by

\begin{equation}
\frac{\varepsilon^2}{2} g^{T} H g - \varepsilon g^{T} g
\end{equation}

where $g$ is the gradient, $H$ is the Hessian, and $\varepsilon$ is the learning rate. Clearly, if the second derivatives are large, then the first term can swamp the second, and the cost function will increase, not decrease. Since the first and second terms scale differently with $\varepsilon$, one way to alleviate this problem is to reduce $\varepsilon$ (although, of course, this can result in learning too slowly).


Faster-RCNN how anchor work with slider in RPN layer? I am trying to understand the whole Faster-RCNN,

From  https://www.quora.com/How-does-the-region-proposal-network-RPN-in-Faster-R-CNN-work

  Then a sliding window is run spatially on these feature maps. The size of sliding window is n×n (here 3×3). For each sliding window, a set of 9 anchors are generated which all have the same center (xa,ya)(xa,ya) but with 3 different aspect ratios and 3 different scales as shown below. Note that all these coordinates are computed with respect to the original image.

It is much more clear than other articles for my opion, but still hard to understand how the feature map generate.

I saw another flow pics :

The problem, I write below steps for example:

If input is 600x1000x3 pic
Through VGG16 convnet , layer 13 output feature map is 40x60x512
Use a 3x3 sliding window, generate 1x1x512 feature map ???

Here, how 3x3 sliding window use a set of 9 anchors ???

Sorry, I am really new to object detection and image proccessing.

I only have a little understand about the steps, I known 9 anchor shapes(not the real anchor) are used to generate a lot of anchors(2400*9 in this case).

I can only imagine that use 9 anchor shape to slide the original image to get the all IoU . I don't understand how to use 3x3 sliding window in conv feature map here.

I know how anchors be selected,  2400*9 -> ignore cross-boundary -> 6000 -> apply NMS -> 2000 ,  in each minibatch, it randomly choose 512 anchors from 2000.

What I can't understand is 3x3 slide with 9 anchor shape . Because from original paper, anchors with is 16, height from 11 to 273 . I don't think it use the 13 layer conv output feature map to calculate IoU . Anchor must be apply in original image,  so what is the 3x3 sliding window doing ??

For each anchor you find an IoU with the object in the picture and set 1 if IoUexceeds the threshold and 0 if it is below a lower tjreshold(e.g. 0.3). If it's a hit a bbox offset is calculated, distance between prediction and true bbox. Hence there are two loss fumctions: object/bg and bbox regression


Adapting Neural Network to new domain without labels Is there an approach for the following problem:

Lets say, I trained a neural network on a big dataset for categorizing different fruits in $k$ classes. Afterwards I got a nice model, which performs very well.

Now I want to use the model for categorizing fruits in the corresponding $k$ classes, as it was planned beforehand. Unfortunately the fruits I want to categorize now are all not ripe yet, but my training set consisted only of ripe fruits. Furthermore I have some pictures of these not ripe fruits, but no labels.

How can I adapt my neural network to these slightly different domain with my pictures of not ripe fruits (and no labels!). Performance on the old task does not matter. The only thing I want, is categorizing not ripe fruits.

My only Idea now is to use virtual adversarial training (VAT) for the unlabeled pictures.

I think those are one of the most cited papers:

https://arxiv.org/pdf/1409.7495.pdf
http://www.jmlr.org/papers/volume17/15-239/15-239.pdf
http://openaccess.thecvf.com/content_cvpr_2017/paper/Tzeng_Adversarial_Discriminative_Domain_CVPR_2017_paper.pdf
http://openaccess.thecvf.com/content_cvpr_2017/papers/Bousmalis_Unsupervised_Pixel-Level_Domain_CVPR_2017_paper.pdf


deep learning for non-image non-NLP tasks? So far there are many interesting applications for deep learning in computer vision or natural language processing.

How is it in other more traditional fields?
For example, I have traditional socio-demographic variables plus maybe a lot of lab measurements and want to predict a certain disease. Would this be a deep learning application if I have lots of observations?
How would I construct a network here, I think all the fancy layers (convolutional etc.) are not really necessary?! Just make it deep?

On my specific data set, I tried some common machine learning algorithms like random forests, gbm etc with mixed results regarding accuracy.
I have limited deep learning experience with image recognition.

Yes you can use deep learning techniques to process non-image data. However, other model classes are still very competitive with neural networks outside of signal-processing and related tasks.

To use deep learning approaches on non-signal/non-sequence data, typically you use a simple feed-forward multi-layer network. No need for convolutional layers or pooling layers. The best architecture other than that needs to be explored with cross-validation, and can be time-consuming to discover as deep NNs take a lot of computation to train.

In my experience attempting to use deep(-ish, typically ~ 5 layers) neural networks in Kaggle competitions:

Dropout is still highly effective for regularisation and improving accuracy
Input normalisation - usually to mean 0, standard deviaton 1, is important
Hidden layer activation functions can make a difference. Although ReLU reduces some problems with vanishing gradients, in my experience it is less robust with non-signal data and you will want some other form. If you have only a few layers, then sigmoid or tanh still work OK. Otherwise, look into leaky ReLU, PReLU, ELU and other ReLU variants that attempt to patch its problems with "dead" neurons.
Make use of optimisers designed for deep learning, such as Adam, Adagrad or RMSProp
Use a weight initialisation approach that works with deep learning, such as Glorot.
Consider using Batch Normalisation layers. Not something I have much experience with, but I have seen other people do well with this approach.

Despite all this, XGBoost can routinely and easily beat deep NNs with minimal tuning and training effort in comparison (depending of course on the problem and the data you have). If accuracy is everything to you though, it is possible - although not guaranteed - that an ensemble of deep NNs and other model such as XGBoost will perform better than either singly.


regarding the inclusion of missclassified data in final training I find it best to ask my question in terms of cross-validation. Here it goes:
Suppose a binary classification problem, for which cross-validation has been applied for a certain learning algorithm. Let's say that both the CV train error and CV test error are  at 90% accuracy, indicating a good fit. Since this performance is acceptable for our problem, we combine the training and validation set into a final full dataset, and train the final model. For the final dataset, only training error is available, which suppose for our example will be 92%.
Now, for the question: Knowing that the final model has achieved a 92% accuracy, does it serve any purpose to keep the 8% of missclasified examples in the final dataset? Since these examples can't be learned, why not remove them and retrain the final model with only the 92% of the data that can be learned?
Notes

To the best of my knowledge, the  aforementioned removal of 8% in the above example is not a standard practice in modelling. Yet, I wonder what is the value of keeping examples that are not learned.
For completion, assume there is also another independent test set, to evaluate the final model.

Since these examples can't be learned, why not remove them and retrain the final model with only the 92% of the data that can be learned?

In general, I think this is a bad idea for the reasons below. This being said, the only sure way to know with a particular dataset is to experiment.

This would modify the distribution of the data. If the errors tend to happen more often with a particular class or a specific combination of features (this is quite likely), these cases would not be seen anymore by the model. Sometimes modifying the distribution can lead to better performance (e.g. when using resampling), but it can also do the opposite. So this is a bias with unknown effects on the performance of the model.
A more serious problem is that these error cases are likely to be important for the model. Typically errors happen with borderline cases, and these cases are usually very important for the model to learn the optimal way to separate the classes. By removing them it might be easier for the model to separate the classes during training, but that's not a good thing since it won't have all the information it needs in order to minimize errors. As a consequence it's likely that it won't find the right optimum and therefore will make more errors.


How are weight updates handled in Batch Gradient Descent vs SGD? My current understanding is that in SGD, after each data sample, the loss is used to update each weight.
Ex: With 1000 samples and a network with 10 weights, there will be 10,000 individual weight updates per epoch.

In Gradient Descent and Batch Gradient Descent, how are these updates deferred over multiple data samples? What is being stored at each sample, that can be applied at the end of the batch? Is the loss at each sample averaged over the batch?

What is being stored at each sample, that can be applied at the end of the batch

You store the prediction and error and then calculate the average after each cycle.

GD - Update after all rows. It will give a correct direction to the weight update but will be very slow

SGD - Update after each row. It will be fast but the direction may swing

Batch - Update after a specific count. This approach gives a balanced solution to the two other approaches.


how to handle outliers for clustering algorithms? I am wondering what's the best way to handle outliers when using non-supervised clustering algorithms?

If you have outliers, the best way is to use a clustering algorithm that can handle them.

For example DBSCAN clustering is robust against outliers when you choose minpts large enough. Don't use k-means: the squared error approach is sensitive to outliers. But there are variants such as k-means-- for handling outliers.


Multi-Output Regression with Keras I am trying to do a multi output regression using tensorflow. I have got a dataset in Excel which includes a column of input points and 2 columns of output.

I coverted all numbers to numpy objects. And I am trying to do a basic regression but accuracy is always 1.0, I also want to draw graph but dunno where to start. Could Anyone please help? My code is here:

import os
import numpy as np
import tensorflow as tf
import pandas as pd
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from keras.models import Sequential
from keras.layers import Dense
from keras.activations import relu

training_data = pd.read_excel(r'C:\Users\lenovo\Desktop\Yeni klasör\training_data.xlsx',sheet_name="i1-o2")
training_data_X = training_data['i1']
traindataX = np.array(training_data_X)
training_data_Y = training_data[['o1','o2']]
traindataY = np.array(training_data_Y)
testing_data = data = pd.read_excel(r'C:\Users\lenovo\Desktop\Yeni klasör\testing_data.xlsx',sheet_name="i1-o2")
testing_data_X = testing_data['i1']
testing_data_Y = testing_data[['o1','o2']]
testdataX = np.array(testing_data_X)
testdataY = np.array(testing_data_Y)

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(128,activation=tf.nn.tanh))
model.add(tf.keras.layers.Dense(128,activation=tf.nn.tanh))
model.add(tf.keras.layers.Dense(2))

model.compile(optimizer='adam',
              loss='mean_squared_error',
              metrics=['accuracy'])
model.fit(traindataX,traindataY,epochs=500)

val_loss,val_acc = model.evaluate(testdataX,testdataY)
print(val_loss,val_acc)

You are in a regression setting, where accuracy is meaningless (it is meaningful only in classification settings).

Simply remove the metrics=['accuracy'] argument from your model compilation, so that model.evaluate returns the loss only (discard also val_acc).

See the SO thread What function defines accuracy in Keras when the loss is mean squared error (MSE)? for more details.


Feature engineering for time series (audio signals) I have a task to perform classification of audio signals using any suitable algorimth. After some research I found out, that CNN from this paper shows promising results. However, it still needs to be improved. Current data is scaled along each sample (that is each audio signals has zero mean and the same variance(i.e. 1)), and that leads to the loss of information about amplitude and variance of signal. Is there a way to append this information(i.e. if I compute it separately before scaling) to current signal to make CNN use it?

Thanks in advance for any help!

As of 2019, the most common approach is to convert the raw audio waveform to a time-frequency representation ("spectogram"). The most commonly used spectrogram for general audio is log-scaled mel-spectrograms.
This allows to use a 2D CNN, very similar to what one would use for images.


Can we change the structure of the feature-extraction layers of deep networks architectures? I want to know if we can change the structure of the feature-extraction layers in the deep networks architectures, for example can we add more Rectified Linear Unit (ReLU) activation function or is it the same sequence that we should respect ?

You can change whatever you like!

The benefits will depend on your data and what exactly you are comparing to. As you didn't say exactly what sequence you have as your starting point, I can't compare anything exactly, but provide a simple outline.

Generally, you follow the following recipe:

Input layer: perform any reshaping or normalisation of your data
Multiplication layer: either fully-connected layer, a convolutional layer or something else
Non-linearity: relu, tanh - gives the network its power to fit non-linear functions
regularisation layer: batchnorm, dropout - help models converge and prevent overfitting

This scheme is then repeated as many times as you feel is necessary.

Doing something like applying several ReLU layers, one after another, definitely doesn't make sense, as it would actually change anything! A ReLU simply filters out negative values, setting them to zero. Applying it a second time would therefore have no effect.