What do we visualize in showing a VAE latent space? I am trying to wrap my head around VAE's and have trouble understanding what is being visualized when people make scatter plots of the latent space. I think I understand the bottleneck concept; we go from $N$ input dimensions to $H$ hidden dimensions to a $Z$ dimensional Gaussian with $Z$ mean values, and $Z$ variance values. For example here (which is based off the official PyTorch VAE example), $N=784, H=400$ and $Z=20$.

When people make 2D scatter plots what do they actually plot? In the above example the bottleneck layer is 20 dimensional, which means there are 40 features (counting both $\mu$ and $\sigma$). Do people do PCA or tSNE or something on this? Even if $Z=2$ there is still four features so I don't understand how the scatter plot showing clustering, say in MNIST, is being made.

When people make 2D scatter plots what do they actually plot?

First case: when we want to get an embedding for specific inputs:

We either

Feed a hand-written character "9" to VAE, receive a 20 dimensional "mean" vector, then embed it into 2D dimension using t-SNE, and finally plot it with label "9" or the actual image next to the point, or
We use 2D mean vectors and plot directly without using t-SNE.

Note that "variance" vector is not used for embedding. However, its size can be used to show the degree of uncertainty. For example a clear "9" would have less variance than a hastily written "9" which is close to "0".

Second case: when we want to plot a random sample of z space:

We select random values of z, which effectively bypasses sampling from mean and variance vectors,

sample = Variable(torch.randn(64, ZDIMS))

Then, we feed those z's to decoder, and receive images,

sample = model.decode(sample).cpu()

Finally, we embed z's into 2D dimension using t-SNE, or use 2D dimension for z and plot directly.

Here is an illustration for the second case (drawn by the one and only paint):

As you see, the mean and variances are completely bypassed, we directly give the random z's to decoder.

The referenced article says  the same thing, but less obvious:

  Below you see 64 random samples of a two-dimensional latent space of MNIST digits that I made with the example below, with ZDIMS=2

and

  VAE has learned a 20-dimensional normal distribution for any input digit

ZDIMS = 20
...
self.fc21 = nn.Linear(400, ZDIMS)  # mu layer
self.fc22 = nn.Linear(400, ZDIMS)  # logvariance layer

which means it only refers to the z vector, bypassing mean and variance vectors.


How can I create a pixel labelled image for Semantic Segmentation? I am following the Semantic Segmentation Examples tutorial by MathWorks.

I understand that I can load pixel labeled images

pxDir = fullfile(dataDir,'buildingPixelLabels');

Define the class names.

classNames = ["sky" "grass" "building" "sidewalk"];

Define the label ID for each class name.

pixelLabelID = [1 2 3 4];

and create a pixelLabelDatastore.

pxds = pixelLabelDatastore(pxDir,classNames,pixelLabelID);

But, how do I create a custom pixel labelled image where every pixel value represents the categorical label of that pixel?

I would then proceed by writing:

pxDir = fullfile(dataDir,'myCustomPixelLabels');

If I understood correctly, imageDatastore holds the actual image and not the pixel labels for that image.

EDIT:

On my system pxDir points to 'C:\Program Files\MATLAB\R2017a\toolbox\vision\visiondata\buildingPixelLabels'. Since I am on Matlab2017a so this example is not included by default, and I cannot compare or view the file to get a better understanding of what I need to do to reproduce this example.

The answer can be found here.

Go to the APPS tab, and search for Image Labeler, or Training Image Labeler

Click on Add Images to add your training images.

Click on Add ROI Labels to add class names for the regions of interest.

Proceed to select the regions of interest manually from the uploaded images.


Leave - one - out - Cross Validation KNN R I have a dataset and I divided it into test data and train data. Can anyone suggest how to perform LOOCV for KNN regression? Is there any library?

val(train.data,train.label,K,numfold)

Thanks in advance

To learn how its done, simplest way is to give a try on sklearn and then try to implement the same by yourself-

https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.LeaveOneOut.html

and a very basic level explanation  here-
https://medium.com/@svanillasun/how-to-deal-with-cross-validation-based-on-knn-algorithm-compute-auc-based-on-naive-bayes-ff4b8284cff4


Training binary classifier on only one data point ( Theoritical question) Say, I'm training a binary classifier to classify Dog vs Cat. Now, say I train my model only on one imagee ( cat). Now I mirror this cat image that I used to train my model. Now on the mirror image I want to make predciton.
Question : What will my model predict ?
does it predict cat, or will it randomly predict either cat or dog
I think it will predict cat beacuse the parameters have been trained only for cat, so irrespective of any input data, the parameters will support prediction for cat. Am I correct ? is my reasoning correct?

It may predict either a cat or a dog depending on your descriptor for cats and dogs, and on your learning algorithm. This is illustrated on the picture below. If the mirrored cat happens to be on the same side of the separating line as the training cat, then it will be classified as a cat. Otherwise, it will be classified as a dog.


How to reconsider MSE metric when doing target engineering through a logarithm? I am a user of Keras with Python, if I know some mathematics I am not very good about algebraic transformation.
The context is the following.
I am trying to reproduce the performances from a paper. In this paper, each individual has his own model: a neural network with the pattern 8-5-7-1. All these models reached a MSE of 100.
In my work, I do not use the same variables, so I found a different pattern.
I considered 3 individuals (3 models). I have remarked models (of the same pattern) learn better when the skewness of the target is lower. /!\ The target is not transformed at all at this moment.
In concrete terms:

Model 1: skewness 0.41 manage to learn until 100 MSE (reached the goal)
Model 2: skewness 0.66 manage to learn until 200 MSE not less (do not reached the goal)
Model 3: skewness 0.31 manage to learn until 80 MSE (beyond the reached goal)

So this is why I am thinking about making a target engineering by using the logarithm of any base (while it reduces the skewness it is ok). Nevertheless, it transforms the reached metric: 100 MSE as maximum threshold. And I am pretty sure I cannot just compute log(MSE).
I searched the formula, of course: $ MSE = \frac{1}{n}\sum_{i=1}^{n}(y_{i}-\bar{y}_i)^{2} $
But I am not able with my limited skills to reconsider it correctly. I need help.
So MSE is dependent of the n observations of the dataset, more the logarithm transformation on y. How can I implement a function that will allow to transform the new form of MSE obtained through a logarithmic transformation in order to compare it with the previous goal: 100 MSE on a non-transformed target ?

You have a $Y_{true}$
Then you calculated $Y^{log}_{true}$
So, your predictions are $Y^{log}_{pred}$
Then you can calculate the $MSE^{log}$ but it can't be compared with 100 MSE.
So, you can do a - $base^{MSE^{log}}$
to make it comparabale to 100 MSE.
A better intuitive approach will be to convert back your $Y^{log}_{pred}$
 i.e. $base^{Y^{log}_{pred}}$
Then calculate the MSE
The inverse of log is just about using power to the base
import numpy as np
input = np.array([100, 1000])
log_out = np.log10(input)

input_re = 10**(log_out)
input_re


How to read contents of a CSV file inside zip file using spark (python) I have multiple zip files containing two types of files(A.csv & B.csv)

/data/jan.zip --> contains A.csv & B.csv
/data/feb.zip --> contains A.csv & B.csv

I want to read the contents of all the A.csv files inside all the zip files using pyspark.

 textFile = sc.textFile("hdfs://<HDFS loc>/data/*.zip")

Can someone tell me how to get the contents of A.csv files into an RDD?

You need to ensure the package spark-csv is loaded; e.g., by invoking the spark-shell with the flag --packages com.databricks:spark-csv_2.11:1.4.0. After that you can use sc.textFile as you did, or sqlContext.read.format("csv").load. You might need to use csv.gz instead of just zip; I don't know, I haven't tried.


In datasets, why don't we represent nominal values that are part of a scale with numbers? I am trying to pre-process a small dataset. I don't understand why I am not supposed to do the thing I explained below:

For example, say we have an attribute that describes the temperature of the weather in a set of 3 nominal values: Hot, mild and cold. I understand these definitions may have derived from numerical values while summarising.

But why would we summarise such values that are on a scale, and lose the scale in the process?

Would it not help to have the algorithm(any classification algorithm) realise that the difference between hot and cold is twofold of the difference between hot and mild by representing hot, mild and cold as integers 1, 2 and 3 respectively?

Nice and necessary question to be clear when doing machine learning preprocessing, there are two points here to take into account:

depending on the learning algorithm, you may need to convert your categorical data into a numeric format; e.g. decission trees do not need it and can handle categorical data, whereas another regression algos do need numbers as input
in case you have to convert data into numbers, you have two possibilities:

integer encoding: this is the case you are correctly describing; with this, you have no problem with just changing the labels by integers keeping the ordinal order (in fact it is better so the algo can learn the natural distance between cold and hot as bigger than cold and mild. Nevertheless, for a decision tree algo, it should not matter as these keep being as labels
one-hot-encoding: when the algorithm type requires numbers and there is NOT an ordinal nature among the data, this is the option required to prevent not ordinal data to be understood as being ordinal.
More info here


Help needed in interpreting the loss, val_loss vs epoch plots for an autoencoder training? I am training a variational autoencoder and I am getting a loss-plot as follows:

Rigt after epoch 224, val-loss overtakes train-loss and sort of getting bigger but at an extremely slow pace as you can notice. I trained for 300 epochs.
Any opinion about the training. I don't think it is overfitting the data. But I want to be sure and hence seeking opinion from the data science community.
Thanks.

You tagged your question with vae, so I will assume that this is a variational autoencoder. In that case, your loss has 2 components: reconstruction error and KL divergence of the bottleneck distribution, probably balanced with a weight $\beta$. You should plot both components separately in order to understand how the training dynamics evolve. The reconstruction loss is what will tell you if there's overfitting. The KL divergence will tell you if there is posterior collapse.


Transformer masking during training or inference? I'm working through Attention is All you Need, and I have a question about masking in the decoder. It's stated that masking is used to ensure the model doesn't attend to any tokens in the future (not yet predicted), so it can be used autoregressively during inference.
I don't understand how masking is used during inference. When the encoder is given an unseen sample with no ground truth output or prediction, it seems to me that there is nothing to mask, since there aren't any output tokens beyond what the decoder has already produced. Is my understanding of masking correct?
Thanks!

The trick is that you do not need masking at inference time. The purpose of masking is that you prevent the decoder state from attending to positions that correspond to tokens "in the future", i.e., those that will not be known at the inference time, because they will not have been generated yet.
At inference time, it is no longer a problem because there are no tokens from the future, there have not been generated yet.


Use Machine Learning/Artificial Intelligence to predict next number (n+1) in a given sequence of random increasing integers The AI must predict the next number in a given sequence of incremental integers (with no obvious pattern) using Python but so far I don't get the intended result!
I tried changing the learning rate and iterations but so far no luck!

Example sequence: [1, 3, 7, 8, 21, 49, 76, 224]

Expected result: 467

Result found : 2,795.5

Cost: 504579.43

PS. The same thread exists on AI Stackexchange & Stackoverflow and I've been advised to post it here!

This is what I've done so far:

import numpy as np

# Init sequence
data =\
    [
        [0, 1.0], [1, 3.0], [2, 7.0], [3, 8.0],
        [4, 21.0], [5, 49.0], [6, 76.0], [7, 224.0]
    ]

X = np.matrix(data)[:, 0]
y = np.matrix(data)[:, 1]

def J(X, y, theta):
    theta = np.matrix(theta).T
    m = len(y)
    predictions = X * theta
    sqError = np.power((predictions-y), [2])
    return 1/(2*m) * sum(sqError)

dataX = np.matrix(data)[:, 0:1]
X = np.ones((len(dataX), 2))
X[:, 1:] = dataX

# gradient descent function
def gradient(X, y, alpha, theta, iters):
    J_history = np.zeros(iters)
    m = len(y)
    theta = np.matrix(theta).T
    for i in range(iters):
        h0 = X * theta
        delta = (1 / m) * (X.T * h0 - X.T * y)
        theta = theta - alpha * delta
        J_history[i] = J(X, y, theta.T)
     return J_history, theta
print('\n'+40*'=')

# Theta initialization
theta = np.matrix([np.random.random(), np.random.random()])

# Learning rate
alpha = 0.02

# Iterations
iters = 1000000

print('\n== Model summary ==\nLearning rate: {}\nIterations: {}\nInitial
theta: {}\nInitial J: {:.2f}\n'
  .format(alpha, iters, theta, J(X, y, theta).item()))
print('Training model... ')

# Train model and find optimal Theta value
J_history, theta_min = gradient(X, y, alpha, theta, iters)
print('Done, Model is trained')
print('\nModelled prediction function is:\ny = {:.2f} * x + {:.2f}'
  .format(theta_min[1].item(), theta_min[0].item()))
print('Cost is: {:.2f}'.format(J(X, y, theta_min.T).item()))

# Calculate the predicted profit
def predict(pop):
    return [1, pop] * theta_min

# Now
p = len(data)
print('\n'+40*'=')
print('Initial sequence was:\n', *np.array(data)[:, 1])
print('\nNext numbers should be: {:,.1f}'
  .format(predict(p).item()))

Another method I tried but still giving wrong results

import numpy as np
from sklearn import datasets, linear_model

# Define the problem
problem = [1, 3, 7, 8, 21, 49, 76, 224]

# create x and y for the problem

x = []
y = []

for (xi, yi) in enumerate(problem):
    x.append([xi])
    y.append(yi)

x = np.array(x)
y = np.array(y)
# Create linear regression object
regr = linear_model.LinearRegression()
regr.fit(x, y)

# create the testing set
x_test = [[i] for i in range(len(x), 3 + len(x))]

# The coefficients
print('Coefficients: \n', regr.coef_)
# The mean squared error
print("Mean squared error: %.2f" % np.mean((regr.predict(x) - y) ** 2))
# Explained variance score: 1 is perfect prediction
print('Variance score: %.2f' % regr.score(x, y))

# Do predictions
y_predicted = regr.predict(x_test)

print("Next few numbers in the series are")
for pred in y_predicted:
    print(pred)

You are solving a problem which is not designed for a ANN, neural network has difficulties when you deal with univariate, few data.

Because they will have to learn a solution based on few examples.

However, a possible solution is achievable:

import numpy as np

# Init sequence
data = [[0, 1.0], [1, 3.0], [2, 7.0], [3, 8.0],
    [4, 21.0], [5, 49.0], [6, 76.0], [7, 224.0]]

X = np.matrix(data)[:, 0]
y = np.matrix(data)[:, 1]

Reg=neural_network.MLPRegressor(solver='lbfgs',random_state=4,hidden_layer_sizes=(100,20,2),learning_rate='adaptive',verbose=True)

y2 = np.ravel(y)
F=Reg.fit(X=X,y=y2,)
F.predict(8)

Note that the random_state parameter has a very large influence when I use a Neural Network. If you move this parameter (integer) you will find that the solution has a really large range.


How to find the relationship between categorical variables? I have a dataset where I have the Outcome of a case and the Country under review, as well as whether a judge (among a panel of judges) reviewing the case was also from the country under review, Country_Judge, categorised as TRUE or FALSE.

How can I measure the relationship between the Country_Judge and the Outcome of a case? I want to know whether a judge's nationality has an impact on the outcome of the cases.

Is Outcome also a boolean variable?  If so, a simple prop.test will do.

Here's a toy dataset where a judge from the same country is less likely to give a guilty verdict.

library(tidyverse)
n<-1000
dataset<-tibble(country_judge = sample(c(TRUE,FALSE), n,
                                           replace=T, prob=c(0.2,0.8))) %>%
  mutate(outcome = ifelse(country_judge,
                          sample(c("Guilty", "Innocent"), n,
                                     replace=T, prob=c(0.4,0.6)),
                          sample(c("Guilty", "Innocent"), n,
                                     replace=T, prob=c(0.5,0.5))))

dataset %>%
  group_by(country_judge) %>%
  summarise(p_guilty=mean(outcome=="Guilty"))

This will give something like:

# A tibble: 2 x 2
  country_judge  p_guilty
          <lgl>     <dbl>
1         FALSE 0.5108835
2          TRUE 0.3698630

Now, pull out vectors of trials, and "successes", and feed those into prop.test.

trials <- dataset %>%
  group_by(country_judge) %>%
  count() %>%
  pull(n)

successes <- dataset %>%
  filter(outcome=="Guilty") %>%
  group_by(country_judge) %>%
  count() %>%
  pull(n)

prop.test(successes, trials)

Which gives something like:

    2-sample test for equality of proportions with continuity correction

data:  successes out of trials
X-squared = 13.068, df = 1, p-value = 0.0003003
alternative hypothesis: two.sided
95 percent confidence interval:
 0.06517776 0.21686317
sample estimates:
   prop 1    prop 2
0.5108835 0.3698630


How do i pass data into keras? I am currently struggling to understand how i should train my regression network using keras. I am not sure how I should pass my input data to the network.

Both the input data and the output data is stored as a list of numpy arrays.

Each input numpy array is a matrix which has (400 rows, x columns) Each output numpy array is a matrix which has (x number of rows, 13 columns)

So input dimension is 400 and output is 13. But how do I pass each of these sets within the list to the training?

# Multilayer Perceptron
model = Sequential()    # Feedforward
model.add(Dense(3, input_dim=400, output_dim=13))
model.add(Activation('tanh'))
model.add(Dense(1))
model.compile('sgd', 'mse')

I know that model.fit trains the model given the parameter, how it actually takes in the data seems a bit like magic to me, and how it knows that the columns of matrix A should be mapped the rows of matrix B this has to be done for the whole matrix, appended to the list.

If you look at the Keras documentation, you will observe that for Sequential model's first layers takes the required input. So for example, your first layer is Dense layer with input dimension as 400. Hence each input should be a numpy array of size 400. You can pass a 2D numpy array with size (x,400). (I assume that x is the number of input examples). Your output has 13 dimensions for each output document and hence your final Dense layer has to be model.add(Dense(13)) instead of model.add(Dense(1)).

Coming to how it knows, it takes the first dimension of X as number of training examples and second dimension as the size of each example. Similarly for output layer.


How to group by one column and count frequency from other column for each item in the previous column in python? I am trying to group my data by the 'ID' column. Then I want to count the frequency of 'sequence' for each 'ID'. Here is a sample of the data frame:

ID     Sequence
101    1-2
101    3-1
101    1-2
102    4-6
102    7-8
102    4-6
102    4-6
103    1118-69
104    1-2
104    1-2

I am looking for a count same as:

ID   Sequence   Count
101    1-2        2
       3-1        1
102    4-6        3
       7-8        1
103    1118-69    1
104    1-2        2

I tried this code in python which doesn't give me what I want

  df.groupby('ID')
  Blockquote

df.groupby('Sequence').count()

This simple Code worked:

Count_sequence = df.groupby(['ID','Sequence']).count()

For obtaining the output in an excel sheet:

Count_sequence.to_excel('sequence_count.xlsx)


How to convert regression into classification? So I have a regression problem with bunch of features X, and labels in the amount (price $). How can I convert it to classification problem? I have read about convert label from continuous to categorical possibly thesholding at some points for instance 0-50 (0 class) 51-100 (1 class) and so on till 500 maybe (here thresh-holding to 50). This approach is ineffective intuitively i.e no data lay 51-100. Is there any way to mitigate this problem or if there is any effective way to do this? Like in clustering we can use density estimation can we do that here?

To add to the number of methods you can use to convert your regression problem into a classification problem, you can use discretised percentiles to define categories instead of numerical values. For example, from this you can then predict if the price is in the top 10th (20th, 30th, etc.) percentile. These values you can easily find out using Python's numpy.percentile(X[i], percentile) method, for example.
This type of labelling will place thresholds with respect to the distribution of the data.


Extracting list of locations from text using R I have a string containing many words [not sentences], I want to know how I can extract all the words that correspond to a location in that string for example:

text<-c("China","Japan","perspective","United Kingdom","formatting","clear","India","Sudan","United States of America","Bagel","Mongolian",...)

The output should be:

 > China, Japan, United Kingdom, Mongolian

something of the type. Basically I am looking at extracting locative information from random text.
This is a very general problem I am looking for guidance on how to model my solution, is there any dataset or something I can use to compare or extract information from. I don't want to carry out word by word comparison.

I have looked up OpenNLP but I am not sure how to use it's location-models for carrying out Named Entity Recognition in R. In the above example there are only countries but I would like to identify other places, such as provinces, states, counties, cities, etc. as well.
I am new to machine learning and R-programming, any guidance is greatly appreciated.

I use the function grepl() for this.

text<-c("China","Japan","perspective","United Kingdom","formatting","clear",
"India","Sudan","United States of America","Bagel","Mongolian")

#Make a logical vector
arg<-grepl("China",text)|grepl("Japan",text)|grepl("UnitedKingdom",text)|
grepl("Mongolian",text)

#Pick out from text
text[arg]


Should I scale data before or after balancing dataset? I have 3 datasets which I each split into 3 separate classes [Buy/hold/sell]. I randomly up-sample each class's frequency in each dataset to 10,000 data points each.

My question is, should I scale the training set before I do random up-sampling, or afterwards? Does it somehow skew the final training set in some way?

I have provided a function which balances each dataset for me, note that at this point, the data has already been scaled.

def balance_dataset(df):
    training_set = df[:round(len(df.values) * TRAINING_LENGTH)]

    label_frequencies = training_set['Label'].value_counts(sort = True, ascending = True)

    highest_occurence = resample(training_set[training_set['Label'] == label_frequencies.index[2]], n_samples = 10000, random_state = 0, replace = True)
    middle_occurence = resample(training_set[training_set['Label'] == label_frequencies.index[1]], n_samples = 10000, random_state = 0, replace = True)
    lowest_occurrence = resample(training_set[training_set['Label'] == label_frequencies.index[0]], n_samples = 10000, random_state = 0, replace = True)

    balanced_training_set = pd.concat([highest_occurence, middle_occurence, lowest_occurrence])

    return balanced_training_set[['MACD', 'MFI', 'ROC', 'RSI', 'Ultimate Oscillator', 'Williams %R', 'Awesome Oscillator', 'KAMA',
            'Stochastic Oscillator', 'TSI', 'Volume Accumulator', 'ADI', 'CMF', 'EoM', 'FI', 'VPT','ADX', 'ADX Negative', 'ADX Positive',
            'EMA', 'CRA']],  balanced_training_set['Label']

Scaling, in general, depends on the min and max values in your dataset and up sampling, down sampling or even smote cannot change those values. So if you are including all the records in your final dataset then you can do it at anytime but, if you are not including all of your original records then you should do it before upsampling.


Image Super-resolution Connecting Subimages I'm working with image super-resolution on terrain height data. Currently, I'm cutting the input data into smaller pieces (20 x 20 rather than 10800 x 10800).

After the upscale 20x20 -> 40x40, piecing the resulting images together causes artifacts along the borders of the smaller pieces. This is because the outputs of the model don't flow well together (we are using the output to run simulations, so it's important that they fit seamlessly).

Is there a recommended way of stitching these smaller images together? Or is there a better approach to this problem than using these smaller cut-down images?

You're probably seeing those artifacts because your model doesn't see those pixels immediately outside your tile and so can't know how to "blend" things.  (I'm assuming your tiles have a stride equal to the input size)
A typical approach I've seen used (and used myself) is, at inference time, to keep only a central portion of each tile and then to have overlapping windows so you can "fill the space".
The more overlap (and less of the output you use) the less apparent the artifacts will be (or have been in my experience), but more computation is needed.
Here's what I mean in cartoon form:
Current situation(as I assume):
input tile1 #####----------
input tile2 -----#####-----
output tile1 ##########--------------------
output tile2 ----------##########----------
My suggestion:
Here, for the output # means the value there is copied to the output,
and * means that the model makes a prediction there but it's not used.
input tile1 #####----------
input tile2 ---#####-------
output tile1 **######**--------------------
output tile2 ------**######**--------------
If you go with this idea, you could even simplify the model so that it doesn't attempt to make predictions at locations you would ignore anyway.
And please comment if things need clarification.


Ideas to check if a given sentence A is the response to a query B I've a sentence 'A' which is a possible response to one of the x number of questions. I need to compare the response with each sentence to see which question's response sentence 'A' is or not a response at all. Any ideas on how to go about this. Pure nlp based solution will be helpful as there are no existing labelled data for the same.

So, I gathered that your question is about how to evaluate whether a sentence A is a response to a given question Q or not.

Drawing from knowledge of question-answer systems, you can evaluate whether a sentence responds to question Q by comparing the underlying lambda calculus (essentially translates language meaning into a logical, more computation-looking form - https://www.cl.cam.ac.uk/teaching/1314/NLP/slides6.pdf) which make both the question Q and a response A.

To get the lambda calculus for a question Q or response A, we can use an encoder-decoder model to generate these.
In an encoder-decoder network architecture, we simply encode input (e.g. question Q using an RNN/LSTM) information into a "hidden representation", which we then subsequently decode into our desired output (e.g. Lambda Calculus, using an RNN/LSTM). This process is actually called semantic parsing and here are a couple of papers which do this and label the architectures they used to generate these, including:

https://www.aclweb.org/anthology/P16-1004/
https://www.aclweb.org/anthology/P18-1068/

Maybe this can be an implicit way of comparing the questions with the responses, by then looking at similarities in their lambda calculus.


How to order a python dataframe by adding the row values? I have the following dataframe:

    M1  M2  M4   M5
N1  45  46  54   57
N2  32  36  29   56
N3  56  44  40   55
N4  57  43  42   54

How is it possible to order it according to the sum of its lines, from the largest sum to the lowest? That is, N1 in the first line, because the sum of the line values is the largest, then N4, N3 and finally N2 (because the sum of the values is the smallest). Thus:

    M1  M2  M4   M5
N1  45  46  54   57
N4  57  43  42   54
N3  56  44  40   55
N2  32  36  29   56

#create data
DF=pd.DataFrame(columns=["M1", "M2", "M3", "M4"])
DF.loc[0]=[45, 46, 54 , 57]
DF.loc[1]=[32 ,36 ,29 ,56]
DF.loc[2]=[56,44 ,40 ,55]
DF.loc[3]=[57 ,43 ,42 ,54]
DF=DF.astype(int)

#main function

DF["sum_"]=DF.apply(lambda x: sum(x), axis=1)
DF.sort_values("sum_", ascending=False)


how should I measure performance if there is no test data? I have 'practice' data set which I can split into training, validation, and test set and I will play with data to make a machine learning model. But in real situation, I will be given a very small data set which I will split into training and validation sets. There is no enough observations to make a separate test set. Then how can I estimate the 'fair' performance of the model on the real data?

The only idea I have is to find a relation between performance on the validation set and test set on the practice data (a regression line etc) and then apply the formula to the performance on the validation set of the real data to find that on the test set of the real data (which actually does not exist.)

Is there any other good idea?

If you are working with only enough data for training and validation, consider using K-Fold Cross Validation:

  One of the main reasons for using cross-validation instead of using the conventional validation (e.g. partitioning the data set into two sets of 70% for training and 30% for test) is that there is not enough data available to partition it into separate training and test sets without losing significant modelling or testing capability. In these cases, a fair way to properly estimate model prediction performance is to use cross-validation as a powerful general technique.

Assuming your "practice" data is drawn from the same distribution as your "real" data, you are on the right track by thinking about measuring the relationship between training accuracy and test accuracy in the first set to model the relationship in the second set.

However you should split your practice data into training and test only and use k-fold cross-validation on the training set. Then you should train a model on the real data with the same cross validation scheme.

For example you might get results like:

practice data:
  training accuracy: 90%
  test accuracy: 88%

real data:
  training accuracy: 89%

Since our cross-validation scheme was resistant to over-fitting, the training and test accuracy are close on the practice set, and our practice training accuracy is close to our test training accuracy. Now we can safely assume that accuracy on the unseen test set for the real data is about 2% worse than training accuracy.

However, imagine you a single validation set or no validation at all on the practice data. Now your model is more likely to overfit and you might see results like this:

practice data:
  training accuracy: 95%
  test accuracy: 75%

real data:
  training accuracy: 85%

Here we have created a model with high variance by overfitting. Training and test accuracy are far apart on the practice data, and training accuracy on the practice data does not match well with training accuracy on the real data. It is not as easy to estimate test accuracy now, because we can't really say if we have overfit by the same amount on both datasets and the second set is harder, or if we have overfit the second set by less. In the first case we might predict 65% test accuracy, and in the second case we might predict 75%.


Dealing with dates in dataframe I have a dataframe which has two date columns - Start Date and End Date,
Let's suppose start date is 4/10/2019 and end date is 4/10/2020 (MM/DD/YYYY) now I want to split these dates in a list which will have all the intermediate months (from october 2019 to october 2020). Is there any function in python that can hep me with this?

You can likely use pandas.date_range as follows assuming your columns are called start and end:
import pandas as pd

df.apply(lambda x: pd.date_range(x["start"], x["end"]), axis=1)


SVM is not fitted when tried to fit it into a model I was trying to apply SVM into a dataset to find its accuracy. I also applied bagging and boosting on the same dataset and they worked properly. But when I tried to fit SVM into a model its not working. I couldn't understand whats the problem.

I'm providing the sample of my dataset

train_new.head()

# define X and y
feature_cols = ['age','workclass','fnlwgt','education','education-num','marital-status','occupation','relationship','race','sex','capital-gain','capital-loss','hours-per-week','native-country']

# X is a matrix, hence we use [] to access the features we want in feature_cols
X = train_new[feature_cols]

# y is a vector, hence we use dot to access 'label'
y = train_new['label']

from sklearn import svm
model = svm.SVC(kernel='linear', C=1, gamma=1)

and while trying to fit with model it's almost minutes passed and it's just look like,

What's the actual problem and could I solve it ?

15 attributes with 30161 instances are not much to cause this delay, I've trained with much more. Your system config also plays a major role in this. For instance if you are using anaconda on chrome and you have multiple processes going on then it would cause delay as distances are calculated on the fly which will be heavy on memory.

You can also try SGD as you are going for linear. It is faster than SVM.


Regression performance with Feature Selection I would like to ask you a theoretical question. In my project I am trying to get a better performance from my regression model by feature selection methods, especially with CatBoost feature importances.
I would like to ask:
1- I know the term "Garbage in Garbage out", so more features do not always mean better performance; moreover it decreases the performance. But can we get a better evaluation score like MSE, RMSE by eliminating less important features from the model? In my project It wasn't the case; MSE increased gradually while I am removing features step by step.
Should I expect a better model and higher predictive performance by eliminating unnecessary features? Or when should I expect that?
2- R2 is another metric, but I think it won't increase by eliminating a feature. In my opinion an extra feature may increase R2 value or do not effect at all. Am I right or an unnecessary feature may decrease R2?

Generally speaking, some ML algorithms include regularization (such as Ridge/Lasso Regression, Random Forest, CatBoost,...) and others don't (k-NN, Gaussian Naive Bayesian,...). Using one that includes regularization may not change the performance when removing less important features.
This applies to your first question. If you use Lasso regularization for example, it will push the weights of "less important" features to zero. So removing them should not change the performance. Using Ridge regularization is a bit different.
So when you say:

MSE increased gradually while I am removing features step by step

There could be multiple reasons for that:

the features you removed contained information and were not "less important" for the regression (see *Note/warning)
even the less important features contain information
you did not retrain your model after removing the "less important" features

Regarding your second question, $R^2$ is related to $MSE$ (it is well explains in this post), so if the $MSE$ increases, $R^2$ should increase as well.
*Note/warning: a feature is often said to be "less important" using one algorithm (such as CatBoost which often uses Gini or entropy). It does not always mean this feature will be "less important" when using it with another algorithm (such as regression).


Getting into data science and NLP outside of university I'm almost 40.

Four months ago I resigned from work to rest from stressing work and change my carerr direction. It took me some time to understand what I want to do and only then I started to learn NLP and machine learning through Coursera and Kaggle. To learn new skills and have a minimal portfolio to show during future job interviews,  I've started to work  on a small NLP project. It involves multiple problems like crawling data from the Web, cleaning them, extracting structured data from unstructured text, linking objects based on document similarity, stemming, POS tagging, lemma disambiguation, etc.

However, I feel I'm getting lost and often going into wrong directions. Actually, it starts to look like doing my PhD in the past, reinventing too many wheels. For instance, last week I've spent on porting existing stemmer for my language from Java to Python just to realize a few things. First, that less ideal but simpler solution (Python wrapper) would be enough for my goals and definitely faster to write. Second, I've learned more about software engineering (difference between Java and Python, calling Java from Python) rather than about NLP challenges of the problem (what are the current approaches for stemming, how to train trie-based stemmers, etc.). Third, I've found that the approach I used might be a bit outdated.

I feel lost because of many unknowns:

I don't know if the project that I took makes sense for my future career?
Isn't focusing on my language in my NLP project too constraining? NLP tooling for my language is quite limited when compared to English.
Who knows what are uknown uknowns?

In the end I think it boils down to lack feedback on my progress and actual skills I have vs. what is needed.  The only feedback I've received so far was from one job interview where I've learned that the approach I'm using for my project might be outdated and not necessarily an industry standard. I've looked at the work of a group of students contributing to SpaCy, NLP framework in Python. They have a project supervised by a univeristy teacher or principal, the project requirements have been defined by an external company and they have a chance to work together.

How to get such a feedback when you're no longer a student? Internship? Mentoring? How do you learn alone such things?

I'm also nearly 40.

2 years ago I decided to change career paths from Engineering to "Data Science".

Despite having many years of Python experience I recognised that I didn't know or understand the latest technology or even the modern terms for things I did everyday.

Therefore I suggest the following:

Put your existing skills on a more formal basis using MOOC like DataCamp or Data Quest.  This will quickly teach you the relevant terms (so you know what to Google) and relatively recent practices.
Read.  Many great books available at low cost or even free, whether it's Python for Data Science or Intro to Sci-kit learn.  Try out the tutorials.  Read articles on Hacker News on the topic.
Practice.  Kaggle is interesting but I found following the tutorials from fast.ai better for real life use.  It's better to apply these skills to something you find interesting. E.g. build a chat bot that turns in the sprinklers in your garden.

After all of this, you'll realise there are opportunities in every business to practice your skills.  Some will use the term "Data Scientist" but I suspect most won't.

You didn't say what you did before, but I suspect if like me you subscribe to the following definition of Data Science then you'd already be there.

Data Science is the combination of Advance Computing Skills, Statistics and/or Maths and "Domain Knowledge".

What is domain knowledge?  Any subject that you know a lot about.  You could be radiologist in a hospital or fraud investigator in an insurance company.


Changing the value of a column based on the condition on another column I have a table whose header looks like this:
    complaint_type  borough street_name incident_zip    latitude    longitude

I want to check if the "incident_zip" column of each row is in a specific list of zip codes and change the "borough" accordingly. There is a large amount of data and i cannot find any better code to do this. I am using python 3.6.

There is a way that is not optimum for performance, but clear to read and understand. Also, you can use more complex logic and update with different values as you wish.

def myfunc(row):
    if row['incident_zip'] in [this is a list of zip codes]:
        return new_value
    else:
        return row['borough'] #return old value

data['borough'] = data.apply(myfunc, axis=1)

If your logic of update is quite simple, then you can do it like that

mask = df['incident_zip'].isin([this is a list of zip codes])
df[mask]['borough'] = new_value


geo spatial clustering based on another feature I have data about houses for sale, that I present over a map.

Each house has coordinates ([lat,lng]) and other features. The data is only for one country, so no need to address the 180deg world wrap.

I want to cluster neighbourhoods, out of those houses, and base the the neighbourhood borders on prices (sqm price).

It's really easy to see those clusters when I paint the houses with gradient color - from cheap to expensive, but I can't find a trivial way to do it mathematically.

Basically, what I want is price clustering, but with a limitation on the clustering algorithm so there won't be any geo-location overlap.

Another way to look at it - I want to blur the data points and create low-resolution areas.

Not sure where to go with this, any help will be appreciated.

There are many such approaches, for example spatial autocorrelation, Lisa, etc.

In the clustering domain, GDBSCAN is a generalization of DBSCAN where you could easily define neighbors as points being within a certain distance and having a similar price.

Nevertheless it is probably a good idea to look into actual geostatistics and beyond the limitations of sklearn.


When Should I Use ggplot2 Instead of Tableau? I am a student getting started with Tableau for the first time. My proficiency with ggplot2 is intermediate. I can create custom versions of the most popular kinds of charts in ggplot2 but nothing too fancy (and not very time-efficiently).

I am interested in creating more polished visuals of my data in Tableau. My ggplot2 ones aren't bad, but they are not what I would call aesthetically pleasing either.

What I would like to know is under what situations I should stick with ggplot2. The only advantages that come to mind with my current (albeit limited) understanding of Tableau is to keep my work reproducible and open source. ggplot2 is also probably more customizable, but I am continually impressed by Tableau's offerings as I learn to use it.

This question is not intended to be a flippant jab at ggplot2. As I did not hear any talk about Tableau in the academic circles I was in, I am wondering what obvious drawbacks to Tableau may exist that would explain its unpopularity there.

There is one big economic difference between the two: ggplot2 is an open source package for an open source programming language. On the contrary Tableau is a proprietary software. That might be a dependency that you might not want to risk, e.g. you do everything in Tableau but then the license gets more expensive and your organization does not want to spend money for it anymore. Academics would often stress this reason, another reason of low adoption there is that Tableau has always been the BI tool and R the "research" tool.

Apart from this, they are simply different tools (in the same domain of data viz) and as such will be suited for different problems. This can be subject to personal preference, but I would say that there is a few basic considerations:

Speed: how fast can you get what you need?

Use the tool that gets you results. Tableau might often be faster.

Reproducibility & automation:

If you know you might want to plot this chart many times in future in a consistent manner or embed your plots in some sort of notebook like R Markdown or dashboard, then R is most likely the better choice. You write the code once and produce thousands of plots within seconds if you need to.

Type of project:

If you are starting with messy data, or you are doing some modeling work, it seems as a good strategy for me to just stick to R, I see it as more efficient to have everything in one place.
If you need to get awesome visualizations from clean data, Tableau will excel
Based on the type of visualizations and output you want, one or the other might be better, ggplot2 is extremely flexible, Tableau has a lot of amazing functionality which is ready to go.


How to convert DataFrame column to Rows in Python? I have the following dataset in df_1 which I want to convert into the format of df_2. In df_2 I have converted the columns of df_1 to rows in df_2 (excluding UserId and Date). I looked up for similar answers but they are providing little complex solutions. Is there a simple way to do this?
df_1
   UserId       Date                   -7  -6  -5  -4  -3  -2  -1   0   1   2   3   4   5   6   7
    87      2011-05-10 18:38:55.030     0   0   0   0   0   0   1   0   0   0   0   0   0   0   0
    487     2011-11-29 14:46:12.080     0   0   1   0   0   0   0   0   0   0   0   0   0   0   0
    21      2012-03-02 14:35:06.867     0   1   0   1   2   0   2   2   0   1   2   2   1   3   1

df_2
day | count
-7   0
-7   0
-7   0
-6   0
-6   0
-6   1
-5   0
-5   1
-5   0
.    .
.    .(Similarly for other columns in between)
.    .
6   0
6   0
6   3
7   0
7   0
7   1

Try using pandas melt
dataT = pd.DataFrame({"userID":[1,2,3],"date":["2020-01-01","2019-01-02","2020-08-12"],"-7":[0,1,1],"-6":[1,0,0],"-5":[0,0,0]})

Input:

dataT.melt(value_vars= ["-7","-6","-5"], value_name="count")

Output:

Update
By taking the comment by Benji the code would be:
dataT.melt(id_vars= ["userID","date"], value_name="count")

With output:


How to choose best classifier for Low positive to negative class ratio in data (training, validation and real time)? Positive class is ~4%. The class weight methods will not work as even if I balance the data while training by scaling positive class samples, in real time (or in test data), the distribution is expected to be low. How to choose a good decision boundary for such cases.

You don't necessarily have to train any differently if you are happy with the result. If you aren't, undersample the majority class and/or adjust the class weights in your loss function. The thing you really have to be more careful about is the evaluation of the result because a naive metric like the accuracy would give you false hope since the imbalance allows you to achieve a high accuracy by simply always predicting the majority class or randomly predicting each class according to its probability of occurrence. It is this baseline against which you have to judge your success; like the coefficient of determination in regression, where the baseline is the sample mean.

You could use a graphical approach (ROC or precision/recall curves), or report a size-weighted average of the class positive rate instead of the accuracy. Here are some other ideas: Comparison of Evaluation Metrics in Classification Applications with Imbalanced Datasets.


Language translation with convolutional neural network Many examples of language translation neural networks:

  "the cat sat on the mat" -> [model] -> "le chat etait assis sur le tapis"

use RNN, and in particular LSTM. See for example Sentences language translation with neural network, with a simple layer structure (if possible sequential) and A ten-minute introduction to sequence-to-sequence learning in Keras
.

Are there (not too complicated) attempts of doing language translation with CNN only and no RNN/LSTM?

Would you have an example in Keras?

I just googled:

A Convolutional Encoder Model for Neural Machine Translation, by Gehring et al., link
Convolutional Sequence to Sequence Learning, by Gehring et al. link
Pervasive Attention: 2D Convolutional Neural Networks for Sequence-to-Sequence Prediction, by Elbayad et al. link

All the implementations I found on GitHub are in PyTorch. I'm not surprised I didn't find much: the application of CNNs to NLP was interesting, but they never beat RNNs at that. After the advent of Attention models, and especially Transformers, these kind of models have not been developed further. In fact the latest paper I linked you is from 2018 - i.e. an Ice Age ago for the pace of NLP.

If you really want to go deeper on this topic of Convolutional NMTs, I suggest you to check the available torch-based code and try to make a replica in Tensorflow/Keras. It's hard work, but a fancy model nonetheless. Good luck!


Data balance -before or after feature selection/engineering Among the different stages of creating a prediction model, when should I balance the data?

Should it be at the beginning before I perform any manipulation on the data and creating new features, or after, when I already selected the most important features based on the real data?

Balancing is using which of the samples to consider in the data set (adding/reducing rows).
Features selection and feature engineering is removing and adding information about each sample (adding/removing columns).

Therefore, you have the liberty about how to combine them.
Many times, the order doesn't matter.

However, if you are normalising features, they will be normalised with respect to the data set and you might get different results before and after balancing.
Feature might change it's importance after balancing, influencing the feature selection.

In general, if you are modeling on a data set, it is better to adapt the feature engineering to it so balance first and do feature engineering later later unless you have a reason to do otherwise.


Remove part of the string in R - dropping some words from it I have a column in dataframe which should represent the date, named "taken", and looks like this:

taken

Sat Jan 01 15:30:27 CET 2011

Thu Jan 06 00:54:26 CET 2011

Sat Jan 08 19:49:02 CET 2011

Sun Jan 16 09:57:16 CET 2011

etc (more than 30000 rows)

Now, I need to change class from factor to date, but I haven't found solution for this kind of format. Since it would be easier with date looking like this:

taken

Jan 01 2011

Jan 06 2011

Jan 08 2011

Jan 16 2011

...How could I drop some words from a string (obviously, Mon/Tue... time and CET in this case) to change it into date?

I wouldn't use them as time at first place. What would I do:

String <- "Sat Jan 01 15:30:27 CET 2011"

As your dates have same number of elements in them we could just count what part of the String we would like to get.

date <- substr(String, start = 5, stop = 10)

year <- substr(String, start =25 , stop = 28)

Attach them togheter

final <- paste(date,year, sep = " ")

as.Date(final, format = "%b %d %Y")

Little bit more information about as.Date
https://www.statmethods.net/input/dates.html
I hope it helps you :)


fit model with sd square If I want to fit a nonlinear regression model with some parameters like $\sigma^2$(where $\sigma$ is the standard deviation, which is positive), how can I guarantee that $\hat{\sigma}$ is positive?

I mean, if I use maximum likelihood(and the model only have square term of $\sigma$), how does the optimization method know $\sigma$ is positive? (well, it seems OK that the result estimation of $\sigma$ is negative, but it looks weird, and this is why I ask this question.)

I would suggest parametrizing with a logarithm of volatility so you don't have to care about positivity, run the estimation and then invert back to original scale. Alternatively, you can consider constrained optimization routine. Without knowing more about the problem (at least the language you're using), that's about it.


Bayes Classification In theory we would always like to predict qualitative responses using
  the Bayes classifier. But for real data, we do not know the
  conditional distribution of Y given X, and so computing the Bayes
  classifier is impossible. Therefore, the Bayes classifier serves as an
  unattainable gold standard against which to compare other methods.

In a real life situation why we are not able to calculate the Bayes Classifier ?
Why can't we calculate the conditional distribution of Y given X ?
Can some one explanation with an example?

Any help is appreciated.

You can't calculate the conditional distribution because that distribution is defined over the entire population and assuming a probabilistic model which you do not know.

In real life you have only a sample of the population. More often than not, in real life you do not have the structure of the model also. You do not know if it is a linear model, a polynomial or another type.

The probabilistic approach is to assume a model, consider some assumptions for that model and fit the model's parameters using data. That is not the Bayes classifier, it's only an approximation.

The trick is 'all models are wrong, some models are useful'. There are many approaches to learn this approximation. You can estimate the joint probability (generative classifiers), you can estimate directly the conditional probability (discriminative classifiers) or you can use a different approach (like SVM, for example) and twist it somehow to approximate the conditional probability. But all of them are approximations. The real one, THE Bayes classifier will remain unknown.

[Later edit]

Looking at the Wikipedia page for Bayes classifier it states that: "The Bayes classifier is a useful benchmark in statistical classification.". My opinion is that this statement is confusing, if not dead wrong. Probably this is related with the wrong usage of expression "Bayes classifier" to designate the Naive Bayes classifier. If one search on google for "Bayes classifier", he will see a lot of materials related with Naive Bayes. Personally I do not agree with the statement from wikipedia. Bayes classifier is that state of nature that we do not know but we aim to approximate as well as possible.


Speed for different kernels in scikit-learn's SVM I'm using scikitlearn in Python to create some models while trying different kernels. I was surprised to see that rbf was fit in under a second, whereas linear took a minute and poly took hours. Can anyone explain why to help me?

My code:

import numpy as np
from sklearn.svm import SVR
import matplotlib.pyplot as plt
from sklearn.metrics
import mean_squared_error, r2_score
f = open(datafile)
X = np.empty([0,1], dtype = int)
y = np.array([])
for i in range(100):
    my_lines = f.readline().split(" ")
    t = np.array([[int(my_lines[0])]])
    n = np.array([int(my_lines[1])])
    X = np.concatenate((X, t), axis =0 )
    y = np.concatenate((y, np.array([int(my_lines[1])])), axis = 0)
f.close()
svr_rbf = SVR(kernel='rbf', C=1e3, gamma=0.1)
svr_lin = SVR(kernel='linear', C=1e3)
svr_poly = SVR(kernel='poly', C=1e3, degree=3)
svr_rbf.fit(X, y)
svr_lin.fit(X, y)
svr_poly.fit(X, y)

The data:

0 22
1 23
2 23
3 23
4 25
5 24
6 25
7 26
8 26
9 30
10 29
11 29
12 30
13 30
14 30
15 31
16 29
17 28
18 29
19 31
20 30
21 30
22 31
23 31
24 30
25 31
26 31
27 30
28 29
29 31
30 33
31 31
32 32
33 30
34 30
35 30
36 29
37 30
38 29
39 29
40 27
41 28
42 28
43 27
44 27
45 28
46 28
47 29
48 29
49 28
50 28
51 29
52 27
53 27
54 28
55 28
56 28
57 30
58 32
59 31
60 30
61 28
62 28
63 30
64 27
65 27
66 28
67 27
68 29
69 33
70 39
71 35
72 34
73 29
74 30
75 28
76 28
77 29
78 27
79 26
80 25
81 17
82 0
83 0
84 29
85 21
86 20
87 18
88 19
89 19
90 19
91 18
92 17
93 18
94 20
95 19
96 20
97 19
98 18
99 18

One option is to speed up your code is to standardize features, removing the mean and scaling to unit variance.
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X = scaler.fit_transform(X.reshape(-1, 1))
y = scaler.fit_transform(y.reshape(-1, 1))

After I standardized the data, each model fit in less than 500ms.
Support vector machines (SVMs) need to calcuate the distance between each data. Standardize numbers make the calculations faster.


Deep reinforcement learning on changing data sizes I have a game that I want to build a model that will learn to play the game. Yet, the environment output is two lists that represent the location and number of soldiers of the user and Opponent.

The lists lengths are changing with each step, so to know the input size to the network is a problem.

How can this be dealt with?

Although this problem can happen quite frequently in Reinforcement Learning (RL) projects, it is not a RL issue. The issue is due to neural network requirements, and also applies to many other parametrised machine learning solutions.

You can vary the number of parameters of a neural network in a limited fashion. However, it can only usually be done before training (or re-training), and is not a suitable approach to deal with changes happening over a small number of time steps, or during post-training prediction (e.g. when playing the game with the trained agent).

There are a few different approaches you can take. Which is suitable will depend on details of the game:

Pad the inputs

This is the simplest approach, and you can do this without needing much additional research. It relies on there being a workable maximum number of inputs you could need at any time step.

Practical considerations:

Have a simple mechanism to identify whether a particular set of inputs are present. E.g. add a ${0,1}$ flag to show present of a unit, leaving rest of inputs that represent the unit at some neutral value such as $0$.
If there are only rare occasions when the last few inputs are used, you should expect the neural network to learn about those situations very slowly. It will not generalise between units in different input "slots", unless you alter the architecture to link them.

Invert the representation from list of units to "world map"

This works nicely for a game where movement is discrete and there are rules that limit the amount of stacking possible in any single space. It also allows you to use a CNN architecture.

For example, this is how AlphaZero represents Go and Chess, which both allow varied number of pieces on the board at any one time, but where the board itself is strictly defined. In the case of Chess, this is also how AlphaZero identifies which piece to move, and how, by output on a matching grid to select the piece and a movement type (each selection/movement type combination in that case is a separate output channel, the output reprsents 1000s of potential moves all at once).

Practical considerations:

If your units have statistics that you need to track, then each of these can go in a separate instance of the grid (sometimes called a "plane" or a "channel"). So you might have one grid showing presence of a piece, another identifying its type, another its attach strength etc, all stacked into a 3D structure assuming the playing board is 2D.
If the map is very large or continuous, then it may be possible to discretise it into a grid.
Using a CNN allows for good generalisation based on similarity between pieces and their stats. However, the approach becomes limited if the grid is very large or stacks of pieces at any one grid position can become large, in addition to requiring a much larger memory footprint to represent the game state.

Feature engineering

Technically, representing the game as a map/board is just one common form of feature engineering in RL. Whether or not it will work, you know the game and may have knowledge of what factors are important to playing it well. You can use this knowledge to simplify the rerpesentation and summarise key details that are independent of the number of units.

For instance, it may be important to prioritise play based on proximity of strongest units in play. So you could sort the available units based on this heuristic and present a smaller fixed number of key units.

Practical considerations:

This can be done in combination with any other approach.
This relies heavily on your creativity and a good technical understanding of both the game and what works well in features used for neural networks.

If you want advice on possible feature engineering approaches, I suggest you open a new question with more details of the game being played.

Recurrent Neural Networks (RNNs)

A recurrent neural network architecture, such as LSTM or GRU, can work directly with mixed length sequences. This can work really well when each item in the sequence is the same kind of object but with different feature values.

Practical considerations:

RNNs are harder to understand and train than fully connected feed forward networks or CNNs.
If the two lists of units are just one part of the game representation, you will need a hybrid system that combines a RNN architecture with other neural network designs.


Understanding action space in stable baselines I was trying to write reinforcement learning agent using stable-baselines3 library. The agent(abservations) method should return action. I went through different models API (like PPO) and they do not really allow us to specify action space. Instead action space is specified in environment.
This notebook says:

The type of action to use (discrete/continuous) will be automatically deduced from the environment action space.

So, it seems that "model" deduce action space from environment.
Q1. But exactly how?
Q2. Also how my agent(observations) method should return action? By returning action returned by model.predict()?

I guess I got answer to Q1. Sharing my understanding below:

PPO extends OnPolicyAlgorithm
PPO calls super(..,env,..), that is OnPolicyAlgorithm(..,env,..)
OnPolicyAlgorithm extends BaseAlgorithm
OnPolicyAlgorithm calls super(..,env,..), that is BaseAlgorithm(..,env,..)
BaseAlgorithm seems to obtain action_space from env

Still I am unable to get answer Q2.


What is the use of [SEP] in paper BERT? I know that [CLS] means the start of a sentence and [SEP] makes BERT know the second sentence has begun.

However, I have a question.

If I have 2 sentences, which are s1 and s2, and our fine-tuning task is the same.

In one way, I add special tokens and the input looks like [CLS]+s1+[SEP] + s2 + [SEP].

In another, I make the input look like [CLS] + s1 + s2 + [SEP].

When I input them to BERT respectively, what is the difference between them?

Will the s1 in second one integrate more information from s2 than the s1 in first one does?
Will the token embeddings change a lot between the 2 methods?

Thanks for any help!

First let's understand why the format is like this.

BERT was pretrained using the format [CLS] sen A [SEP] sen B [SEP]. It is necessary for the Next Sentence Prediction task : determining if sen B is a random sentence with no links with A or not.
The [SEP] in the middle is here to help the model understand which token belong to which sentence.

At finetuning time, if you use a different format than pretraining format, you might confuse the model : he never saw 2 sentences formatted as [CLS] Sen A Sen B [SEP]. The model doesn't know there is 2 sentences, and will consider it as a single sentence.

If you finetune on enough data, BERT can learn the new format. This can be helpful if you need to change the input format.

But in your case, you don't need to do this. Changing the format for the sake of changing the format is just going to confuse your model, he will have to learn more thing, and there will be inconsistencies between pretraining and finetuning.

  Will the s1 in second one integrate more information from s2 than the s1 in first one does?

No. Inserting a SEP token or not will not change the amount of information exchange between the tokens of the 2 sentences. In both case the model will compute attention based on the 2 sentences. Each sentence can see the other sentence's tokens, no matter of the SEP.
The only thing you will do by removing the SEP token is confuse your model.

  Will the token embeddings change a lot between the 2 methods?

We don't know. It will definitely change, but how much ? We cannot answer. My guess is that token representations will not change a lot (because tokens are the same), but CLS representation will change a lot (instead of representing the links between 2 sentences, it will represent something else).


model with only positive responses Could any one help me know about different approaches, methods or algorithms to build a model only with positive responses.

Let's assume we have a set of customers with a 'positive' behaviour. We want to find customers with a similar profile in the database and that are more likely to have this positive behaviour in the future. This can't be modeled with 0/1 target variable because there's not a clear definition of a 'negative' behaviour.

It looks like a look-alike modeling. However I think that look-alike modeling assumes knowing the response (0/1) for the given set of people (treatment group) and finding similar people in the control group to predict their responses.

Thanks in advance of your response!

MK

This sounds like a one-class or unary classification problem where you can build a model of normality using the class that you have.

Take a look at novelty and outlier detection documentation for scikit learn. They discuss the one-class SVM there, which attempts to model the decision boundary given the observations that you have. The example that is implemented is pretty straight forward. Sadly, this implementation does not provide you with a probabilistic measure (although, there are papers on how to add this functionality).

Alternatively, you can consider an auto-associative neural network (replicator network).

There is a nice discussion on time-series anomaly detection here on DS.


Tensorflow Object Detection API Looking at the Tensor-flow object detection API, and walking through the "How to train your own Object Detector" for raccoons, which is a handy guide to get you up and running.

What I would like to do is append the SSD_Mobile model to include my own images along with the old pre-trained data.

Is there any guide out there that shows how this can be done?

To build a model to detect the old categories and your new ones, you need to re-train the model with your own dataset and the dataset used to pre-train the model. Fortunately, this dataset is available to the public for downloading. The dataset used to train the model is written in the name of model you choose. It's probably the COCO dataset.

Since the model as been pre-trained to detect the old categories, I think you don't need to download all the dataset. The data is just necessary to avoid the model to "forget" what it has learned.

I hope you understood me, and that I have answered well your question.


What's SHAP contribution dependency plots from xgboost package in R? clf = xgboost(data.matrix(train2), label = train$MACROSOMIA,objective = "binary:logistic", nrounds = 20)
pred = predict(clf, data.matrix(test[, -12]))

xgb.plot.shap(data.matrix(test2), model = clf, plot_loess =TRUE,features= "GESTAC")

This is my code, and I get a SHAP feature contribution to prediction dependencies on feature value. I just have a hard time understanding it. I wonder what is it? How do I interpret it?

Here's a link! Could you help me understand what does this plot mean?

xgb.plot.shap()

On the y axis. That represents the log odd ratios. The formula is log(p/1-p)

P/1-P  P represents probability of success. 1- P represents probability of failure.

When log odd ratio is 0. It means odd is 1. Probability of success is the same as the probability of failure.

When log odd ratio is below 0, it means that the probability of sucess is less than probability of failure.

When log odd ratio is above 0, it means that the probability of sucess is higher than probability of failure.


How to assign a weight for classifiers when using weighted majority voting? I am trying to apply weighted majority voting on an ensemble as a combiner method. I read different papers and articles, however, I am still a bit lost on:

How the weighted majority voting works

How to assign a weight for every ensemble base classifiers when using weighted majority voting.

You can test each base classifier on a hold-out dataset and come up with a performance metric say accuracy for each model. You can then use each model's accuracy as a weight when combining predictions in the ensemble.
Here is a useful article.


How to combine two differently scaled, but equally important "running" signals into a reward function? I asked this question on Artificial Intelligence, but got no answer, so I am moving it here.
I have two signals that I want to use to model a reward for a reinforcement learning algorithm.
The first one is the CPU TIME: running mean from this diagram:

The second one is the running mean of the MAX RESIDUAL from this diagram:

Both signals are equally important, but they have very different scales. I could combine the signals linearly together like this:
$r = w_\rho \rho + w_\tau \tau$
where $r$ is the reward function, $\tau$ is the CPU TIME: running mean, and $\rho$ is the MAX RESIDUAL. The problem is, how to set the weights $w_\tau,w_\rho$ to make the contributions equally important if $\rho$ and $\tau$ are on very different scales?
Reinforcement learning algorithms will learn policies based on the reward, and if one signal has values that are much smaller than the other, it will influence the reward much less, which is not the behavior I would like to model.
Edit: Dataset on Kaggle
Edit: comment from Pedro
It seems that a linear combination of signals is possible for the scaled mean CPU Time (mean to get rid of oscillations) and the scaled MAX residual:

Using z-normalisation ensures that they have same mean and Standard Deviation, but ofcourse values will be still different since mean and Standard Deviation depend on the Distribution of the data.
Alternative is to use feature scaling, where you force the values between 0 and 1 for both signals.


Does this encoder-decoder LSTM make sense for time series sequence to sequence? TASK
given $\vec x = [x_{t=-3}, x_{t=-2}, x_{t=-1}, x_{t=0}]$
predict $\vec y = [x_{t=1}, x_{t=2}]$
Whith an LSTM encoder-decoder (seq2seq)
MODEL

NOTE: the ? symbol in the shape of the tensors refers to batch_size, following tensorflow notation...
QUESTION
Is it worth trying this architecture? (I think it took me more time to draw the picture than coding it...)
The difference with typical seq2seq is that in the decoder, the input for the second time step is not the output of the previous step. The input for both time steps in the decoder is the same, and it is an "encoded" version of the all hidden states of the encoder.

Yes, it makes sense. Seq2seq models represent, in the RNN family, the best for multistep predictions. More classical RNNs, on the other side, are not that good for predicting long sequences.

If you need to implement a seq2seq model in TensorFlow 2.0 / Keras, each model follows the following structure:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, RepeatVector, Dense, TimeDistributed
from tensorflow.keras.activations import elu

# these are just made up hyperparameters, change them as you wish
hidden_size = 50

seq2seq = Sequential([

    LSTM(hidden_size, input_shape = (input_sequence_length, no_vars)),

    RepeatVector(prediction_length),

    LSTM(hidden_size, return_sequences = True),

    Dense(hidden_size, activation = elu)

    TimeDistributed(Dense(1, activation = elu))

])

and then train it as usual with seq2seq.compile() and seq2seq.fit().

If you want to stack more LSTM() layers on top of each other, simply add them to the model I depicted above. Please keep in mind that LSTM models are very computationally expensice; without a good GPU even this "basic" model could be painfully long to train.

One modification I'd suggest, looking at your image, is to make the LSTM-encoder and -decoder parts of equal size and depth.

Alternatively, you can implement a more classical "Autoencoder-like" architecture, with LSTM() layers for encoding and decoding, and Dense() layers in the middle. However, seq2seq models are the most powerful at the moment.

To my knowledge, the only models more state-of-the-art than this are attention models. The problem is that they are so much state-of-the-art that TensorFlow/Keras doesn't have built-in layers for them, and you'd have to create your own custom layers (it's a pain). The only extensive implementation of attention models I found is from this blog post, but things are going to be very very complicated here. I didn't try to implement one yet.


how can i detect medicine name and info(use and contents) by using medicine wrappers I got one project idea creating a Cross-platform react-native app
the project title is creating an app that can detect medicine name and other info from the medicine wrapper
I'm thinking of using Google Cloud Vision to extract all the text from the medicine wrapper.
The text contains medicine mfg info, address, pin code, expiry date, ingredients, and most importantly medicine name.
I want to extract only the medicine names and other info out of it.
What are the possible ways to get medicine name and other info?
and for rural people in India they can't spell the word correctly for that I can give one button that can spell that name for folks can you suggest me any ideas how can I do this and one important part
how/where can I get medicine wrapper ( Packaging ) of different medicines data????

For data, you can search on IEEE Dataport , Kaggle.

For detecting medicine names and other info trained deep learning models like CNN or you can also perform fine-tuning from the existing model.


What is the best way to visualize the relationship two categorical variables I am currently working on an ambulance dataset and one of my tasks is to find when a patient was misdiagnosed by the call dispatcher.

I have two codes; a dispatch code(what the dispatcher believes is wrong with the patient) and a hospital code(what a doctor decides the actual diagnosis is in hospital)

I'm looking for a way of visualizing the relationship between the two codes i.e. given ambulance code x what is the probability of each ambulance code being the outcome.

This can be computed using SQL pretty easily but I'm looking for a way to visualize/cluster it would be great. Any help would be appreciated.

EDIT:
Some good feedback in the comments

firstly the dimensionality:
Dispatch code can take 1 of 1722 unique values

hospital codes can take 1 of 1058 unique values

The two codes are entirely different an example would be as follows

Dispatcher 17D03:Unconscious
Hospital R41:Other symptoms and signs involving cognitive functions and awareness

My interest is in visualizing the relationship. So for example given a dispatcher code of unconscious what are the most common hospital codes?

Again this is fairly easy to calculate numerically but a visualization would make it easier to explain to my stake holders.

You can use a confusion matrix to generate the heat map of your data.

Suppose you have:

labels = ['cardiac arrest', 'choking', 'seizure']
dispatch_code = ['cardiac arrest', 'choking', 'seizure', 'choking', 'seizure', 'seizure', 'cardiac arrest', 'cardiac arrest']
hospital_code = ['cardiac arrest', 'choking', 'cardiac arrest', 'choking', 'seizure', 'seizure', 'seizure', 'cardiac arrest']

Then you can plot with:

from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt

cm = confusion_matrix(hospital_code, dispatch_code)
fig = plt.figure()
ax = fig.add_subplot(111)
cax = ax.matshow(cm)
plt.title('Confusion matrix of the classifier')
fig.colorbar(cax)
ax.set_xticklabels([''] + labels)
ax.set_yticklabels([''] + labels)
plt.xlabel('Predicted')
plt.ylabel('True')
plt.show()

This example shows that often there is a correct code coming from the dispatcher (yellow stripe in the middle). Cardiac arrest and seizure are often confused (blue) and chocking is never misclassified (purple).


Sklearn classification report is not printing the micro avg score for multi class classification model There are 6 class labels encoded as 0,1,2,3,4,5

While executing  classification report score it outputs accuracy,macro avg,weighted avg  .The micro average score  is missing in the output . Im not sure why micro average score is not getting printed . What should i do to get micro average score as well ?

print(classification_report(y_test, best_preds,labels=[0,1,2,3,4,5]))

          precision    recall  f1-score   support
       0       0.65      0.76      0.70        46
       1       0.74      0.56      0.64        41
       2       0.60      0.68      0.64        41
       3       0.65      0.59      0.62        41
       4       0.75      0.79      0.77        61
       5       0.76      0.70      0.73        40

accuracy                           0.69       270
macro avg      0.69      0.68      0.68       270
weighted avg   0.69      0.69      0.69       270

I went through the sklearn documentation and i changed the class labels from 0-5 to 1-6 to just simply see what scores are printed by classification report and to my suprise the micro average score was printed in the output but  the entire score in this classification report is wrong as the class labels are 0-5 and not 1-6 that is  why there is a warning printed in the output as well .

print(classification_report(y_test, best_preds,labels=[1,2,3,4,5,6]))

          precision    recall  f1-score   support
       1       0.74      0.56      0.64        41
       2       0.60      0.68      0.64        41
       3       0.65      0.59      0.62        41
       4       0.75      0.79      0.77        61
       5       0.76      0.70      0.73        40
       6       0.00      0.00      0.00         0

micro avg       0.70      0.67      0.69       224
macro avg       0.58      0.55      0.56       224
weighted avg    0.70      0.67      0.69       224

  UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.
    'precision', 'predicted', average, warn_for)
  UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples.
    'recall', 'true', average, warn_for)

The micro-averaged precision, recall, and F1-score are all the same, and are all equal to the overall accuracy (!) when including all the classes.

That fact is hinted at in the documentation for classification_report, in the "Return" field ("it is only shown for multi-label or multi-class with a subset of classes because it is accuracy otherwise").  It's explained in a little more detail at https://towardsdatascience.com/multi-class-metrics-made-simple-part-ii-the-f1-score-ebe8b2c2ca1 (near the end).


What kind of research can be done with an email data set? I found a data set called Enron Email Dataset. It is possibly the only substantial collection of "real" email that is public. I found some prior analysis of this work:

A paper describing the Enron data was presented at the 2004 CEAS conference.
Some experiments associated with this data are described on Ron Bekkerman's home page
Parakweet has released an open source set of Enron sentence data, labeled for speech acts.
Work at the University of Pennsylvania includes a query dataset for email search as well as a tool for generating spelling errors based on the Enron corpus.

I'm looking for some interesting current trend topics to work with.please give me some suggestions.

The following are some research that can done on e-mail dataset:

linguistic analysis to abbreviate an email message
Categorize e-mail as spam/ham using machine learning techniques.
identifying concepts expressed in a collection of email messages, and
organizing them into an ontology or taxonomy for browsing


How to replace words in a sentence with their POS tag generated with SpaCy efficiently? How is it possible to replace words in a sentence with their respective PoS tags generated with SpaCy in an efficient way?

You can not replace tags in a sentence because Python strings are immutable.

You can make another string with just the tags:

>>> import spacy
>>> nlp = spacy.load('en_core_web_sm')
>>> doc = nlp("Apple is looking at buying U.K. startup for $1 billion")
>>> " ".join(token.tag_ for token in doc)
'NNP VBZ VBG IN VBG NNP NN IN $ CD CD'

That example is based on the spaCy documentation.

If the tokens are split by several different non-space delimiters (called "multiple infix tokenization" in SpaCy), you would have to track that and the code would be more complex.


Which kind of model is better for keyword-set classification? There exists a similar task that is named text classification.

But I want to find a kind of model that the inputs are keyword set. And the keyword set is not from a sentence.

For example:

input ["apple", "pear", "water melon"] --> target class "fruit"
input ["tomato", "potato"] --> target class "vegetable"

Another example:

input ["apple", "Peking", "in summer"]  -->  target class "Chinese fruit"
input ["tomato", "New York", "in winter"]  -->  target class "American vegetable"
input ["apple", "Peking", "in winter"]  -->  target class "Chinese fruit"
input ["tomato", "Peking", "in winter"]  -->  target class "Chinese vegetable"

Thank you.

You can leverage word-vector similarity in embedding models.

TL;DR similiar vectors of words (for example fruits) will be clustered together in this high (vector) dimensional space. For every possible class-set you will have a class-set representative (centroid) that is actually a key (so in your case fruit, vegetable etc) all you need to do is train/find a representative word embedding model of your corpus.


is it possible determine a best neural network classifier by considering only accuracy Below, you have the accuracy plots for training and testing set for 6 different neural networks. is it possible to say, which of the following neural network classifier is better?. Having this little known information (training and testing accuracy by number of epochs plot) ? . Personally, I believe, the first classifier (top-left corner) is better because it shows that the accuracy of the testing data is stable as the number of epochs increases.

There are several issues you should consider before choosing the best classifier.

  Precision, recall and F-score

It is highly recommended that you also check precision, recall and F-score metrics in addition to accuracy. That's because if your dataset have imbalanced labels (for example when 90% of labels are positive and only 10% are negative) considering just accuracy may mislead you. For example in this case if you choose a classifier that always outputs 1, it will have 90% accuracy but it will fail to work well on unseen data. In such cases, precision, recall and F-score comes in handy.

  Validation and test set

In addition to the test set, you should set aside a validation set and tune the hyper-parameters of your model using this set. If you use the test set to tune the hyper-parameters of your model, you can not expect your model to generalize well so it performs poorly on unseen data.

  Overfitting and underfitting

Overfitting happens when your model performs dramatically well on training dataset but poorly on validation and test set. Underfitting happens when your model performs poorly on training dataset. To ensure that you are not underfitting or overfitting, you should check the performance of your model on the training and validation datasets, simultaneously.

So, first you should change your accuracy plots so it shows training and validation loss instead. Then, we consider choosing the best model. If after some epochs, the training score is still low, you are underfitting. If training score is much higher than validation score, you are overfitting. The best case is when training and validation scores are close together after enough epochs.


Cannot make user directory on a new CDH5 installation (Hadoop) I download and installed CDH 5 package succesfully on a single linux node in pseudo-distributed Mode on my CentOS 6.5

Starting Hadoop and verifying it is Working Properly as in this link

I succesfully finished the following steps

Step 1: Format the NameNode.

Step 2: Start HDFS

Step 3: Create the /tmp Directory

Step 4: Create the MapReduce system directories:

Step 5: Verify the HDFS File Structure

Step 6: Start MapReduce

while following command in step 7 I get the following error.

Step 7: Create User Directories

$ sudo -u hdfs hadoop fs -mkdir -p /user/hadoopuser

mkdir: '/user/hadoopuser': No such file or directory

(where hadoopuser is my linux login username)

If I create the directory manually as /user/hadoopuser in the filesystem, it is not accepting.

How to success the step 7:?

Please provide the sloution to procced the remaining installation.

The simplest way to setup a user directory is to login to Hue as an admin and create a user account. This will create a directory in HDFS located at /user/yourusername with the correct permissions to work in.


k-means: Only one-dimensional cluster predictions in two-dimensional space For this dataset, it seems that the predictions of my k-means model only consider the horizontal axis, although the cluster centers seem reasonable.

Is something wrong with this classification? Please note the color of the grid in the background.

I use scikit-learn, here is the code fragment of classification and visualization.

model = KMeans(n_clusters = 5)
model.fit(df_stuff[['Stuff','Other Stuff']])

fig = plt.figure()
ax = fig.add_axes([0,0,1,1])

ax.scatter(df_stuff['Stuff'], df_stuff['Other Stuff'],c=model.labels_,s=80,cmap='rainbow')
ax.set_xlabel('Stuff')
ax.set_ylabel('Other Stuff')
ax.set_title('Strange Clusters')

# Draw Cluster Centers
for center in model.cluster_centers_:
    ax.scatter(center[0],center[1],c='black',s=5120,alpha=0.2)

# Draw Cluster Grid
cluster_grid = {'x': [], 'y': [], 'cluster': []}
for x in np.linspace(df_stuff['Stuff'].min(),df_stuff['Stuff'].max(),25):
    for y in np.linspace(0.35,0.6,25):
        cluster_grid['x'].append(x)
        cluster_grid['y'].append(y)
        cluster_grid['cluster'].append(model.predict([[x,y]])[0])

ax.scatter(cluster_grid['x'],cluster_grid['y'],c=cluster_grid['cluster'],cmap='rainbow',alpha=0.4,s=10)

KMeans does correctly do what it is supposed to do.

Just plot your data correctly, with the same scale on both axes...

Y deviations do not matter, they are tiny compared to the X axis. Deviations there are 100x larger, so squared deviations even 10000x. Since KMeans minimized squared errors, only x matters

When plotted correctly, your data more looks like this:


Performance difference between decision trees and logistic regression when one of the features is a string I have a set of features, one of which is a string. I convert the string to an integer by treating the string as a base 36 number (I only use the first 13 characters). Then I can use DecisionTrees since in the sklearn implementation you need to convert it to a number. When I tried a different model, say Logistic Regression, performance drops drastically, say from 80% to 30% accuracy.

I might have accepted this result if I had been able to use the strings as such in the DecisionTrees model, but since I used the same string to integer conversion for both models, why such great difference?

I cannot go into the details, but let me provide you with an analogy. Let's say you are classifying millions of objects by their usefulness. So you say hammers are 4, screwdrivers 6, washers 10, etc. Of course you have more than one screwdriver, and sometime you forget and give it a value of 5, or something else. The model goes through millions of example, and then makes a prediction about the number for each object. I converted the names into integers, as I explained, and decision trees gives me an 80% accuracy, linear regression 30%. I assume that the problem is that linear regression tries to figure out some mathematical rule that does not exist. But why is decision trees immune from this problem?

String data can be either categorical (where you have e.g. more than 10 examples of each string) or free text.  If it's the former, a decision tree can deal with it no problem. You don't have to convert it into a numeric.

For regression you cannot directly use categorical variables.    If you want to use them in a regression, you will need to create dummy variables to encode the values. e.g. if your categories are "Red", "Yellow", "Blue" for the colour variable, you create variables "Red" (which will take a 1 or a 0) and "Yellow" (which takes a 1 or 0).  If both are 0, the colour must be "Blue".  There are functions in sklearn to do this automatically.

If your string is just free text then you will need a better way of grabbing information out of it.  You can use text mining such as tokenizing, TF-IDF etc. to convert it into numerical and categorical information that can be fed into a classifier.


Similarity measure for multivariate time series with heterogeous length and content I am interested in clustering multivariate N time series of T'values' each(different lengths) using python. Each variable have many trends and values which are simultaneously numeric and nominal.

A sample $T_{i}$ in the dataset has the following format:

TimeStamp       | Sensor0 | Sensor1| Sensor2
2015-02-05 11:30|<Min     | On     | off
2015-02-05 11:31|<Min     | on     | off
2015-02-05 11:32| Action2 | 10     | 0.0001
2015-02-07 11:33| Action2 | 10     | 0.00012
2015-02-07 11:34| Action2 | 10     | 0.00012
2015-02-07 11:35| Action2 | 20     | 0.00015

Another sample $T_{j}$ in the dataset has the following format:

TimeStamp       | Sensor0 | Sensor1| Sensor2
2015-10-05 11:30| Action2 | 11     | off
2015-10-05 11:31| Action1 | 11     | off
2015-10-05 11:32| Action2 | NAN    | 0.0001
2015-10-07 11:33| Action3 | NAN    | 0.00012
2015-10-07 11:34| <Min    | 10     | 0.00012
2015-10-07 11:35| <Min    | 15     | on

For the missing values (not numeric), they were not collected by the sensors so my idea was to replace them by minimum values., given that all values are strictly positive.
Otherwise, they would be considered as missing values. In which case the problem would be of finding a similiraty measure that can compare missing values (off,on..) and numeric values.

I am wondering if there is a similarity / distance measure already exist in the litterature to compare such multivariate timeseries, with hetergonuos lengths, and whether this kind of problem has already been formulated in the papers, books or else for R and python.

Thanks for your advice.

Try this recent paper: Consistent Algorithms for Clustering Time Series.

Your question is very much a current research topic.

Here's an older but excellent paper which talks about the fundamentals: Generalized Feature Extraction for Structural Pattern Recognition in Time-series Data.


Simple ANN visualisation TLDR: Please help me understand the graph representation of the network in the image below.

Hi, this is pretty stupid, but I'm just have trouble visualising what I'm actually doing with this neural network.

I've read about neural networks and multilayer perceptrons for some time and I'm just getting started with actually using them. I started with a super simple example, just to get warmed up but now I've confused myself.

I artificially generated some data and used nntools in matlab to attempt to "predict" the results. I built a neural network with the following parameters:

feed forward backprop network.
Gradient Descent training algorithm.
Gradient Descent learniing algorithm.
Performance/loss function of mean squared error.
two layers:

first with three neurons and Tansig activation function.
the second with one neuron and linear activation.

I end up with something looking like this:

However, I don't know what this actually represents, I'm all sorts of confused right now. Could someone please explain/upload an image/draw some ascii to represent the neurons and edges in the above network? It would really help clear my head.

Currently I think it's like this:

           T  L
           o
          / \
         /   \
   IN > o--o--o--o > OUT
         \   /
          \ /
           o

With linear activations in columns L and Tanh activations in columns T. Is that right? Doesn't make sense to me.

I believe this is the representation you're after, please excuse the rough sketch but I think it explains the structure appropriately.

Single input going to three hidden units, each with a bias and tansig activation.
The outputs of the hidden layer are summed (via linear activation) with a bias to produce the output.


What is the One Max Problem in detail? I am looking a python lib named deap, but stuck at beginning.

The first paragraph says:

  The problem is very simple, we search for a 1 filled list individual.

What does 1 filled list means? Search a 1 filled list, from where? individual list or individual 1?

Google One Max Problem only gives some information which seems to be useful:

There is a Max One Problem

I can understand this, but is it same as One Max Problem? If so, I have a question that why need evolutionary algorithm to "evolve" our population until eventually the target emerges. ? If I am a Medical Researcher, I should already have the entire DNA(gene list) , what I need to do is just search in that list, not evolve a random list.
There is a Maximum_satisfiability_problem

This is understandable too, seems relate to One Max Problem , another saying?

deap is an evolutionary algorithm library. In an evolutionary algorithm you usually want to optimize a function. For this, you define individuals as a collection of genes (e.g. a string of numbers) that condense a possible solution, you create a population of such individuals, and define a fitness function to evaluate how good they are; then you apply evolutionary operators (e.g. mutation, pairing) to evolve the population, effectively searching the solution space.

The problem referred to as "max one problem" in the linked page can be reworded as:

  let's create a toy evolutionary algorithm where we want to evolve a population of individuals (where each individual is a list of N integer numbers) until one of them is exactly comprised of N ones (i.e. 1,1,....,1).

The "maximum satifiability problem" is not related to this.


Should I seperate the data for a k-means model? I would like to cluster some user reviews and I'm doing this with k-means. In my dataset I have English and German reviews. Is this manipulating the cluster result if I don't seperate them? Or should I do a k-means prediction on each language?

Yes you should definitely split these two up. When calculating the TF-TDF matix, it will give different terms to objects of the same entity, because of the difference in language. This will affect your clustering results.


Wich GPU-card to make deep learning with a MacBook Pro? I'm a owner for a MacBook Pro. I want to accelerate my deep learning models, because with the CPU, it's very slow !
Could you tell me which GPU card I can buy for it ?
Thanks.

I also have a Macbook and I do data science on it. I looked at this same issue about  1 year ago and it's very possible that the landscape has changed since then.

But, at that time, I came to the conclusion that my best and most cost-effective options were to either (1) get a notebook with a GPU (of which there are many now due to the popularity of e-sports) or (2) get some space on AWS and their GPU servers. I really couldn't find an "enhanced Mac" option that would be as quick or work with as many tools as the above 2 options can offer you.

FYI - on the surface, the AWS option may seem expensive. But if you write your code locally and test on smaller samples and then only move that code to AWS when you need to do the deep learning phases (and only keep the AWS server running for those minutes) it can be a surprisingly affordable way to go.


What is the difference between global and universal compression methods? I understand that compression methods may be split into two main sets:

global
local

The first set works regardless of the data being processed, i.e., they do not rely on any characteristic of the data, and thus need not to perform any preprocessing on any part of the dataset (before the compression itself). On the other hand, local methods analyze the data, extracting information that usually improves the compression rate.

While reading about some of these methods, I noticed that the unary method is not universal, which surprised me since I thought "globality" and "universality" referred to the same thing. The unary method does not rely on characteristics of the data to yield its encoding (i.e., it is a global method), and therefore it should be global/universal, shouldn't it?

My primary questions:

What is the difference between universal and global methods?
Aren't these classifications synonyms?

Consider the following chunk of data:

1010010110100101

Universal - these are generic compression algorithms that are data agnostic. A crude version of run length encoding would fall into this category. The advantage is that it is very fast to compress and decompress. The downside is that it may be extremely ineffective based on the data to be compressed.

1111111111111111 -> 16 1 (lucky case)

1010010110100101 -> 1010010110100101 (unlucky case)

Local - this method would consider smaller segments of a fixed length, say 4, look for patterns and compress them. Eg. This data only contains these two types of patterns - 1010 and 0101. These patterns can be represented as 0s and 1s and the overall data will be a table representing the mappings, and something like 0101. This has the potential to result in a much smaller compressed size.

1010010110100101 -> 1010 0101 1010 0101 -> 0101 (0=1010,1=0101)

Global - this method would look at the entire data and find the optimal / much better patterns to compress the data. The example data contains just one pattern 10100101 and represent it as 00 along with the mapping table. This has the potential to obtain the smallest possible compressed size, but is also computationally the heaviest.

1010010110100101 -> 10100101 10100101 -> 00 (0=10100101)