Balance data using different criteria I have a dataset of audio and text files that I want to balance using different criteria to train a neural network. The text and matching audio file are grouped under one ID.

For each ID, I have a number of words and some metadata.

Let's say my data have three metadata/criteria that need to be balanced (well... kind of, I want to be able to choose the percentage for some criteria values) :

The name of the speaker, each speaker should have approximately the same number of words.
The status of the speaker (active if the speaker is still in activity or inactive if not), where I want approximately 70 % active speakers and 30 % inactive.
The gender of the speaker, where I want approximately 50 % male and 50 % female.

Example : ID12345 with 10,000 words. Metadata : John, male and active

If I balance one of the criteria, then another, I will surely unbalance the previously balanced one.
Moreover, if I balance using only one criterion at a time, I could throw away data that could be useful to the balance of the following criteria, which mean I will end up with less data that I could have had optimally.

Is there an algorithm that could balance the data using all criteria with their percentage target, and maximise the number of words remaining ?

I'm looking for a general algorithm, with a variable number of criteria.

EDIT : I use C++ and MongoDB (but I'll accept other languages/tools if it can help me, as well as algorithms).

You would need to balance the data according to the joint distribution of all 3 features.

One simple option is to treat each feature as categorical (binning any continuous features), then map each combination of features to a single new feature:

doc_len_tokens  | status   | gender | level
----------------|----------|--------|------
1               | active   | male   | 1
1               | active   | female | 2
1               | inactive | male   | 3
1               | inactive | female | 4
2               | active   | male   | 5
2               | active   | female | 6
2               | inactive | male   | 7
2               | inactive | female | 8
... etc ...

You would then stratify on level when sampling.

Here is an example in SQL:

SELECT
    data.unique_id,
    data.doc_num_tokens,
    data.status,
    data.gender
FROM (
    SELECT
        data.unique_id,
        data.doc_num_tokens,
        data.status,
        data.gender,
        row_number() OVER (
            PARTITION BY
                data.doc_num_tokens,
                data.status,
                data.gender
            ORDER BY
                newid()
        ) AS row_num
    FROM
        my_data AS data
) AS data
WHERE
    row_num <= ${number of rows to sample}


R: Comparing dissimilarity between metabolic models with discrete wavelet transformation I’m working on comparing bacteria metabolic models. Each model has a set of metabolites (around 2000) and their concentration for 200 time points. I’m in the process of comparing the models to cluster them based on their similarity. One method I followed is I did a pair wise comparison for each of the metabolite pairs in two models using Euclidean distance. Below is how my data look like. This is a sample data file.

I computed pair wise Euclidean distance for Met1 from Model A and Met1 from Model B. Likewise computed the distances for all the common metabolites between the 2 models (Met4 in Model A and Met4 in Model B) and summed up the distances to get a distance (dissimilarity) between the two models. Similarly I computed the dissimilarity matrix for all the models and I used hierarchical clustering to cluster them.

As mentioned above, now I want to compute the dissimilarity of the models using Discrete Wavelet Transformation as my distance measure. I would like to know how to use Discrete Wavelet Transformation to compute a dissimilarity distance between 2 time series and hence for my models.

Previously I used DWT as a distance measure with DBSCAN for clustering metabolites in one model according to their behavior. It worked fine.

Take a look at the TSclust package. Here how you would apply it to your sample data.

require(TSclust)

#read in the data
model_a <- read.csv("~/Desktop/Model A.csv", header = TRUE, stringsAsFactors = FALSE)
model_b <- read.csv("~/Desktop/Model B.csv", header = TRUE, stringsAsFactors = FALSE)

#data must be in rows rather than columns
model_a <- as.data.frame(t(model_a))

model_b <- as.data.frame(t(model_b))

#calculate dissimlarities between metabolites in models 1 and 2
met1_DWT.diss <- as.numeric(diss.DWT(rbind(model_a['Met1', ], model_b['Met1', ])))
met1_DWT.diss
[1] 90.80332

met2_DWT.diss <- as.numeric(diss.DWT(rbind(model_a['Met2', ], model_b['Met2', ])))
met2_DWT.diss
[1] 1.499241


Error: float() argument must be a string or a number, not 'StandardScaler' I am getting this error for the mentioned code. Please help me with this as I am a beginner and learning machine learning

# Polynomial Regression

# Importing the libraries
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

# Importing the dataset
dataset = pd.read_csv('Position_Salaries.csv')
x = dataset.iloc[:, 1:2].values
y = dataset.iloc[:, 2].values

#Feature scaling
from sklearn.preprocessing import StandardScaler
sc_x=StandardScaler()
sc_y=StandardScaler()
x=sc_x.fit(x)
x=sc_x.fit(y)

#Fitiing SVR to the dataset
from sklearn.svm import SVR
regressor = SVR(kernel = 'rbf')
regressor.fit(x,y)

#Predict the results
regressor.predict(np.array([6.5]).reshape(1, 1))

#Visualizing the results
plt.scatter(x,y,color = 'red')
plt.plot(x,regressor.predict(x))
plt.label("Positon Level")
plt.label("Salary")
plt.show()

.fit method returns the standard scalar object. You are using that to train the model. please use fit_transfor or transform after the fit. like below

sc_x.fit(x)
x = sc_x.transform(x)

or

x = sc_x.fit_transform(x)


Natural Language to SQL query I have been working on developing a system "Converting Natural Language to SQL Query".

I have read the answers from the similar questions, but was not able to get the information that I was looking for.

Below is the flowchart for such system which I have got from An Algorithm to Transform Natural Language into SQL Queries for Relational Databases by Garima Singh, Arun Solanki

I have understood till part of speech tagging step. But how do I approach the remaining steps.

Do I need to train all the possible SQL queries?
Or, once part of speech tagging is done, I have to play with the words and form a SQL query?

Edit: I have successfully implemented the from step "user query" to "Part of speech tagging".

Thank you.

If you want to tackle the problem from another perspective, with an end to end learning, such that you don't specify ahead of time this large pipeline you've mentioned earlier, all you care about is the mapping between sentences and their corresponding SQL queries.

Tutorials:

How to talk to your database

Papers:

Seq2SQL: Generating Structured Queries from
Natural Language using Reinforcement Learning  salesforce
Neural Enquirer: Learning to Query Tables in Natural Language

Dataset:

A large annotated semantic parsing corpus for developing natural language interfaces.

Github code:

seq2sql
SQLNet

Also, there are commercial solutions like nlsql


how to encode labels for relationship extraction I am trying to extract relationship from text. So, lets take the following text
"He went to movie. but, they went to school"

So, here the relationship's are "He and Movie", "they and school".
How can i possibly encode these as labels. my initial thought was to mark all pronouns as 1's and nouns as 2's. But, is there a better way to do this?.

I would reommend parsing the sentences using something like spaCy, which will be able to build such relationships for you.

It will then be a matter of extracting the realtionships and deciding how you want to label them. Let's work through a small example. Imagine your big block of text contains many sentence. We first parse the big block, then could (optionally) extract just the sentence that we care about:

import spacy

nlp = spacy.load('en_core_web_sm')      # this defines our 'parser' for English
doc = nlp(big_block_of_text)            # you may want to clean the text beforehand

Assume we only want sentences that contain the word 'went':

sents = [sent for sent in doc.sents if 'went' in sent.string]

Here is an example of grammatically walking down the parse tree. I say semantically because we are using the grammar parts-of-speech to conditionally navigate (using e.g. nouns):

for sent in sents:                          # loop over each sentence
    for word in sent:                       # analyse each word
        if word.pos_ == 'NOUN':             # if it is a noun
            for child in word.children:     # find children (go down the tree)
                if child.pos_ == 'NOUN':    # select only nouns from the children
                    print(word, child)

I believe this snippet would extract the pairs in your given example, so the output would hopefully indeed be:

"He movie"
"they school"

There are many other parts-of-speech that you will be able to use to be more specific or match other use cases - check out the relevant spaCy docs.

How to actually encode the relationships between your target words will depend on their final purpose, i.e. what you eventually want to do with the encodings. You now have the noun pairs. If you want to retain the knowledge of where they actually came from (from which sentences), you could indeed build a hierarchy, that keeps a code for the origin sentence and then a code for each pair of words.


KL divergence in VAE If I understand correctly KL-divergence is relative entropy of two distributions. To calculate KL divergence of two distributions, you would need two vectors of random variables.

What I do not understand it, how you can calculate KL divergence in VAE (latent space vector and N(0,1) as it is stated in many tutorials.

Latent space vector is not a vector of random variables. It is a vector of product of input, weights,bias and activation functions. All these do not make your vector a random variable vector. My question is, how to properly create latent space vector as random variable vector, so you could eventually calculate KL divergence.

You are right that the output of your encoder neural network is not a random variable, this is the mean $\mu$ and the standard deviation $\sigma$ of your latent random variable. For example if the output of your encoder is $\sigma = 1$ and $\mu = 0.5$, your latent random variable will be normal with mean 0.5 and standard deviation 1. Theb, you can calculate the KL divergence between your random variable and the prior for the latent random variable which is $\mathcal{N}(0, 1)$ in many tutorials.


Automated Function (python) I am trying to create a function that automates the process of taking a CSV file, splits in the data in features and responses, apply different models (regression) to the data and score them according to some metric such as MAE, RSME, etc. Model parameter should be easily interchangeable. Here is a little of what I have so far.

import numpy as np
import pandas as pd

def get_data(file_name):

data = pd.read_csv(file_name)

para = int(raw_input('Enter the no of parameters to be used '))
print(para)

param= []

for k in range(0,para-1):
    param[k]= raw_input('Enter the parameter')

rec = int(raw_input('Enter the no of records in the dataset '))
print(rec)
x_parameter = []
y_parameter = []
x1= []
for i in range(0,para):
    for x1[i] in data[i]:
        x_parameter[i].append(x1[i])

for j in range(0,rec):
    print x_parameter[j]
    print y_parameter[j]

get_data('C:\Users\Douglas\Desktop\trainingset.csv')

from sklearn import model_selection
from sklearn import linear_model
from sklearn import SGDRegressor
from sklearn.tree import DecisionTreeRegressor
from sklearn.neighbors import KNeighborsRegressors
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.svm import SVR

dataframe = getdata(file_name)
array = dataframe.values
X = array[]
#confirguartion for cross validation
seed = 7
#prepare models
models = []
models.append(('LR', LinearRegression()))
models.append(('SGDR', SGDRegressor()))
models.append(('KNR', KNeighborsRegressor()))
models.append(('DTR', DecisionTreeRegressor()))
models.append(('GBR', GradientBoostingRegressor()))
models.append(('SVR', SVR()))
#evaluate each model in turn
results = []
names = []
scoring = 'neg_mean_absolute_error'
for name, model in models:
    kfold = model_selection.KFold(n_splits=10, random_state=seed)
    cv_results = model_selection.cross_val_score(model, X, Y, cv=kfold, scoring=sc)
    results.append(cv_results)
    names.append(name)
    msg = "%s: %f (%f)" % (name, cv_results.mean(), cv_results.std())
    print msg

Your code looks decent, although this is not really the place to ask these types of questions. You're looking to write a wrapper around SK-Learn,

What I would try to do is to write a method to handle different types of datasets(json or csv) and then call it on a filepath and then have a function which reads in a set of models and applies each of them to the dataset in question, and finally uses metrics present in sklearn to get some results.

There is auto-sklearn which does this for you and they have a pretty solid codebase that is well documented. Here is a link for your convenience (https://automl.github.io/auto-sklearn/stable/)


Understanding LSTM structure I am trying to learn LSTM's, and struggling a bit with the structure and the inputs/outputs of LSTM layers. Say I have a network definition like this:

timesteps = 50
features = 10
inp = Input(shape=(timesteps, features,), name="input_layer")
x = LSTM(128, return_sequences=True, name="lstm1")(inp)
x = LSTM(64, return_sequences=False, name="lstm2")(x)
x = Dense(128, activation="relu", name="dense1")(x)
x = Dense(64, activation="relu", name="dense2")(x)
x = Dense(1, activation='sigmoid', name="output")(x)
model = Model(inputs=inp, outputs=x)
model.compile(loss='binary_crossentropy', optimizer='adam')

I can find plenty of resources on the internet on the internal structure of the LSTM cell. For example, this has the figure with one input and some outputs.
But I would first like to understand at a higher level, what inputs and outputs an LSTM layer gives in different configurations.

Lets say I have input data shape of (1000,50,10).
So 1000 observations, each with 50 timesteps for 10 features.

Questions:

My understand is, the first layer ("lstm1") would take the input as shape (50, 10).
One at a time for each of the 1000 observations.
How are the 10 features input into the LSTM layer?
Is it similar to a Dense layer, so each input feature goes into each LSTM cell with some weight?
This answer seems to indicate so, but I do not quite understand what are the multiple inputs for each LSTM "unit" in that answer (perhaps even what "unit" is).

And the internal workings.
Does "lstm1" here process all the 50 timesteps at once and then produce an output?
After it is done for all 50 timesteps the next layer gets its input from this output?
And what is the shape of that output? Is it shape=(50, 128)?
So is the set of output "features" in this case the number of units in the LSTM layer?
And because of "return_sequences=True", it also outputs the intermediate value at each timestep to produce the shape=(50,XXX) outputs?

This leads to my question on the "lstm2" layer.
Using "return_sequences=False", does it now output a single array of 64 values? So shape=(64)?
No value for every timestep but rather only one per unit at the end of the "unrolled" cell loop?


Can a trained recognition model be used to generate a sample? Suppose we have trained a cat classification network. It takes in an image (as a vector) x and returns $\hat{y}\in(0,1)$. The loss function is the typical cross entropy function. Shouldn't it be possible to now perform gradient descent on the space of images to obtain an example of a "cat picture" that our network really thinks is a cat?
I expect that this process would give us something very ugly, and not actually cat-like, but I am just curious if this works.

Generative models can. Among others, you can use GAN, auto-encoders or VAE, ... As far as I know, discriminative approaches can not generate a sample.
Hope it helps.


why accuracy on training dataset is 100% using random forest caret , does it indicate overfitting? I tried to predict on training set but i got accuracy of 100%. However on the testset, i got an accuracy of 62%. Should i be worried of getting high accuacies on training dataset using caret?
why accuracy on training dataset is 100% using random forest caret , does it indicate overfitting?

Yes, those accuracies dont say anything.

You have overfitted to 310 data points which is pretty easy using RF. The fact is that you said you "predicted" on training data. Well, when you learned on training data and you ask prediction for points you already gave the model it simply knows them.

You need to use a cross validation technique (here i propose Leave-One-Out technique as you dont have much data points). RF is more robust than DT towards overfitting, but that is not applied to this setting. Here you getting an exam in which you ask a question you already gave to the student. He just simply knows the answer.


What does it mean for the training data to be generated by a probability distribution over datasets I was reading the Deep Learning book and came across the following para (page 109, second para):

  The training and test data are generated by a probability distribution over datasets called the data-generating process. We typically make a set of assumptions known collectively as the i.i.d. assumptions. These assumptions are that the examples in each dataset are independent from each other and that the training set and test set are identically distributed, drawn from the same probability distribution as each other. This assumption enables us to describe the data-generating process with a probability distribution over a single example.The same distribution is then used to generate every train example and every test example. We call that shared underlying distribution the data-generating distribution, denoted $p_{\text{data}}$. This probabilistic framework and the i.i.d. assumptions enables us to mathematically study the relationship between training error and test error.

Can somebody please explain to me the meaning of this paragraph?

On page 122 the last paragraph, it also gives an example

  a set of samples $\{x(1), \dots, x(m) \}$ that are independently and identically distributed according to a Bernoulli distribution with mean $\theta$.

What does this mean?

There is a common assumption that data that is being modeled is independent and identically distributed (i.i.d.) samples from a probability distribution. There is the same underlying probability distribution for both the training and test datasets. And each sample is independent of the other samples.

Examples of these assumptions being violated:

Data is generated by a completely random process such as random walk.
The training and test dataset come from different probability distributions, either completely different probability distributions or the same probability distribution with different parameters.
Samples are not independent. An example of dependent samples is dealing cards from a single deck, the probability of later cards is dependent on previously dealt cards.

The model fitting process only has access to the data samples, not the underlying probability distribution. Parametric modeling fitting makes a guess about the functional form of that probability distribution (e.g., Bernoulli or Gaussian) and then, estimates the associated parameters.


Convolutional Neural Networks - When do I adjust parts of it, and how? I'm still training a Convolutional Neural Network model in Tensorflow to recognize age groups from facial images. I'm using the Adience 3D (frontalized with a 3D model - it might not be good data) dataset, with four folds for training, and one fold for testing.

I'm currently using three Conv-Conv-Pool layer sets, the first one's Conv layer using 32 filters with 5x5 size, the second and third using 64 filters with 3x3 size, and to top it all off, a Dense layer with flattened input, then a Dropout layer, dropping out the outputs of the dense layer by 20%, then on to the logits layer.

I normalized my cropped color images with the per-channel normalization - the (A - np.mean(A)) / np.std(A) method for each 64x64 array in the 64x64x3 input.

Currently I'm training it with Batch Size 64 and 100 steps so as to evaluate the weights every 100 steps, and while I don't see much improvement for now, I could see it increasing from steps 100 to 500 (0.27 to 0.35), before dropping on stage 600 then going up again up to step 800 (0.30 - 0.34).

Is this a sign I have to change my layer structure?

On that note, when should I change up my layer structure/hyperparameters like training steps, batch size, filter size, etc., and how?

If you wanna get the optimal parameters, you can try using grid search. A grid search or parameter sweep does an exhaustive searching through a specified subset (you have to define this subset) of the hyperparameter space of the learning algorithm which in your case is the Convolutional Neural Network. Although this may take a long time but this is one of the ways you'll know the good parameters for your model.

Try searching for optimization algorithms or parameter tuning techniques.


Looking for research for separating conversational audio files I have been looking for conversational audio speech separation. Looking for researches around this topic, I came to across asteroid framework that was implemented on LibriMix dataset for audio speech separation where two voices are overlapping on each other (You can listen to such a sample here). But when we talk about conversational speech data (e.g. two people are conversing with each other) the audio of those people do not overlap with each other. Instead, people take turns in speaking and speak (Here is a sample).
While looking online for research on audio separation for conversational audio, I wasn't able to find one. Datasets like WSJ, LibriMix, etc do exist, but they are not designed for conversational problems and are more oriented towards the problem overlapping voices.
If there is any project, dataset or research around this that one knows of, it would be really helpful.

The task of separating a conversation with multiple speakers taking turns is called Speaker Diarization. This is different from Speech Separation, which is a special case of Source Separation, refers to separating out multiple overlapping/concurrent sources from an audio stream.
Some free datasets include

ICSI Meeting Corpus
AMI Meeting Corpus
VoxConverse

More can be found in the Github project wq2012/awesome-diarization.


Question about K-Fold Cross Validation In a machine learning procedure, suppose we've chosen k=10 for the "K-Fold Cross Validation".
After we do the k steps of "K-Fold Cross Validation", how do we choose the final model for the classifier ? (The one will we use in order to predict new data)

K-fold cross validation only helps you giving an estimation of the error you are going to make with a given model. K-fold can be used in order to tune hyperparameters: we have a model that depends on some hyperparameters, we compute the K-fold validation error using different values of the hyperparameters, and we take the model with the lowest cross-validation error. Finally, we train the model with the chosen hyperparameters with all the training data and we build a classifier based on that model.


How to convert a pytorch geometric graph to Networkx Multigraph? I have a weighted graph stored in Data object, and I want to convert this graph to Networkx MultiGraph. Here is what I have tried:
data = Data(x=x, edge_index=edge_index)
print(edge_index).

.

torch.Size([3, 237])

conversion
first trial
from torch_geometric.utils import to_networkx, from_networkx
G=to_networkx(data, to_undirected=True)

ValueError: too many values to unpack (expected 2)

second trial
G = nx.MultiGraph()
edges = data.edge_index.numpy()
edge_list = []
for i in range(data.num_edges):
    edge_list.append(tuple(edges[:, i]))
G.add_edges_from(edge_list)
data.G = G
adj=nx.adjacency_matrix(G)
print('here the adj is', adj.shape)

Result=(18, 18) expected (23,23)

What else can I do? Thank you very much for your Input

Check out the provided utility method torch_geometric.to_networkx (link to docs).
I am not sure how this would handle multigraphs.
There's also a from_networkx.


Does "feature importance" depend on the model type? I was working on a small classification problem (breast cancer data set from sklearn), and trying to decide which features were most important to predict the labels. I understand that there are several ways to define "important feature" here (permutation importance, importance in trees ...), but I did the following: 1) rank the features by coefficient value in a logistic regression ; 2) rank the features by "feature importance" from a random forest. These don't quite tell the same story, and I'm thinking that a feature that might be "unimportant" in a linear model could be very discriminative in a non-linear model that can "understand" it.
Is that true in general? Or should "important" features (those that contribute most to a classification score) be the same across all types of models?

When it comes to feature importance I always go with a model-agnostic measure, as you well mention if you have two different models, they will interpret importance in different terms (Linear models as the coefficient and Tree-based models as the information gain/impurity decrease on each feature.
So you already mention one measure that does not depends on the model, rather on the metric you are interested in; Permutation importance does not care about what model you are using, but the impact that a feature has on the global performance.
This reference might give you a better idea of the advantages of using permutation importance over tree-based models importance Permutation Importance vs Random Forest Feature Importance


Normalization for a 2d input array I am new to machine learning and trying to apply it to my problem.
I have a training dataset with 44000 rows of features with shape 6, 25. I want to build a sequential model. I was wondering if there is a way to use the features without flattening it. Currently, I flatten the features to 1d array and normalize for training (see the code below). I could not find a way to normalize 2d features.
dataset2d = dataset2d.reshape(dataset2d.shape[0],
                              dataset2d.shape[1]*dataset2d.shape[2])
normalizer = preprocessing.Normalization()
normalizer.adapt(dataset2d)
print(normalizer.mean.numpy())

x_train, x_test, y_train, y_test = train_test_split(dataset2d, flux_val,
                                                    test_size=0.2)

# %% DNN regression multiple parameter
def build_and_compile_model(norm):
    inputs = Input(shape=(x_test.shape[1],))
    x = norm(inputs)
    x = layers.Dense(128, activation="selu")(x)
    x = layers.Dense(64, activation="relu")(x)
    x = layers.Dense(32, activation="relu")(x)
    x = layers.Dense(1, activation="linear")(x)
    model = Model(inputs, x)
    model.compile(loss='mean_squared_error',
                  optimizer=keras.optimizers.Adam(learning_rate=1e-3))
    return model

dnn_model = build_and_compile_model(normalizer)
dnn_model.summary()
# interrupt training when model is no longer imporving
path_checkpoint = "model_checkpoint.h5"
modelckpt_callback = keras.callbacks.ModelCheckpoint(monitor="val_loss",
                                                     filepath=path_checkpoint,
                                                     verbose=1,
                                                     save_weights_only=True,
                                                     save_best_only=True)
es_callback = keras.callbacks.EarlyStopping(monitor="val_loss",
                                            min_delta=0, patience=10)
history = dnn_model.fit(x_train, y_train, validation_split=0.2,
                        epochs=120, callbacks=[es_callback, modelckpt_callback])

I also tried to modify my model input layer to the following, such that I do not need to reshape my input
inputs = Input(shape=(x_test.shape[-1], x_test.shape[-2], ))

and modify the normalization to the following
normalizer = preprocessing.Normalization(axis=1)
normalizer.adapt(dataset2d)
print(normalizer.mean.numpy())

But this does not seem to help. The normalization adapts to a 1d array of length 6, while I want it to adapt to a 2d array of shape 25, 6.
Sorry for the long question. You help will be much appreciated.

Dense layers inherently work with 1d-data and lose any positional "importance" of your input data (e.g. the location of a person in an image, how digits come together to form a number). Theoretically, you'll achieve the same training output if you decided to randomly shuffle the positions of your input features. If you want to stick with a Dense model as you've provided, I believe that normalizing on a 2d array would not benefit your training.
But it sounds like the position of each feature in the 6x25 matrix is important for your predictions. In that case, I would suggest switching from a Dense model to a convolutional model, which incorporates the position of each feature during training. Here's an example. In this case, here's a useful list of normalization techniques for 2D or 3D data inputs.
Please let me know if I can clarify anything. Good luck!


Are decision tree algorithms linear or nonlinear Recently a friend of mine was asked whether decision tree algorithms are linear or nonlinear algorithms in an interview. I tried to look for answers to this question but couldn't find any satisfactory explanation. Can anyone answer and explain the solution to this question? Also, what are some other examples of nonlinear machine learning algorithms?

A decision tree is a non-linear mapping of X to y. This is easy to see if you take an arbitrary function and create a tree to its maximum depth.

For example:

if x = 1, y = 1
if x = 2, y = 15
if x = 3, y = 3
if x = 4, y = 27
...

Of course, this is a completely over-fit tree and won't generalize. But it demonstrates why a decision tree is a non-linear mapping.


generate graph from .eps file (preferably using R) Using the R package 'Deducer', I saved a graphic (chart) as an .eps file.

I can open the .eps file, it's just a bunch of text.

How do I re-generate the graphic (chart) from the .eps file (preferably using Deducer or R)?

EPS is "Encapsulated PostScript". Its meant for embedding like an image in documents, or sending to printers. You can view it with a PostScript document viewer, and there are free PostScript document viewers for Linux, Windows, and Mac OSs. Ghostview, Evince, etc etc.

So although you can view the graphic once you've got a PostScript document viewer, you cannot load it into R as if you had just plotted it.


Resampling : My dataset is categorical or numerical? I have a dataset with 203 variables. Like age>40 (0 -yes, 1-no), gender(0 or 1), used or not 200 types of drugs (one hot encoded into 200 variables), and one target variable (0 or 1). This is an imbalanced dataset where Counter({0: 5607, 1: 1717}).
May I know what kind of resampling strategy I should adopt for this kind of dataset?
Is this dataset considered as numerical or categorical datset?
I tried random under sampling and over sampling, but not satisfied with the ROC curve obtained after modeling.
Can I apply SMOTE considering this as numerical dataset?
I read in this , that In case the dataset only contains categorical variables, the Hamming distance is applied for resampling purpose and If the dataset only contains numerical variables, it is possible to apply traditional distances such as Euclidean, Manhattan or Minkowski.
In case of my dataset, is it okay to apply Euclidean distance for resampling? Could you please direct me to some sources showing how this is done for a datset with only binary values?

For purely categorical data like yours Chewla et al. proposed SMOTE-N in the original paper. SMOTE-N is implemented in imbalanced learn and the user guide describes the differences compared to vanilla SMOTE and SMOTE-NC as follows:

If data are made of only categorical data, one can use the SMOTEN variant [CBHK02]. The algorithm changes in two ways:

the nearest neighbors search does not rely on the Euclidean distance. Indeed, the value difference metric (VDM) also implemented in the class ValueDifferenceMetric is used.
a new sample is generated where each feature value corresponds to the most common category seen in the neighbors samples belonging to the same class.

Also note that your dataset is not extremely imbalanced so depending on the differences in misclassification cost of your classes you may or may not benefit from sampling techniques.
As a side note: ROC Curves can have some caveats when used on imbalanced datasets see, for example, "The Relationship Between Precision-Recall and ROC Curves" and "The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets".


What is the name of the sub-field within spacial Data Science where the data relates in some way to physical objects viewed as 3D (graphics) Models? What is the name of the specialism / sub-field in spacial Data Science where the data relates in some way to physical objects viewed as 3D (graphics) models?, ... that is, excluding GIS and BIM systems. One such example being when applied to the analysis of 3D models of machine parts created by CAD systems in Engineering, where Data Analytics and Data Visualisation may be applied to data arising from that analysis.

There are many possible answers to this question.
Data science applied to mechatronics and cybernetics can involve physical information, which I believe is what you are describing. The DS side of mechatronics often involves IoT and engineering cybernetics principles. More specific sub-fields include design optimization and control theory.
Edit to add a few more examples:

Fault detection
Anomaly Detection
Simulations

Monte Carlo

Modeling and Simulation


Develop a “Credit Risk Scoring System” with R I want to achieve during my internship a credit risk scoring system based on "credit german dataset".

I develop a model with R using logistic regression and i validate it. Now i must use this model to calculate a new applicant score and deduce whether good or bad.

I don't know how to proceed?

First of all, you need to decide how you evaluate "good" vs "bad", which means picking a metric. From the top of my head you have three choices:

accuracy: simplest, but potentially misleading (if you have few defaults and predict all 0s, you still get high accuracy)
logloss: penalize assigning a high def probability to good quality credit and vice versa
ROC: sort the credit from best to worst.

Once you've made your pick, make sure your model can generate predictions in the right format (i.e. output probabilities and not just 0/1) and the go through the training / validation / test paradigm. This should get you started.


Cross Validation and bias relation I found a question (Question 7) here:

Question: For k cross-validation, larger k value implies more bias
 Options:  True or False

My answer is: True.
Reason:
Larger K means more folds means smaller test set which means larger training set. As you increase training data you bring down variance which means increase bias.
So as K increases --> Training data size increases --> Variance reduces --> Bias increases
Hence answer is True
But the website says answer is False.
Can someone explain if my logic is wrong and why their answer is right?

Why should bias increase as the training set size increases?
My intuition is that as you increase K, your test sets get smaller, increasing the variance of your evaluation metric. At the same time, you're fitting your model to more and more similar training data (since the training set in each fold is approaching the full set), so you're more likely to overfit to the training data because you're fitting your candidate models on almost the entire training data set in each fold.


The sum of multi-class prediction is not 1 using tensorflow and keras? I am studying how to do text classification with multiple labels using tensorflow. Let's say my model is like:
model = tf.keras.Sequential([
    tf.keras.layers.Embedding(vocab_size, 50, weights=[embedding_matrix], trainable=False),
    tf.keras.layers.LSTM(128),
    tf.keras.layers.Dense(4, activation='sigmoid')])
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=tf.metrics.categorical_accuracy)

I have 4 classes, the prediction function has the following results:
pred=model.predict(X_test)
pred
array([[0.915674  , 0.4272042 , 0.69613266, 0.3520468 ],
       [0.915674  , 0.42720422, 0.69613266, 0.35204676],
       [0.915674  , 0.4272042 , 0.69613266, 0.3520468 ],
       [0.9156739 , 0.42720422, 0.69613266, 0.3520468 ],
   ......

You can see that every data has 4 prediction values. But the sum of them is not 1, which I do not understand. Did I do something wrong, or how to interpret this?

To get the summation in the last layer to be one, you need to use Softmax activation not Sigmoid. Often Sigmoid is used in a special case when there is a binary classification problem.


Is machine learning suitable for detecting screen elements? I want to test a few devices as pure black box tests. That would mean some kind of robotics to press buttons and the touch screen and detecting screen elements like buttons in varying light conditions with a video camera. I can handle the mechanics but I am not sure how to got about screen detection.

Would machine learning be suitable for such a task? I imagine training the system under different lighting conditions.

Object recognition is a subfield of Computer Vision which consists of methods/algorithms aimed at determining whether a given object is present on a 2D image. Many object recognition systems just return a 'yes/no' answer, but there are also many methods that return the location of the object on the image by means of a bounding box.

This subfield comprises a vast range of methods, from low-complexity ones (like methods based solely on applying simple filters on the image, like colour/edges/etc.) to more complex ones in which machine learning methods (like deep learning) play a very important role.

Depending on the complexity of the objects you are planning to detect, and on the overall complexity of the images, machine learning may be completely unnecessary.

This Wikipedia page summarises many of these methods. You can start exploring from there, or you can start by using any introductory computer vision text.


Research in random forest algorithms able to switch data sets I'm curious as to whether research been done into random forests that combine unsupervised with supervised learning in a way allowing a single algorithm to find patterns in, and work with, multiple different data sets. I have googled every possible way to find research on this, and have come up empty. Can anyone point me in the right direction?

Semi-Supervised Learning

The combination of unsupervised learning and supervised learning is referred to as semi-supervised learning, which is the concept that I believe you are searching for.

Label propagation is often cited when outlining the heuristics of semi-supervised learning.  The essence is to employ clustering, but to use a tiny set of known cases in order to derive (or propogate) the labels of the clusters.  Hence one is able to use a small set of labeled cases to classify a much larger set of unsupervised data.

Here are some references:

Wikipedia has an entry on the semi-supervised learning.
The scikit learn User Guide is often a useful starting point and has a label propogation routine.
There are, in fact, papers treating semi-supervised random forest models.
Another one here

Hope this helps!


NLP techniques to label unlabeled data in a dataset So I have a .xls file with negative and neutral reviews of a medicine. However, this dataset does not have labels. I converted this .xls into a dataframe and I am using the Spacy Lib.

I have to use NLP techniques to label the data. Can you guys help me out on how to use NLP techniques to label this dataset as a neutral review or a negative review. So far, I did data cleansing (remove stop words, punctuation, etc.)

What are the next steps?

This is still a problem undergoing active research. I think it's commonly referred to as unsupervised labeling. If the dataset is not too large, it's probably more accurate and time efficient to either do it by hand, or hire someone on Mechanical Turk to do it.

If you're dead set on doing it automatically, trying some form of semi-supervised learning where you train it on a few examples, and then attempt to generalize is probably your best bet.


Which scoring for GridSearchCV is best, when imbalanced multiclass dataset? I have an unbalanced multiclass dataset (GTSRB) and want to optimize the hyperparameters of an SVM through GridSearchCV.
I know that accuracy is not suitable for scoring in this case.
Which evaluation method for scoring would be most appropriate in this case?

At the moment I tend between the following:
- f1_score (average='macro')
- cohen_kappa_score

What are your experiences in such cases?

There is lots of metrics to measure performance of classifiers. The fundamental ones are based on the idea of:

true positive (TP) — sample’s label is positive and it is classified as one
true negative (TN) — sample’s label is negative and it is classified as one
false positive (FP) — sample’s label is neg., but it is classified as positive
false negative (FN) — sample’s label is pos., but it classified as negative

From what I have seen in white papers, F1-score is the most used metric that consider in imbalanced classification scenarios. But I also see ROC-AUC as a frequent used metric. As I mentioned, there is lots of metrics, but I strongly recommend you to keep these most used to provide to the others some standard sense of performance.


Comparing Language Model of two corpora I know using Conditional Language Model I can learn the probability of a sentence given the corpus I used to train my model. I will then be able to generate meaningful text by sampling from the distribution of sentences.

Now what I want to do is to compare the text generated from the language model of two different corpora on the same topic.

Use case: I want to compare the headline that a right winning vs left winning news outlet would use for a given news content. (my training data would be a large set of headline+news content from both news outlet)

If I understand your question correctly (please comment if I don't), you want to compare the output text from two different language models. Therefore, I don't think you should worry too much about the language models themselves.

If I was you, I'd probably do a simple TFIDF analysis so that you can gain a better understanding of what terms are more prevalent in certain news outlets.

Take a look at scattertext, it allows you to create visualizations like the following:


Why does Naive Forecast,Moving average,Simple Exponential Smothing model predictions flatten? I have applied simple forecasting models such as Naive Forecast, Moving Average, Simple Exponential Smoothing, Holts Linear Trend Model on 2018 sales data of a salesperson.

All the model resulted in flatten or prediction line flattens at zero. Could be it be an issue with data? as most of the data is flatten at zero.

model = ARIMA(train_log, order=(0, 1, 2))
output = model.fit(disp=-1)

#convert fitted values in to series
output_series=pd.Series(output.fittedvalues, copy=True)
print(output_series.head())

#Calc Cumm sum
output_series_cumsum= output_series.cumsum()
print(output_series.head())

#Add cumsum values
output_tr_log=pd.Series(train_log.ix[0],index=train_log.index)
output_tr_log=output_tr_log.add(output_series_cumsum,fill_value=0)
output_tr_log.head()

#convert to predicted ARIMA vlaues to original format
convert_output = np.exp(output_tr_log)
plt.title('RMSE: %.4f'% (np.sqrt(np.dot(convert_output, train_log))/len(train_log))

Date       Sales
----       -----
2018-01-27  1
2018-01-30  60
2018-01-31  22
2018-02-01  490
2018-02-04  53
2018-02-05  30
2018-02-06  204
2018-02-07  234
2018-02-08  64
2018-02-10  70
2018-02-11  81
2018-02-12  10
2018-06-01  40
2018-06-02  669
2018-06-06  1188
2018-06-07  1250
2018-06-10  3861
2018-06-14  40
2018-06-21  44

The naive forecast = random walk with no drift will always give a flat straight line as your forecasts. This is just the last observed data point in your time series extended forward n steps ahead.

The moving average model with m =50 will be the average of the most 50 recent data points. As we can see, your time series is mostly flat with some erratic peaks that have been smoothed away by the relatively large window (50) that you have chosen. In general, the larger the window the more flat and smooth your forecasts. Try changing the window to be smaller and you will probably get more erratic behavior in your forecasts.

Simple exponential smoothing always gives a flat forecast since all forecasted values are equal to the first forecasted value (i.e. y(t+k) = y(t+k-1) =....y(t+1), for all k > 1). This can be proven quite easily using basic induction.

Holts Linear Trend model breaks up the forecasts into three different components; the trend, level, and seasonality. If your time series exhibits no seasonality and no level changes then all you will be left with is your trend which is a flat line in your case.

Your time series looks to be a flat line with unpredictable random fluctuations. Hence, in terms of forecasting error I am not too surprised to see that a flat line may give the best forecasts. Try calculating RMSE, MAE, or MASE (which compares to the naive method directly) and see for yourself. Often, a flat line is the best forecast especially for financial time series in particular.


Why can't we sample without replacement for each tree in a random forest if the subsample size is large enough? Usually if we have $n$ observations, for each tree with form a bootstrapped subsample of size $n$ with replacement. On googling it one common explanation I've seen is that with replacement sampling is necessary for independence of individual trees.
But why can't we just resample as follows: for tree 1, randomly sample $m$ observations without replacement out of the $n$, where $m$ is still large enough (of course, provided that $n$ is large enough in the first place). Then replenish all observations and repeat the resampling for tree 2, and so on.
Even in this case, I'd imagine that the individual subsamples would be independent. So is there an additional reason for resampling with replacement in bagging?

No, the samples will not be independent, there is possibility the data samples will be skewed.
For example, imagine a class-imbalanced binary problem, once the minority class is already sampled (large possibility that this can happen given $n$ and $m$) then, without replacement, the rest trees will only sample from the majority class which will produce skewed trees.
Some references:

“Bagging based on resampling with and without replacement is equivalent”, is it?

For random forests, generally, the concept of replacement is
considered essential. This is because the underlying concept of random
forests is bagging to prevent overfitting, i.e., bagging builds an
ensemble of estimators trained on data with a high variance (with
regard to the training data they have seen).

Why does random forest use sampling with replacement instead of without replacement?

The basic idea of bootstrapping is that you use your sample as a
population. And from it you sample repeatedly, with replacement, to
build other samples of the same size as your original sample.
Replacement is an integral part of this process because you are trying
to create other possible sample distributions which could have come
for your original population based on the sample you have.


Explain output of a given classifier w.r.t features Given a binary classifier, is it always possible to explain why it has classified some input as a positive class ? And by that I mean, if we have a big set of features, is there a tool that says : 'For this output, these are the features that were the most responsible for labeling it as a positive' ?

Thanks  !

Try out some of the examples in this library, which attempts to use machine learning to understand black box machine learning models in Python.


How to overcome training example's different lengths when working with Word Embeddings (word2vec) I'm working on Sentiment Analysis over tweets using word2vec as word representation.

I have trained my word2vec model. But when I'm going to train my classifier, I'm facing the issue that every tweet has different length and the classifier (RandomForest) needs all examples to be of the same size.

Currently, for every tweet I'm averaging the vectors of all of its words, to end up with a vector representation of the tweet.

For example:
My word2vec model represents each word as vectors of size 300.

I have Tweet1 formed by 10 words, and Tweet2 formed by 5 words.

So what I do is, for Tweet1

(v1_Tweet1 + v2_Tweet1 + ... +v10_Tweet1)/10 = v_Tweet1 #avg vector of 300 elements.

For Tweet2:

(v1_Tweet2 + v2_Tweet2 + ... +v5_Tweet1)/5 = v_Tweet2 #avg vector of 300 elements.

*Being v1_TweetX the vector of the first word of the TweetX and so on.

This works 'fine' but I would like to know what other approaches do you take to overcome the different sizes on the train and text examples for the classifier.

Thanks.

Let me suggest three simple options:

average the vectors (component-wise), i.e., compute the word embedding vector for each word in the text, and average them.  (as suggested by others).
take the (component-wise) maximum of the vectors.  (max, instead of average)
take the (component-wise) minimum of the vectors.  (min, instead of average)

Each of these yields a feature vector that is independent of the length of the text.

There is some research suggesting that concatenating the max and the min yields a pretty effective feature space: it's not the absolute optimal, but it's close to optimal, and is simple and easy to implement.  See this question on Statistics.SE for details.

Here is an alternative idea, inspired by cubone's answer, that as far as I know hasn't been tested before.  The idea is to tag the text using a part-of-speech tagger and then use those tags to inform the featurization process.

In particular, write down a list of all possible POS tags that could be emitted by the POS tagger.  Suppose there are 20 possible tags (CC, DT, JJS, MD, NNP, ...).  Then the feature vector will be 20*300 = 6000 elements long: it will have one 300-vector per POS tag, concatenated in some canonical order.  The 300-vector for each tag could be computed by averaging the word embedding vectors of all words that are tagged by the POS-tagger with that tag.  Or, you could get one 600-vector per POS tag, obtained by computing the min and max over all vectors of words with that tag.

This might yield a richer feature space, I don't know if it would yield any improvement, but it's something you could try if you wanted to experiment with different ideas.


keras CNN lstm add model depth Would anyone have any advice on how to add model depth? This works below but I was hoping to experiment with adding in additional non-TimeDistributed layers.

# reshape from [samples, timesteps] into [samples, subsequences, timesteps, features]
n_features = 1
n_seq = 2
n_steps = 2

# define model
model = Sequential()
model.add(TimeDistributed(Conv1D(filters=16, kernel_size=1, activation='relu'), input_shape=(None, n_steps, n_features)))
model.add(TimeDistributed(MaxPooling1D(pool_size=2)))
model.add(TimeDistributed(Flatten()))
model.add(LSTM(50, activation='relu'))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')
# fit model
model.fit(X, y, epochs=500, verbose=0)

For example, if I add this in:

# define model
model = Sequential()
model.add(TimeDistributed(Conv1D(filters=16, kernel_size=1, activation='relu'), input_shape=(None, n_steps, n_features)))
model.add(TimeDistributed(MaxPooling1D(pool_size=2)))
model.add(TimeDistributed(Flatten()))
model.add(LSTM(50, activation='relu'))
model.add(LSTM(40, activation='relu'))
model.add(LSTM(30, activation='relu'))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')
# fit model
model.fit(X, y, epochs=500, verbose=0)

This will throw ValueError: Input 0 is incompatible with layer lstm_2: expected ndim=3, found ndim=2

Any tips help! Sorry not a lot of Wisdom here but learning :)

CTry to change

model.add(LSTM(50, activation='relu'))

to

model.add(LSTM(50, activation='relu', return_sequences=True))

as explained in the documentation ( https://keras.io/layers/recurrent/#lstm ) return_sequences allow the LSTM to return the complete sequence of vectors of the LSTM rather than a single vector.


StratifiedKFold: ValueError: Supported target types are: ('binary', 'multiclass'). Got 'multilabel-indicator' instead Working with Sklearn stratified kfold split, and when I attempt to split using multi-class, I received on error (see below). When I tried and split using binary, it works no problem.

num_classes = len(np.unique(y_train))
y_train_categorical = keras.utils.to_categorical(y_train, num_classes)
kf=StratifiedKFold(n_splits=5, shuffle=True, random_state=999)

splitting data into different folds

for i, (train_index, val_index) in enumerate(kf.split(x_train, y_train_categorical)):
   x_train_kf, x_val_kf = x_train[train_index], x_train[val_index]
   y_train_kf, y_val_kf = y_train[train_index], y_train[val_index]

ValueError: Supported target types are: ('binary', 'multiclass'). Got 'multilabel-indicator' instead.

Is there a way I can used KFold with multi-class?

I had the same problem, you can find my detailed answer here.

Basically, KFold does not recognize your target as multi-class because it relies on these definitions:

  'binary': y contains <= 2 discrete values and is 1d or a column
  vector.
  'multiclass': y contains more than two discrete values, is not a
  sequence of sequences, and is 1d or a column vector.
  'multiclass-multioutput': y is a 2d array that contains more
  than two discrete values, is not a sequence of sequences, and both
  dimensions are of size > 1.
  'multilabel-indicator': y is a label indicator matrix, an array
  of two dimensions with at least two columns, and at most 2 unique
  values.


Deploying machine learning modules I am looking  to find some resources about what I wan't to do :I wan't to make some GUI of my machine learning models and  finally deploy them as a web  app.I find R Shiny to be somehow  ok , but it limits me only to use R,and I have also Python things.Looking at the azure ml studio  it brings  somekind of workflow  like in Disco and ProM from process mining, but I am not  sure  how much pain will be to feed everything  like ensembles/grid search etc to those  flow components,also the full paid  version is  very expensive.
Trying to do a web application  from scratch could also not be ok  cause it will be a lot  of pain  when I will modify my  models.
Any suggestions?

Correct me if I'm wrong here:

You've already written the code that test/trains the model and you're looking for a way to put that model on the web somewhere so users can interact with it?

There's a couple of way's you can go about that, but it depends on what you need out of the final product. Is this something clients will be using? Friends? Just you? Flask allows you to prototype quick web apps and still maintain a good amount of control over what the finished product looks like. Spyre is an amazing way to make a web app at near light speed, although it may not be suited well to your use case, depending on what you're doing. It's a little harder to make unique-looking things with Spyre, but I always suggest it to people.

At any rate, the standard thing to do here is to Pickle your model after it's trained and then load it to memory in a web app. After that you can call the model.predict(X) on an X that is sent to your program by a user.


The use of Keras self-attention module This question calls people to share their personal experiences with keras_self_attention module.
I also summarized the problems I encountered and the solutions I found or received from answers.

Background
I am building a classifier using time series data. The input is in shape of (batch, step, features).
The flawed codes are shown below.
import tensorflow as tf
from tensorflow.keras.layers import Dense, Dropout,Bidirectional,Masking,LSTM
from keras_self_attention import SeqSelfAttention

X_train = np.random.rand(700, 50,34)
y_train = np.random.choice([0, 1], 700)
X_test = np.random.rand(100, 50, 34)
y_test = np.random.choice([0, 1], 100)

model = tf.keras.models.Sequential()
model.add(Masking(mask_value=0.0, input_shape=(X_train.shape[1],X_train.shape[2])))
model.add(Bidirectional(LSTM(units, dropout=dropout, recurrent_dropout=recurrent_dropout)))
model.add(SeqSelfAttention(attention_activation='sigmoid'))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss='categorical_crossentropy',
                  optimizer='adam',
                  metrics=['accuracy'])

history = model.fit(X_train, y_train,
                        batch_size=batch_size,
                        epochs=epochs,
                        validation_data=(X_val, y_val),
                        verbose=1
                        )

yhat = model.predict_prob(X_test)

Problems
1. The model raised an IndexError at the SeqSelfAttention layer.
Solution: This problem was because the `return_sequences` in the last `LSTM` layer was not set to `True`

2. The model raised a ValueError before the Dense layer due to the shape incompatible problem.
Solution: The solution is to add a `Flatten()` layer before `Dense` layer.

*3. How do Muplicative attention and Multi-head work?
*4. Does stacking multiple layers of self_attention(SA) work for improving accuracy? Should it be like LSTM+LSTM+SA+SA or LSTM+SA+LSTM+SA?

It seems that the SeqSelfAttention layer is expecting all the time-steps.
i.e. return_sequences=True
 Same is shown in the home page example. Link
import tensorflow as tf, numpy as np
from tensorflow import keras
from tensorflow.keras.layers import Dense, Dropout,Bidirectional,Masking,LSTM
from keras_self_attention import SeqSelfAttention

X_train = np.random.rand(700, 50,34)
y_train = np.random.choice([0, 1], 700)
X_test = np.random.rand(100, 50, 34)
y_test = np.random.choice([0, 1], 100)

model = tf.keras.models.Sequential()
model.add(keras.layers.Masking(mask_value=0.0, input_shape=(X_train.shape[1], X_train.shape[2])))
model.add(keras.layers.Bidirectional(LSTM(20, dropout=0.25, recurrent_dropout=0.1, return_sequences=True)))
model.add(SeqSelfAttention(attention_activation='sigmoid'))
model.add(keras.layers.Flatten())
model.add(Dense(1, activation='sigmoid'))

model.summary()

model.compile(loss='binary_crossentropy')
model.fit(X_train,y_train,epochs=2)


How exactly do I extract the important features from strings for machine learning? Forgive me for my ignorance. Linked below is an image of my dataset with 1000 tuples.

https://i.stack.imgur.com/WHIlx.png

I have the following questions
(1) How exactly do I go about extracting information from the Ad topic line?
(2) How should I approach the categorical variables Country and City? ( I've heard about encoding but won't the large number of countries/cities be a problem?)

Thank you so much for your time. All help is greatly appreciated.

(1) How exactly do I go about extracting information from the Ad topic line?

The best way to deal with the test string is to Convert it into a term-document matrix and calculate the TFIDF Scores for every word in the string.

(2) How should I approach the categorical variables Country and City? ( I've heard about encoding but won't a large number of countries/cities be a problem?)

Yes you said the right large number of cities will be a problem i suggest still you can go with that, Rather there is a way find out the distribution of all your cities and cities and convert them into integers.
Even another way could be give weights to the cities which are repeated most,let me give you an example:

Suppose you have 10 observations and 5 cities A(5 times),B(2 times),C(once),D(Once),E(Once):

so A will get a weight of 5/10=0.5 which B will get a weight of 2/10=0.2 and so on.

Hope this helps!


Build a tool for manually classifying training data images I have a large number of images that I need to classify for training a clustering algorithm, and I would like to do so offline (the data is proprietary). Basically, I'd like to build a desktop survey tool that enables me to rapidly place each image into one or two categories. Ideally, the tool would:

Search in a pre-specified desktop folder for an image;
Display the image and a static list of categories, allowing me to click on one;
Upon clicking, record the category associated with the image;
Store the image filename and associated category in a dataset somewhere;
Display the next untagged image in the folder and repeat the process.

Is there an easy way to build this kind of tool in Python, or some other pre-built utility that I could use for free offline?

I would recommend building your own database-backed Web app, since you have proprietary data and few (only two?) classes. I would create tables for the images, users, and labels.

user: (id, name)
image: (id, url)
label: (user.id, image.id, time, class)

The label class can be an enum. If you don't want to let users rate the same image multiple times you can drop the time column, and set the first two ID columns as the compound primary key.

If you've never dealt with Web applications and databases it will seem complicated, but it is easy once you get the hang of it. Here is a tutorial. The benefit of this approach is persistence; you can turn off your computer and the start where you left off thanks to the database.

A simpler alternative is to collect all your data in one session using GUI components such as ipywidgets for jupyter, and writing the labels to a file. With this approach you do not get persistence.


The automatic construction of new features from raw data Some observations are far too voluminous in their raw state to be modeled by predictive modeling algorithms directly.

Common examples include image, audio, and textual data, but could just as easily include tabular data with millions of attributes.

Feature extraction is a process of automatically reducing the dimensionality of these types of observations into a much smaller set that can be modelled.

For tabular data, this might include projection methods like Principal Component Analysis and unsupervised clustering methods. For image data, this might include line or edge detection. Depending on the domain, image, video and audio observations lend themselves to many of the same types of DSP methods.

What about generating new features with higher predictive values from the raw data and concatenate them to the raw data?

For example I have data about student wealth, health, family status and I want somehow generate a new feature I can call Social status whic is generated from the raw data and with high predictive value? Is this possible? linear regression can be a good way I need to discover?

'high predictive value' is only defined if you a target which you are trying to predict upon. It seems that you don't, and your goal is to cluster data points according to some scale defined by a variety of factors. These can undoubtedly be used to cluster data points, and I'd advise you to look into the various methods available: some that may be interesting for you are Agglomerative and Hierarchical clustering.

Now to answer the question, you can surely generate new features from the ones present in your dataset that may or may not help you achieve your goal. You can:

Bin your data: define some categories such as 'rich', 'average', 'poor' with specified ranges and create a new feature that maps a numerical value (wealth) to a bin
One-hot-encode categorical variables

After these pre-processing steps are done, you could go ahead an apply the clustering methods I mentioned to group data together into their respective 'Social Status'. Of course, lots of tweaking and experimentation will be needed. As far as what I've encountered, there are not really automagic ways of generating new features, and mostly the available methods will depend greatly on the type of data and problem you are working on.


Extracting Products Name from Unstructured text I have unstructured text like this
Sample1
From: Gujarat, IN
To: Kerala, IN
Milk and egg, 100kg
Please handle with care.

Sample2
Hello,
(Tooth Brush and Paste)120KG in total
P/U: London, GB
Dest: QC, Canada
Keep the product away from heat
F:111-222 333
demo@gmail.com
http://www.sample123.com

Like these, I have lakhs of samples with different products
From the above 2 samples, I want to extract the product name
From sample 1 Milk,egg from sample 2 Tooth Brush,Paste.
Things I have done so far
Based on my research it seems to be a NER problem
I worked on Custom NER using Spacy to extract the product from the samples but it is not efficient, I watched some videos related to NER and also explored similar content in stack overflow but it didn't work well for my problem.
How can I approach this problem further ?

It's not a great dataset for NER because normally NER relies on trigger words close or inside the NE. For example "he went to X" would indicate that X is a location. Here there's no usable context and even the order of the lines can change.
What I would try here is to classify the lines, because apparently there's a kind of structure to each sample:

sender/recepient
email address
phone no
website
Instructions (e.g. "handle with care")

Once all the types of lines which are not a product are eliminated, the ones remaining are likely to describe the product.


How to use SVM in Orange How to use SVM in Orange for sentiments analysis? We will generate sentiments using sentiments analysis widget but now how to use SVM and how to teach SVM in Orange? i am use xlsx file format then it work good below example ... but i want to use text file (csv) format in which generate error target variable set how to remove this error

You can connect the output of the sentiment analysis to Test&Score widget together with SVM learner. Test&Score widget can perform different validations (e.g. cross-validation).

If you want to perform testing on different data please use Predictions widget. For more information please check widgets help available with pressing f1 shortcut in Orange.


Normalization(minmax) gives me worse results than before in KNN, follow up actions? Hello I'm studying a classification problem with KNN right now.

I have many numeric features that I normalized with MinMaxScaler, I also got some OHE categorical features that not seem to cause the drop, and with this method my accuracy dropped almost 10%.

I used holdout method and cross validation for the results.

What are some follow up actions I could use to try to up my accuracy?

Should I try another normalization technique?
Should I try another model ?
Anything else?

Is bad luck.

Look what (I think) is happening:

When you use MinMaxScaler, what you do is relativize (reduce or augment) the distance between individuals in a way it was not before the change.

Let's suppose your best model is defined by the variable $X_j$ from the variable set $X=\{X_1,X_2,...,X_n\}$. When you change the set to an scalled one: $S=\{S_1,S_2,...,S_n\}$, what you do is giving the same importance to every variable, instead of preserving what the variables said before scalling.

$X_3$ gives better information on the output than $S_3$ (and in $S$, all are scalled), $S_3$ gives less information because is mixed with all other variables.

Imagine $X_3$ as something like "age" (and you are determining probability of having cancer), the greater the age, the greater the probability. This is true for $S_3$ also, but when $S_3$ is combined with the rest of $S$, $S_3$ looses importance amongst them. With $X_3$, age keeps its relative importance (bigger values amongst lesser values).

This does not happen very often, you could find yourself on the opposite situation: When combined $S$ might be a very powerful set.

That is why I think you were simply, unlucky.


How to rank documents using Bag of words approach I want to cluster the documents I get for Google scholar search using the Bag of words model. I thought of using Java as the language.

Assume for the keyword k, Google scholar gives me 50 results. If I have a predefined set of words w1, w2, w3... wn, how can I rank the the documents which have the predefined set of words most? How can I apply Bag of words model for this? Do I need a clustering algorithm like k-means? And do I need to perform NLP techniques as well?

Say that the word w1 has several synonyms. How can I consider those synonyms as well for document ranking? Will I have to create a corpus containing all the abbreviations, synonym etc for that?

Are there any good tutorials available for this? Will choosing Java over Python will be an disadvantage since most of the resources (Ex - Scikit) were in Python?

If you're just looking to rank documents according to how many appearances your words w1,..,wn contain, then there's no need for clustering or machine learning in general: Clustering your 50 results will give you a partition of these results into clusters containing results that are similar to one another and different from the results in other clusters.

If you just need to rank by word occ, just count the frequencies of your words in each document (including synonyms, which you can get e.g. from Wordnet automatically if you prefer) and sum them up.


K-fold crossvalidation: how do MSE average and variance vary with K? I'd like to get an intuition about how varying k impacts k-fold validation.
Is the following right?

Average of the OOS MSEs should generally decrease with k
Because, a bigger "k" means the training sets are larger, so we have more data to fit the model (assuming we are using the "right" model).

Variance of the OOS MSEs should generally increase with k.
A bigger "k" means having more validation sets. So we have have more individual MSEs to average out. Since the MSEs of many small folds will be more sparse than MSEs of few large folds, variance will be higher.

Average of the OOS MSEs should generally decrease as k increases. This is right but the difference is much less then on your chart. Suppose we have a dataset where the error will halve if we increase the data 10 times (approximately true for the paper Scaling to Very Very Large Corpora for Natural Language Disambiguation). Then the difference between 5-fold and 20-fold validation will be about 5% (1/(2^log10(0.95/0.8)) not halving like on your graph. And the difference between 20-fold and infinity-fold will be only about 1.5% (1/(2^log10(1/0.95))

For the chart you could use the formula: Average OOS MSE = 1/(2^log10(1-1/k))*MSE_inf. This will assume that you have MSE = MSE_inf at infinity.

Variance of the OOS MSEs should generally increase as k increases. MSE is an average and according to the Central Limit Theorem (if squared errors (SE) are independent and identically distributed which, in my opinion, is supposed for most of the machine learning algorithms) variance should equal to Var(SE)/N, where N is the number of data points used to calculate MSE. So for 5-fold you will have variance Var(SE)/(Npop/5) Where Npop is the total number of points that you have. For the average MSE between all k-folds the variance will be the same and equal to Var(SE)/Npop. So the answer is that the variance of individual MSE numbers of each k-fold increases with k, but the variance of the final average MSE does not depend on the number of folds.

To calculate the variance of the final MSE based on MSE of folds:

Var(MSE_final) = Mean(Var(MSE_folds))/k = Sum((MSE_folds - Mean(MSE_folds))^2)/k^2

MSE change based on number of folds

Here MSE at infinity is assumed 0.05.

Variance of individual k-fold MSEs change based on number of folds

Here Var(squared error for one point) is assumed 10 and number of observations is 1000.


How to backup a Caffe (DL framework) project? The Caffe folder (downloaded from GitHub) can't be copied or compressed I am using Ubuntu18.04 and Caffe deep learning framework to do my project. However, when I tried to backup my project, I found that the Caffe folder (downloaded from GitHub) can't be copied or compressed.

Here is what I did:

Download Caffe from GitHub, and install it.
Do some modification to the source code for my project.
Backup the whole Caffe folder.
However, if I copy the folder to a USB flash drive, I will have a
folder that misses files (I compared the size and number of files of two folders). If I compress the folder then copy it to the USB flash drive, then I get error says I can't open the compressed file.

I wonder if I did anything wrong, what is the correct way of backup Caffe project?

Thanks.

Based on the comments, the problem is due to wrong file permissions for certain files. Assigning read permissions to the full directory should solve it:

chmod -R u+r <caffe directory>


Orange: Data has no target variable error I am trying to apply Random Forest algorithm on a data set using Orange. The target variable is not set in the data set. However, I know which column is the target variable.

How can I specify the target variable in a .csv file using Orange or any other tools?

You can specify the target by choosing target in the feature column


Why spectral clustering results in disjointed cluster? I'm working on a project where I have to dynamically cluster the position of objects with respect to one coordinate. So I'm essentially dealing with subsequent frames and each frame represents a one-dimensional dataset. The intuition behind clustering is to form clusters out of points that are in similar distance to other points within the cluster and can be naturally connected. I use spectral clustering due to its ability to cluster points by their connectedness and not the absolute location and I set rbf kernel due to its non-linear transofrmation of distance. However, in some frames the algorithm results in unnatural assignments. One example is

import numpy as np
from sklearn.cluster import SpectralClustering

X = np.array([[51.08354988], [57.10594997], [70.51259995], [76.74425011],
[61.24844971], [89.00615082], [98.55859985], [61.26575031], [88.35105019],
[87.40859985]])

clustering = SpectralClustering(n_clusters = 4, random_state  = 42,
                                gamma = 5 / (X.max() - X.min()))
clustering.fit(X)

and the result of clustering is presented in a form of swarm plot below, so only x coordinate matters here (each color represents a cluster and labels are array indices):

What I cannot understand is why points marked as red are clustered together as the similarity between points {4, 7} and {5, 8, 9} should be really low. My first thought was that maybe this is caused by unlucky, random initialisation, but I tried with many different random states and the resulting clusters seems to be persistant. So I guess this is connected to the chosen affinity measure (rbf_kernel) and its gamma parameter. As points move with each frame, and distances between them are kind of dependent on their overall range, I tried to set gamma to 5 / (X.max() - X.min()). The intuition behind this was that if the range is bigger, then the distances between points are usually bigger and we should penalise them more to obtain similar values of exp(-gamma * ||x-y||^2) to those obtained within smaller range. But it doesn't seem to work as expected and results in faulty clustering where red cluster is formed out of points divided by green cluster). My expectation would be rather clusters formed as follows: {0, 1, 4, 7}, {2, 3}, {9, 8, 5} and {6} or {0}, {1, 4, 7}, {2, 3}, {5, 6, 8, 9}.

So my question are:

Is affinity choice and its gamma parameter really the problem here?
If so, how can I choose gamma better?
Otherwise, what approach should I consider to deal with faulty assignments with separated points within the same clusters as presented here?
(Side question) Is there any measure/index that would be suitable to automatically compare clusterings with different number of clusters?

@Edit: As it can be observed under this link, those separated clusters occurs for a short period of time, but still, the problem seems to be recurrent.

First, please note that spectral clustering is very sensitive to the affinity kernel. With the standard RBF kernel, my experience is that spectral clustering often isolates outliers (in the spectral space), leaving clusters with numerous observations which can be separated by great distances. This is a major difference with direct k-means: there is no notion of distance anymore, it's just about connectivity.

In the case shown in your example, the results are "strange" because of the low dimensionality: 1D is typically a case where spectral clustering does not provide "logical" results. In addition, you are quite unlucky on several aspects:

The numbers of clusters (4): try it out with different numbers of clusters, and you will find more trivial results
The positions of observations: move #0 to the left and you may find results that make more sense
The low number of observations: #6 is not considered as an outlier, due to its vicinity to the #5, #8, #9 triplet. Indeed, the more points there are in an area, the largest its influence is. This is also how #4 and #7 can be considered part of the (#5, #6, #8, #9) cluster, rather than the (#2, #3) cluster. Try removing #9 and see how the results suddenly make sense.
The chosen kernel (not tried on my side, but it is likely that the results may vary a lot if you increase the gamma value or change the kernel shape)

TL;DR: your case is an example of how spectral clustering can go wrong with low number of observations and dimensions.


Selecting dataset splitting strategy I found this very informative figure, on how to split the dataset depending on how much data (or how many observations to be more precise) you have.

My question is, since "less data" is very subjective, is there a statistical test you can perform or even a rule of thumb on which split to follow?

My current problem is a classification problem with 145 observations, 22 features, 2 labels (18-True, 127-False), but I'm interested on the general approach.

Thank you.

As usual in these cases, there is no magic wand to determine which splitting method to use. It all depends on your specific data. Is your collected data redundant enough so that k-fold cross validation is not necessary? Does your data capture most of the input space?

Now, taking a look at your numbers, I'd say that the number of observations you have (145) is not likely to be large enough to capture all the potential variability in the input space, giving the fact that you have a high number of features (22). This conclusion, of course, depends on the type of the features (are they binary? categorical? numerical?), and whether all these features are actually necessary to make a prediction (are there redundant/correlated features? features that do not give any information about the output variables?).

In your case, and not knowing more about the data, I'd go for a cross-validation splitting scheme.


Data that's not missing is called...? Is there a standard term for data that are not missing? I.e. is it called non-missing, present, or something else?

Depends on content, but I would probably go for "observed" (vs. "unobserved"). A suitable direct antonym of "missing" might be "extant".


Colouring points based on cluster on matplotlib I have a set of points where I performed a KMeans classification. How make a plot where the color of the point is based on the cluster they belong?

EDIT: for clarification, having the set of points, I want to use the values of the array generated from KMeans.predict() ( from sklearn) to choose the color of each point.

The sklearn documentation shows you how:

colors = np.array([x for x in 'bgrcmykbgrcmykbgrcmykbgrcmyk'])
colors = np.hstack([colors] * 20)
...
if hasattr(algorithm, 'cluster_centers_'):
        centers = algorithm.cluster_centers_
        center_colors = colors[:len(centers)]
        plt.scatter(centers[:, 0], centers[:, 1], s=100, c=center_colors)


How to show two pictures in one cell in Jupyter Notebook? (matplotlib) (python) As a beginner, can someone please help me with this? Is there any way to show both pictures in a single cell's output? My output could only display one picture. Thank you for your time and attention!
Here is my code:
from skimage import data
image_coffee = data.coffee()
image_horse = data.horse()
fig = plt.figure(dpi=100)
plt.imshow(image_coffee)
plt.imshow(image_horse)

One way to do it (without getting into lots of the inner-workings of Jupyter notebooks), it to use two matplotlib Axes in one plot. Then you show one image in each of these:
from skimage import data
image_coffee = data.coffee()
image_horse = data.horse()
fig, axs = plt.subplots(1, 2, figsize=(15, 8))  # one row of Axes, two columns = 2 plots

The axs variables is a list containing the two Axes, so just access each one and plot your image on it like this:
axs[0].imshow(image_coffee)
axs[1].imshow(image_horse)

If the plots don't pop automatically, either run plt.show() or make sure your notebook has executed %matplotlib inline in a cell.


How to obtain a count of the classes of a categorical var within a certain time interval for a time-stamped data? I have a dataframe with several categorical variables. But for simplicity let's assume there is only 1 categorical variables with 3 classes. I want to obtain the counts of these classes within a certain time-intervals, say 15mins. To make it easier to understand what I am looking for, here is a toy example and the output I am looking for.
                    _time       AN
0 2019-04-09 16:00:00.050        a
1 2019-04-09 16:00:00.050        a
2 2019-04-09 16:00:00.050        b
3 2019-04-09 16:00:00.050        a
4 2019-04-09 16:00:00.050        b
5 2019-04-09 16:02:38.992        a
6 2019-04-09 16:06:41.884        c
7 2019-04-09 16:15:00.051        a
8 2019-04-09 16:15:00.051        b
9 2019-04-09 16:15:00.051        a

The output that I am looking for is below:
_time               AN
2019-04-09 16:00:00 a                4
                    b                2
                    c                1
2019-04-09 16:15:00 a                2
                    b                1

So for this toy example, the timeline has two 15mins intervals. For the first one, 'a' appears 4 times, 'b' appears 2 times and 'c' appear 1 time. In the 2nd 15min interval, 'a' appears 2 times and 'b' appear only 1 time and 'c' doesn't appear at all.
I obtained this result by running this code:
c.resample('15T',on = '_time').agg({'AN':'value_counts'})

However, when I am running this same code on the entire dataframe, df, I am getting the following error:
ValueError: operands could not be broadcast together with shape (10267,) (4315,)
I am not sure why I am getting this. Is there a different method to obtain the same result? or is there any suggestion to fix this error?
Thanks in advance.

import pandas as pd
data = pd.read_csv("/content/da_time.txt",header=None) # File has your example data
data.columns=['num','date_time','var']     # Named columns
data.date_time = pd.to_datetime(data.date_time) # To Pandas datetime
data['date_time_copy'] = data.date_time   # Created a copy of datetme column

# make second as multiple of 15 and millisecs=0
data['date_time_copy'] = data.date_time_copy.apply(lambda x:x.replace(minute=x.minute//15*15,second=0,microsecond=0))

data.groupby(by=['date_time_copy','var']).count()['num']

Output


Create a Predictive model to find users that would book an exam I'm currently working for a health company in Brazil, and I want to create a Predictive model that would find users that are one step close to book an appointment.

We currently have 1 million visits per month, and 60% of this visits are not booking anything (exam, consult, checkups, etc). I believe that with ML, we could find which users would book if we make a call or send an email.

I have been a professional web development for the past 7 years, but I don't have the skills to create a Predictive model, so I'm looking for directions, a way to start.

We do have Google Analytics, but I'm thinking to create a simple software that would track:

Age
Gender
Which pages the user visited
How many time the user expended in each page
Name

Then, I could segregate the users in two groups: the ones that booked, and the users that didn't book, and then... I'm stuck.

Could someone give a direction? Should I create a software to track this information or use GA? I want create a demo to get financing, so I don't have much time :/

Thank you.

Your problem is a kind of Binary Classification which predicts the probability of a target variable to be Yes(1) or No(0).
But, first of all, you need some ground-truth data, because most of the state-of-the-art solutions to this problem are supervised.

If you're the admin of this system or have access to their server, you can collect these data by logging everything you need from every user and label their traces with 1 and 0 when they book or not.

After a while, you have a big labeled dataset and you can normalize and shape your inputs in a way that it fits for example to a deep neural network for prediction.


KFold Cross Validation # The code is as below.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

bankdata = pd.read_csv("bill_authentication.csv")
X = bankdata.iloc[:, 0:4].values
y = bankdata.iloc[:, 4].values

from sklearn.model_selection import KFold
kf = KFold(n_splits=10, shuffle=True,random_state=50)
for train_index, test_index in kf.split(X):
   print("TRAIN:", train_index, "TEST:", test_index)
   X_train, X_test = X[train_index], X[test_index]
   y_train, y_test = y[train_index], y[test_index]

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

from sklearn.neighbors import KNeighborsClassifier
classifier = KNeighborsClassifier(n_neighbors=5)
classifier.fit(X_train, y_train)
y_pred = classifier.predict(X_test)
Predicted=pd.DataFrame({'Actual':y_test, 'Predicted':y_pred})

from sklearn.metrics import classification_report, confusion_matrix, accuracy_score
Confusion_matrix = confusion_matrix(y_test,y_pred)
ClassificationReport = classification_report(y_test,y_pred)
AccuracyScore = accuracy_score(y_test, y_pred)

NewData = pd.read_csv("PredectionData.csv")
NewPredictededtion=classifier.predict(NewData)
NewPredictededtion=pd.DataFrame({'Variance':NewData.Variance, 'Skewness':NewData.Skewness, 'Curtosis':NewData.Curtosis, 'Entropy':NewData.Entropy, 'Predicted':NewPredictededtion})

I have problem

  when I run the code I have the prediction result for only one fold

how I edit code to show the predicted result of all folds

It's really hard to understand your code but it seems that you are only seeing the last fold's y_pred . Create a list and in each iteration append the prediction to it.


Sentiment Analysis Label Distribution I am working on Sentiment Analysis model.
The dataset I have has three labels POSITIVE , NEGATIVE and NEUTRAL

But the problem is the data is not equal for labels.
Say out of 100K , 75 K are neutral, 15K positive and 10K negative.

I wanted to know whether it is necessary to chose equal distribution of labels while training or I can go ahead with unequal data and if so till what extent.
Are there any ways to deal with such problem.

For training, close to equal distributed data will give you better results.

Type of data that you have, generally produced a biased model towards "neutral" class.

Are there any ways to deal with such problem?

I generally perform oversampling of the minority classes, such that for training(only),  have sufficient uniform count of data set.
 SMOTE, ADASYN are the few techniques of oversampling.


What is the best way for synthetic data generation while maintaining privacy? For one of the projects where we are working as third party contractors, we need a way for the company to share some datasets which can be used for data science. It is not possible for the company to share the real data as that would be a privacy issue.
We are exploring ways so that the company can either share the data while maintaining privacy or else ways to generate fake data that matches the statistics/demographics of the actual data.
We are currently looking at a couple of options:

Using differential privacy to add noise to the data and then sharing the transformed data with us. Can this approach lead to any privacy issue? I am concerned about reverse engineering. Does "privacy budget" apply here? How should it be tackled?
Using encoder-decoder neural networks to learn vector embedding of the real data. Once the vector embedding is learned, the decoder can be destroyed and the encoder's output can be shared with us.

Is there any other approach that can be used for synthetic data generation that resembles the actual data in terms of demography and statistics. Or else what would be the best way to access the real data without violating privacy?

Whichever method you choose is fine but assuming that you wish to mitigate inference attacks something like differential privacy is required for either approach.
Formally speaking, differential privacy provides some of the strongest guarantees against reverse engineering. Specifically, it promises that any attacker, regardless of attack methodology or available computing power, will be unable to conclude with certainty whether or not any individual has contributed data to a dataset. This is because the results of differentially-private methods are ambiguous up to the addition or removal of the input contributions of any individual. In essence, every individual gets deniability about their participation (or non-participation) in the input.
The problem with synthetic data is that it is generated from a model that is fit to real data. This means that model parameters are aggregate functions of the real data. This is problematic as it is often possible to make inferences from aggregates of data or estimates thereof (this is the motivation for differential privacy in the first place), and parameters of the generative model can be often be estimated from the synthetic data. I am happy to give an example of such an attack if there is interest. Further, it should be noted that this reasoning also implies that white-box exchange of the model is at least as risky and comes with additional concerns like has this network memorized training data. A straight-forward mitigation for this is to apply differential privacy in building the generative model for synthetic data.
In regards to privacy budgets, one can interpret the budget (often referred to as $\varepsilon$ and also the privacy-loss) as the amount of information (say in bits) which can be inferred about any individual by an adversary who has access to differentially-private results. Perhaps surprisingly, it can and ideally should be much less than 1. If there are future updates which reference the same individuals then one has to worry about how much individual information can be inferred from the aggregate collection of releases. There is a straight-forward composition theorem (See e.g. Sect 3.5) that follows directly from the definition of differential privacy. It states that aggregate privacy-loss is at most the sum of the individual privacy-losses of the constituent releases. In other words it's additive in the worst case. It may also be helpful to know that when the inputs are disjoint it goes like the max.


HBase connector - Thrift or REST There seem to be at least 2 ways to connect to HBase from external application, with language other then Java (i.e. Python):

HBase Thrift API
HBase Stargate (REST API)

Does anyone know which one should be used in which circumstances?
I.e. what are their main differences, and pros/cons?

Thrift is generally faster because the data being exchanged is smaller.  Stargate offers a web service which is an integration method that is widely supported, which is a concern when you are working with commercial products with limited integration possibilities.

In a closed environment where everything is controlled, I would prefer Thrift.  If I'm exposing data to external teams or systems I would prefer Stargate.


Linear Learning Machines I was reading about Linear Learning Machines (LLMs) and learned that it is closely related with SVMs. Would like to know an example of any concrete problems that can be classified by LLM as I couldn't find any promising example or explanation for this.
Appreciate your time.

Not sure at all since I am not very familiar with these old concepts.
But I think what you call LLM (not a very common concept it seems) are algorithms that solve linearly separable data classification. SVM are algorithms that look for an hyperplane that maximize the margin with data and thus can be considered LLM. Quick reminder how a SVM solves a linearly separable problem :

We escape LLM's grasp when trying to solve a nonlinear classification problem, as represented on the left of the following picture.

SVM can still solve these type of classification with the kernel trick, that applies a non-linear function to turn the classification into a linear problem.
Hope this can help your understanding of the difference, all images are from Wikipedia. My apologies for not knowing how to resize these images :(


What tools do you use to clean corrupt data? Customer often send currupt data for analysis. I spent a lot of time in cleaning the data or waiting for a correct dataset.
Can you recommend a tool that can handle the most common curruptions (eg. wrong set quotes)...

If you know the nature of data you require as suggested by @Emre for manual cleaning you can use Apache Spark or Pandas. Utilizing the data frame aspects of both Spark and Pandas will help you preprocess data with a much easier convenience. Because data frames account for any data type it's easy to access and clean specifically the segments you require. If the corrupt data is of repetitive nature you can use several Map reduce techniques to clean it fast as well. If it's the case of big data.


How to determine feature importance in a neural network? I have a neural network to solve a time series forecasting problem.  It is a sequence-to-sequence neural network and currently it is trained on samples each with ten features.  The performance of the model is average and I would like to investigate whether adding or removing features will improve the performance.  I have constructed the neural network using keras.

The features I have included are:

The historical data
quarterly lagged series of the historical data (4 series)
A series of the change in value each week
Four time invariant features tiled to extend the length of the series. (another 4 series)

I am aware I could run the model many times changing the combination of features included each time. However, along with tuning the hyperparameters (for it might be that 8 features works really well with one set of hyperparameters but not with another set) this is really a lot of possible combinations.

Is there any separate way that I can use to guage if a feature is likely to add value to the model or not?

I am particuarly concerned that I have four time-invariant features being fed into the model which is designed to work with time varying data and I would like a way to measure their impact and if they add anything or not?

Don't remove a feature to find out its importance, but instead randomize or shuffle it.

Run the training 10 times, randomize a different feature column each time and then compare the performance. There is no need to tune hyper-parameters when done this way.

Here's the theory behind my suggestion: feature importance


How to do spelling correction for a language but also correct some words in other language I want to do spell correction for the portuguese language, specifically for restaurant bots. The problem is that food names aren't always in portuguese as well, and for that reason are the most likely to need correction as the user would not always know how to spell it correctly.

I though of a few things. For example:

Training the model with portugese words + food words spelled wrong,
but it'd be very difficult to find these food words
Training the model for several different languages (but I guess would
make it confused and actually correct a lot of things wrong).
Training the model with portuguese words and for the food words, use
something that picks words that the user wrote and try to approximate
them to the list of food words. (woudln't it be slow?)

What would be a good solution to this problem that can be fast to be used with lots of requests?

There are many ways to build a spell corrector. One of the simplest is:

Detect an incorrect word
Generate candidate suggestions
Score and rank the candidate replacements

For detecting an incorrect word, a simplifying assumption is that any word not a dictionary is a spelling error. Otherwise, you have to build a separate model to detect if a word is a potential spelling mistake based the current context.

To generate candidate words, you need to find dictionary words that are similar to the incorrect word. That requires defining "similar", typically similar is measured by edit distance (e.g., deletes, transposes, replaces, or inserts of individual characters).

For scoring and ranking, probabilities of candidate words are weighted by a language model and channel model. A language model weights how likely a word will appear in the current context. A channel model reflects if an error happens depending on how the word is transmitted (e.g., full computer keyboard errors are different from mobile phone errors).

For your example, you need to construct a custom dictionary of all possible words and a corpus reflecting the frequency of occurrences for the custom dictionary words. If possible, construct an error model of common mistakes.

Peter Norvig goes into greater detail here.

Later you can add Deep Learning methods, but it is better to start with a simpler approach that requires less data.