Building document classifier based on keywords, what would be the steps? I have a requirement of classifying documents(.doc files) based on the profiles. I have a csv file with data:

label          keyWords
Web developer  ["html","css","php"]
Developer      ["core","java","python"]
Embedded Dev   ["ARM", "CORTEX", "C"]

Now I want to classify the .doc files.

Also please do tell me what model to use ? or broad steps for proceeding (like building dataframe-> then splitting data and so on...)

EDIT:Any pointers on how to proceed with document classifier ?

Got your email...

Built this example repository for ya:
https://github.com/dancrew32/resume_classifer/

If you go through the notebook, I outlined the process of setting up labels, different folders for different kinds of resumes, loading them up, vectorizing text, classifying using logistic regression.

https://github.com/dancrew32/resume_classifer/blob/master/Resume%20Doc%20Classifier.ipynb

Let me know if you have any questions.

P.S. Never adding a skip button ;)


Is there a good systematic approach to explore and analyze data (prior to modelling)? I have found a few examples of kernels on Kaggle people have made where they seem to follow a certain methodology in order to systematically analyze and explore the data, to make sure to find all outliers, missing values etc. They are just practical examples and leave quite a few questions.
I would just assume there must be one, or a few, recipes/methodologies/flowcharts that I can learn/use in order to have a good checklist and systematically work my way through my data and catch all data quality issues as well as note them in a well defined way. I.e. 1. Do this, 2. Do that, if result A do this, if result B note that for later this way 3. Do this etc.
What should I look into in order to find something like this? I can't imagine this not existing. It seems very much like Data Science Course 101 (which I am currently taking but it seems like a lousy course) but I can't find anything.
Any references to YouTube-videos or Coursera-courses that explain this would be welcome as well.

Yes, there is this.

Always check the distribution of the target
. If classification,% of each class. This is important for many type of metrics and losses.
If regression, check for eventual high concentration in some fixed value. For example, it happens often that because of some mistakes in processing the target ends up being imputed and there are a lot of zeros.

Always check for drift. Typically these are id's. They have no meaning, but often they drive models to absurdly high performance because of some hidden bias.

Understand the distribution of features. often what you expect in production will not match what is in test. Think of some distribution of populations, that were sampled in particular way in training but will be more evenly matched in production. This happens often in churn/targetting models

If you are working with structured data, take time to understand what your features mean. If there is a data catalog available, read it. Example: Users tend to register on the website before purchase. Then if you are fitting a targeting model, probably due to use registration as a feature (BTW: This is a real life example that I have seen happen not only once, nor twice, but over three times at three different companies.)

If you are working with text. Take some time to read a couple of examples. Just don't throw it in the model. NLP is one of the few use-cases where you have the most intuition on what the model needs to understand.


What to do about non-responses to demographic survey questions? Can anyone point me to some think pieces on what to do with census type data where at least some of the people surveyed do not self-identify a race/ethnicity,  gender, or other demographic data?

In studies (and datasets) examining diversity in various communities, I see a wide range of strategies. Some simply remove records where a person declines to specify a race/ethnicity or gender. Some include them and create an "unknown" category.  If looking at race/ethnicity, for example, some studies just state they assume all non-responders are white.

None of these solutions seem perfect. Are there any others? And if you remove responses from people who, to take one example, decline to specify a gender, aren't you potentially just erasing part of the population that does not identify as male or female?

A number of surveys now allow participants the option to "decline to state" a gender or "decline to state" a race/ethnicity or some similar option, or to just not answer the question at all, is there any developing consensus on what to do after that data is collected?  Does it depend on the population being studied?

It depends on what you want to do with the data. One common approach is to treat non-response as „missing“, meaning, you drop this observation. This will possibly lead to a bias in the data, e.g. if a certain group refuses to answer a question more often than others. As a cure to this problem, some use ex-post weighting of data. You can do this if you know how often some group will occur in the population as a whole. So if you see that group A is underrepresented in your data, you assign a higher weight to each observation in group A. You can use this weights to adjust descriptive statistics or regression results. The technique is called probability weighting: https://en.m.wikipedia.org/wiki/Inverse_probability_weighting.

Note that there are limitations of this approach. You can reasonably claim that ex post weighting is okay for things such as income if you have sufficient data, assuming that (important!) say a low income bracket in the available data is representative for the whole population of low income households/people. This will not necessarily hold in other cases, e.g. if you study certain lifestyles or so.

Non-response may also occur if people are ashamed to answer some question. In this case, there is a systematic bias in the data with unknown data generating process, so that it is more or less impossible to cure this problem/bias.

Yet another option would be imputation, meaning you estimate what someone would answer (often applied to income for instance). However, in this case you can never be sure that you get things right. So imputation often is very problematic.


I'm trying to build a ResNet 18 model for Cifar 10 dataset, but I'm not able to fit the data dimension At avergae pooling after the ConvNet, the error is displayed as the dimensions cannot be negative because the shape the previous output layer is 1,1,512 and on this the maxpooling cannot be done. Is it something that i did wrong in the architecture design?
def identity_block2(X,f,filters):
  f1,f2 = filters
  X_init = X

  X = Conv2D(f1,(3,3),strides=(1,1),padding='same')(X)
  X = BatchNormalization()(X)
  X = Activation('relu')(X)

  X = Conv2D(f1,(3,3),strides=(1,1),padding='same')(X)
  X = BatchNormalization()(X)
  X = Activation('relu')(X)

  X = Add()([X,X_init])
  X = Activation('relu')(X)
  return X

def conv_block2(X,f,filters,s=2):
  f1,f2 = filters
  X_init = X

  X = Conv2D(f1,(3,3),strides=(s,s),padding='same')(X)
  X = BatchNormalization()(X)
  X = Activation('relu')(X)

  X = Conv2D(f2,(3,3),strides=(1,1),padding='same')(X)
  X = BatchNormalization()(X)
  X = Activation('relu')(X)

  X_init = Conv2D(f1,(3,3),strides=(s,s),padding='same')(X_init)
  X_init = BatchNormalization()(X_init)

  X = Add()([X,X_init])
  X = Activation('relu')(X)]
  return X

def resnet18(input_shape,classes):
  X_input = Input(shape=input_shape)

  X = ZeroPadding2D((3,3))(X_input)
  X = Conv2D(64,(7,7),strides=(2,2))(X)
  X = MaxPooling2D((3,3),strides=(2,2))(X)

  X = conv_block2(X,3,[64,64],s=1)
  X = identity_block2(X,3,[64,64])
  X = Dropout(0.4)(X)

  X = conv_block2(X,3,[128,128],s=2)
  X = identity_block2(X,3,[128,128])
  X = Dropout(0.4)(X)

  X = conv_block2(X,3,[256,256],s=2)
  X = identity_block2(X,3,[256,256])
  X = Dropout(0.4)(X)

  X = conv_block2(X,3,[512,512],s=2)
  X = identity_block2(X,3,[512,512])
  X = Dropout(0.4)(X)

  # It shows error at this point as the dimension of the previous output layer is (1,1,512)
  # X = AveragePooling2D((2,2))(X)
  X = Flatten()(X)
  X = Dense(classes,activation='softmax')(X)

  model = Model(inputs=X_input,outputs=X,name='ResNet18')

  return model

The problem is that your ResNet-18 follows the architecture for ImageNet as outlined in the ResNet paper:

However, spatial input dimensions of ImageNet are different from CIFAR10 (32x32) so the architecture does not match your input. Instead you can follow the author's description of their CIFAR10 architecture in section 4.2 of the same paper:

The plain/residual architectures follow the form in Fig. 3
(middle/right). The network inputs are 32x32 images, with
the per-pixel mean subtracted. The first layer is 3x3 convolutions.
Then we use a stack of 6n layers with 3x3 convolutions
on the feature maps of sizes 32; 16; 8 respectively,
with 2n layers for each feature map size. The numbers of
filters are 16; 32; 64 respectively. The subsampling is performed
by convolutions with a stride of 2. The network ends
with a global average pooling, a 10-way fully-connected
layer, and softmax.


What is the difference between residual sum of squares and ordinary least squares? They look like the same thing to me but I'm not sure.

Update: in retrospect, this was not a very good question. OLS refers to fitting a line to data and RSS is the cost function that OLS uses. It finds the parameters that gives the least residual sum of square errors. It is called ordinary in OLS refers to the fact that we are doing a linear fit.

Here is a definition from Wikipedia:

  In statistics, the residual sum of squares (RSS) is the sum of the
  squares of residuals. It is a measure of the discrepancy between the
  data and an estimation model; Ordinary least squares (OLS) is a method
  for estimating the unknown parameters in a linear regression model,
  with the goal of minimizing the differences between the observed
  responses in some arbitrary dataset and the responses predicted by the
  linear approximation of the data.

So RSS is a measure of how good the model approximates the data while OLS is a method of constructing a good model.


Multi-class classification with mostly zero valued data I implemented a multi-class classification and wanted to test it using the MNIST dataset. I realized that if I use standardization

$X \leftarrow \frac{X-mean(X)}{std(X)}$,

over 50% of all features will be zero. Is that a problem?

Does it make more sense to work in such a case with normalization

$X \leftarrow \frac{2(X-min(X))}{max(X) - min(X)} - 1$,

such that all features are between -1 and 1?

What about doing first a standardization followed by a normalization step?

Welcome! Sounds like your data set is highly imbalanced. For techniques on handling an imbalanced data set, please have a look at this. https://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/

EDIT: Looks like I misunderstood the question, sorry! Also: I think the term "multi-class regression" does not exist. It sounds more like you're doing multi-class classification. If your input features are consistently 0, then it sounds like your actual value of each feature and the mean of the feature are identical, in which case you have very poor features. The features will not do a good job of differentiating between the classes. You should try (or hopefully use a neural network to learn) a new set of features.


Classification problem approach with Python I am a Python beginner, just getting into machine learning and need advice on the approach i should use for my problem.

Here is an example of my data-set.

Where the RESULT is a corresponding INDEX in each VALUES array and every row is a separate array, i need to find the probability of each index of an array being the RESULT based on its configuration, where the overall configuration and distribution is most important rather than any individual value.

You are looking at a Classification problem.

Logistic regression, Decision trees, SVM. Any of the above can solve the job for you. But selecting the best model depends upon how good it is able to predict the test set. Use cross-validations and see which model can give you better accuracy. So split you test and train set accordingly. You don't expect the model to predict, if you haven't trained it. See stratified sampling for starters.

Though your predictors might be categorical or ordinals, they can be treated as numericals.

You have built in functions like predict_proba to find class probability. The references for this can be found in above links. But go through how they work and what is the policy for selecting best plane in any classification, so that you don't feel like you are using a black-box. Since you are saying that you are a beginner, pandas will be a very useful module for reading data into dataframes and mending it as you like.

Hope this clears something.


When to stop calculating values of each cell in the grid in Reinforcement Learning(dynamic programming) applied on gridworld Considering application of Reinforcement learning(dynamic programming method performing value iteration) on grid world, in each of the iteration, I go through each of the cell of the grid and update its value depending on its present value and the present value of the taking action from that state. Now

How long do I keep updating value of each cell? Shall I keep updating unless the change in the previous and the present value function is the least? I am not able to understand how to implement the stopping mechanism in the grid-world scenario(discount not considered)
Is the value function the values of all the grids in the grid world?

1- You should set a threshold (a hyper-param) that will allow you to quit the loop.

Let V the values for all state s and V' the new values after value iteration.

if $\sum_s|V(s) - V’(s)| \le threshold$, quit

2 - V is a function for every cell in the grid yes because you need to update every cell.

Hope it helps.


Time Series Generation - Multi Dimensional Time Series Data Disclaimer: Mathematicians please don't be mad at me for the use of some of the terminologies in this post. I am an Engineer. :-)

Background:
So I am currently working on a problem where I have to generate a time series sequence of a process in which n actors are moving in a 2d space. But i don't know if this is even possible .The process being learned by some machine learning model M.

BTW! I have never worked with time series data, but have a good experience with training models on images and signals, without a sequence, so i have been reading up on it on the go.

So to start of with trying something very simple, I took a football player position dataset from : Here . And I am trying to model it as a supervised learning problem where I try to predict the positions of n players at timestamp T, given that at timestamp T-1. But I very quickly realised that it wouldnt work because the positions of the players also depend on the position of the ball and that of the opponent team players.

Anyway my questions are as follows :-
1. How do i model the dataset? Will it just be a (Nx2xNo.Timestamps) like 3-d tensor dataset(N corresponding to the players. 2 for the x-position and y-position. and No. of timestamps as the last dimension)?

Is my way of modelling the time series generation problem as a supervised learning problem correct?
what Preprocessing steps should i be looking at? Also how do i handle missing values.
The reason why i dropped the idea about using the soccer dataset : Here again because it only includes positions of one team. The other team didnt wear sensors :-( . I read something about exogenous variables also affecting the process, when reading something about the ARIMA model.
If all this is possible and I hope it is (cos impossible is nothing!) what models should i be looking at? Because i ultimately have to work on this problem on a very different dataset... I have past experience with training Neural Network models like CNNs and ANNs, and feel very comfortable working with Neural Networks, and ideally would love to do so here. Uptil now my research has pointed me towards LSTMs RNNs and the ARIMA model.

Please guide me on the same as i'm very new to time series analysis.

Time series data must contain all your observations with some standard effect of time (Bit obvious here). If i wants to test theory on some model & need some dataset then the parameters would be like, Timestamps * no. of features (includes player's position with respect to a source). A 2d tensor would suffice. My reason for not making a 3d tensor is that it would lead to more complex scenario where i would have to co-relate the 3rd dimension (No of players [N]) with first 2 dimensions to predict. Better to simplify the positions & in a single row i can have multiple labels as every position will matter to my model.

Time series generation generally falls under continuous prediction with previous data predicted taken to be an observation. I would rather make it fall under Reinforcement learning. Yes you can work with supervised learning in mind but try RL approach as well.

As per the missing values i would say that remove them if they don't make more than 10-15% of data. There are no fix bars on the mentioned percentage. If it is more than that please fill in with interpolation or rolling average (both benefited me). Rest pre-processing depends upon type of data, normalise data, remove outlier etc.

Yes it will affect the data, but you can generate data using sign wave's different fluctuation for your testing or use any other functions to generate a signal(scipy preferred).

Currently i am testing bidirectional lstm - CNN combination for my time series, yes ARIMA is good but doing a little of this expirmenet won't hurt. I would say go for CNN-Any of RNN combination for time series.

Hope this helps.


Getting a ValueError from train_test_split I'm working on this dataset.  I'm trying to select features using Random Forest.  This is the relevant code:

x_train, y_train, x_test, y_test = train_test_split(train, test_size = 0.3)

sel = SelectFromModel(RandomForestClassifier(n_estimators=100))
sel.fit(x_train, y_train)
sel.get_support()
selected_feat = x_train.columns[sel.get_support()]

The code x_train, y_train, x_test, y_test = train_test_split(train, test_size = 0.3) throws the following ValueError:

Traceback (most recent call last):
  File "C:/Users/security/Downloads/AP/Boston-Kaggle/Boston.py", line 25, in <module>
    x_train, y_train, x_test, y_test = train_test_split(train, test_size = 0.3)
ValueError: not enough values to unpack (expected 4, got 2)

I don't know what it means by not enough values.  The dataset has a big dimension.

This is the implementation of the train_test_split function

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

Split your train into X and y and try again!


Does adding of many FC layers during re-training increase the model size ? Are there any ways to optimize the size of model? I am re-training a pretrained model VGG16.

In the last layers, im using two FC layers of size 2048 each, with dropout=0.5.

When i saved the model, The size of the model was found to be 2 GB (which is so huge). I have trained this model for 9 classes.(900 images)

Is there any way, where i could reduce the size of the model but the performance shouldn't be affected ?

The reason is i have to use this model in AWS cloud, where the systems are having 8GB ram. I am worried if their performance would be affected by such models.

You can change the type of your weight, you will loose precision but reduce the size of the weights. You can also do some regularisation during training to learn sparse weight matrix

You can read these two articles :

Deep compression optimisation techniques for inference
Model quantisation for production level nn inference


How can I get total sum of each group by using pandas I have a dataframe shaped like below

size ind weight weighted_hour
1    A    17       40
1    B    20       41
1    C    19       42
2    A    50       30
2    B    20       10
2    C    13       16
3    A    17       19
3    B    21       23
3    C    23       30

and I want to insert some sum of criteria like below

size ind     weight weighted_hour Total_WH
1    A       17       40          57
1    B       20       41          61
1    C       19       42          61
1    Total   56       123         179
2    A       50       30          80
2    B       20       10          30
2    C       13       16          29
2    Total   83       56          139
3    A       17       19          36
3    B       21       23          44
3    C       23       30          53
3    Total   61       72          133

I prefer to use pandas but if someone can suggest more efficient method
It's good either.
Give me some advice. Thank you!

Assuming you have the above dataframe as df:

df.weight = df.weight.astype(int)
df.weighted_hour = df.weighted_hour.astype(int)
df["Total_WH"] = df["weight"] + df["weighted_hour"]

df1 = df.groupby(['size'], as_index=False).sum().assign(ind='Total')

df = pd.concat([df, df1]).sort_values('size')


Resize instead of transposed convolutions I'm trying to build a decoder version of ResNet, i.e. one that goes from the prelogits layer and attempts to recreate the image. I can get it working by using transposed convolutions, but the quality of reconstructed images isn't that great (checkerboard artifacts). I've read that the state of the art is to use UpSampling (i.e. resize) followed by a convolutional layer instead. Whenever I try this my network no longer converges and the images produces are all white/black.

What am I missing?

Presumably, you have the seen the Distil article discussing checkerboard artifacts due to transposed convolutional layers, where they recommend upsampling (resizing) and then doing regular convolution. This is known to be a reasonable approach, so it is likely something specific to your case. Without architectural details, I can only guess that it might be (a) your hyper-parameters have been tuned to work with transposed conv layers, so it breaks with this different approach, or (b) you have introduced some kind of vanishing gradients or convergence issue (e.g., not using residual connections, not using batch norm, too large of a learning rate, etc...). The loss (e.g., $L_2$ reconstruction) and it's gradients may be of a different magnitude than what the other (hyper-)parameters were expecting, for example. Feel free to post back with more info.


intercalate matrices by Rows in R How can I concatenate matrices with unequal rows by intercalating the rows? For example, I want to concatenate A matrix ( dim(A) = 15 20 ) and B matrix (dim(B) = 17 20) and I want the result to be a matrix of 32 rows by 20 columns. I want result matrix with one row from matrix A and the second row from matrix B. Simple rbind just stacks up one matrix on top of other.

The following function works in case of equal number of rows:

# drop false ensures that coercion does not change the data structure.
interleave(A, B, drop=FALSE)

But I want the rows to be interleaved and then the remaining rows from B matrix to be added at the end. If the number of rows from one matrix is added and no more are remaining then the remaining rows should get rowbind (normally stacked).

Is there any way that I can tweak this interleave function from gdata package in R? or is there any other way?

The following approach works perfectly,

l <- list(a=A,b=B)
dataset <- do.call(rbind, l)[order(sequence(sapply(l, nrow))), ]


How to think about - and sometimes impute - geographic distances I have a dataset with one of the (important) features being the geographic distances from NYC. Of course, some of the values are missing.... The goal is predicting whether people with certain attributes (proximity being one of them, and the typical age, sex, education, etc. being the others) will engage in an activity in NYC (e.g., visiting MOMA, taking a Broadway show, moving to the city altogether, enrolling in an area school, things like that).

My basic question is - missing values aside - is it correct to just take distances into account "as is" or should they somehow be divided between "driving/train distances" and "flying distances" (essentially, converting them into "the number of hours it takes someone to get to NYC by the most efficient means")?

If we take Los Angeles and Richmond, VA as examples - the distance to NYC from LA is about 10 times that of Richmond; flight times are only 4 times longer, but flight time from LA and drive time from Richmond are approximately the same. So what's the right way to think about that?

And once the right approach to distances is determined, how does one go about imputing distances for missing values?

I don't think time alone is what you are after, although it would certainly be useful.

It sounds to me like you would also want to consider the cost of transport.

Yes the drive and flight may take the same amount of time, but you could be looking at orders of magnitude difference in the total cost. If you are only looking at extremely wealthy people this may not matter, but otherwise this could have as big an impact as time alone.

My advice would be to use the existing distance column to approximate the minimum cost of transportation, similar to how you would approximate the minimum time of transportation. I would then use both time and cost as inputs to my model.


Why Mini batch gradient descent is faster than gradient descent? According to me:
Mini Batch Gradient Descent :
1.It takes a specified batch number say 32.
2.Evaluate loss on 32 examples.
3.Update weights.
4.Repeat until every example is complete.
5.Repeat till a specified epoch.
Gradient Descent :
1.Evaluate loss for every example.
2.Update loss accordingly.
3.Repeat till a specified epoch.
My questions are:
1.As Mini batch GD is updating weights more frequently shouldn't it be slower than normal GD.
2.Also I have read somewhere that we estimate loss in SGD (ie. we sacrifice some accuracy in loss calculation for speed). What does it means and does it helps in increasing speed.

It is slower in terms of time necessary to compute one full epoch. BUT it is faster in terms of convergence i.e. how many epochs are necessary to finish training which is what you care about at the end of the day. It is because you take many gradient steps to the optimum in one epoch when using batch/stochastic GD while in GD you only take one step per epoch. Why don't we use batch size equal 1 every time then? Because then we can't calculate things in parallel and computation resourses are not used efficiently. It turns out in every problem there is a batch size sweet spot which maximises training speed by balancing how parallelized your data is and number of gradient updates per epoch.
mprouveur answer is very good; I'll just add that we deal with this problem by simply calculating average or sum loss over all batches' losses. We don't really sacrifice any accuracy i.e. your model is not worse off because of SGD - it's just that you need to add up results from all batches before you can say anything about the results.


Cleaning a certain feature to predict salary using Machine Learning Info:
I am working on a dataset, and i would like to create a model that would predict salary.
Columns are as follows:
Index(['ID', 'Salary', 'DOJ', 'DOL', 'Designation', 'JobCity', 'Gender', 'DOB',
       '10percentage', '10board', '12percentage', '12board', 'CollegeTier',
       'Degree', 'Specialization', 'collegeGPA', 'CollegeCityTier',
       'CollegeState', 'English', 'Logical', 'Quant', 'Domain',
       'ComputerProgramming', 'ElectronicsAndSemicon', 'ComputerScience',
       'MechanicalEngg', 'ElectricalEngg', 'TelecomEngg', 'CivilEngg',
       'conscientiousness', 'agreeableness', 'extraversion', 'nueroticism',
       'openess_to_experience'],
      dtype='object')

Issue:
The train dataset contains around 3000 entries. For the feature 'JobCity', there are around 200 unique values. Out of 3000 entries, 180 are cities whose value counts are below 10.
For example, the city 'Agra' occurs only twice and 'Ranchi' occurs only once.
I plan to encode all the cities to feed into the machine learning (polynomial regression) model.
How should the cities with value counts less than 10 be handled?

There are two ways to handle this:

Aggregating values and feature engineering

Instead of simply inputting the individual city, you could try to aggregate this value into a new variable "JobCitySize" to cluster cities by inhabitant size or a variable "Region" to cluster by geographic region. This will increase cases and also help solve the question of how to deal with unseen data.

Defining NA or unseen values

With any model and categorical encoding you have to think ahead what you want to do with "unseen data" in your case cities that aren't present in your training data set. Do you want to set them as NA, give a standard value ("Other"), etc.? Whatever process you use for these values you could also use for any value in your data set that falls below a threshold of "usefulness".


How to determine input shape in keras? I am having difficulty finding where my error is while building deep learning models, but I typically have issues when setting the input layer input shape.

This is my model:

model = Sequential([
Dense(32, activation='relu', input_shape=(1461, 75)),
Dense(32, activation='relu'),
Dense(ytrain.size),])

It is returning the following error:

 ValueError: Error when checking input: expected dense_1_input to have 3

 dimensions, but got array with shape (1461, 75)

The array is the training set from the kaggle housing price competition and my dataset has 75 columns and 1461 rows. My array is 2 dimensional, so why are 3 dimensions expected? I have tried adding a redundant 3rd dimension of 1 or flattening the array before the first dense layer but the error simply becomes:

ValueError: Input 0 is incompatible with layer flatten_1: expected

min_ndim=3, found ndim=2

How do you determine what the input size should be and why do the dimensions it expects seem so arbitrary?

For reference, I attached the rest of my code:

xtrain = pd.read_csv("pricetrain.csv")
test = pd.read_csv("pricetest.csv")
xtrain.fillna(xtrain.mean(), inplace=True)
xtrain.drop(["Alley"], axis=1, inplace=True)
xtrain.drop(["PoolQC"], axis=1, inplace=True)
xtrain.drop(["Fence"], axis=1, inplace=True)
xtrain.drop(["MiscFeature"], axis=1, inplace=True)
xtrain.drop(["PoolArea"], axis=1, inplace=True)
columns = list(xtrain)
for i in columns:
    if xtrain[i].dtypes == 'object':
        xtrain[i] = pd.Categorical(pd.factorize(xtrain[i])[0])
from sklearn import preprocessing

le = preprocessing.LabelEncoder()
for i in columns:
    if xtrain[i].dtypes == 'object':
        xtrain[i] = le.fit_transform(xtrain[i])
ytrain = xtrain["SalePrice"]
xtrain.drop(["SalePrice"], axis=1, inplace=True)
ytrain = ytrain.values
xtrain = xtrain.values
ytrain.astype("float32")

size = xtrain.size
print(ytrain)
model = Sequential(
    [Flatten(),
     Dense(32, activation='relu', input_shape=(109575,)),
     Dense(32, activation='relu'),
     Dense(ytrain.size),
     ])
model.compile(loss='mse', optimizer='adam')
model.fit(xtrain, ytrain, epochs=10, verbose=1)

Any advice would be incredibly helpful!

Thank you.

The number of rows in your training data is not part of the input shape of the network because the training process feeds the network one sample per batch (or, more precisely, batch_size samples per batch). And in input_shape, the batch dimension is not included for the first layer. You can read more on this here.

So, the input shape for your problem will be:

input_shape=(75, )


Calculating possible number of configuration I am wondering how did they get the $19200$ possible configurations? Like, $5^6 = 15625$, where $6$ is the number of hyper-parameters:

The total number is:

$$5 \times 5 \times 6 \times 4 \times 4 \times 8$$

which is equal to $19200$. Here, we just count the number of possible values for each parameter.


How to integrate time series data into normal features for machine learning? I confront a problem where one data source is a "normal" DF with customers as rows (each customer occurs once) and static customer features as columns. The other DF other hand is a big pile of timestamped data of the aforementioned customers with event features and event timestamps (each customer occurs many times according their activity). The task is a simple classification by customer.

My problem is how to integrate the time series data into the first DF as columns? Should I simply compute somekind of aggregations from the second DF? If so, what kind of? If not, what is the canonical way to merge these two kind of data sets?

Assuming: DF1 and DF2 look something like:

So going to make some assumptions about what you want to do:

(1) you want a classification model like: I want to classify buyers/non buyers based on their overall attributes as well as their historical activity for a given time window:
- then you first have to create features based on the transaction data frame DF2, which would rollup transactions and performance information to customer id level for each customer. Some metrics that you can create from the detail transactions table DF could be recency of visits,  frequency of visit, average purchase amount, recency of conversions etc.., so you would have customer id's that did not buy anything over the time period and customer that bought at least once over the time window.

join the first table and the rolled up data by customer id to have all features and label (buyer,non buyer) given an appropriate time window

(2) if you want a model for classification at each timestamp for each customer id then most likely your first data frame contains some sort customer characteristics and demographics that do not change overtime or at least over the chosen time window. In that case you join the two data frames by customer id. Here each customer could be buyer or non buyer depending on the each timestamp


Shifted feature distribution across different datasets I am trying to validate a classifier using two different training and testing datasets.

The feature I am considering is a feature constructed doing the fold-change between two original features, i.e. $log2(feat_A / feat_b)$

The problem is that, across the two different datasets, I see the same distribution of fold-change across groups but with different values. See, in the images attached, the values for each group (each box) in the different datasets (each plot).

What may be causing the differences in fold-change? In this case the fold-change is different also in sign, which means that while in one dataset $feat_a$ is greater than $feat_b$, the opposite is true for the other dataset. However, the pairwise relation between the two groups (the two boxes in each plot) is maintained.

I was thinking about different normalization procedures on each feature across different dataset, which may explain the shift of the distribution of the fold-change. What other problems may be in these datasets?

It'd be easier to understand if the plots were labeled. Nonetheless, I think I understand the question, and perhaps it's just sampling error. Why do you have two different pairs of train and test data? Are they from different sources? In that case, you probably don't want to be using the same classifier on both. Also, what do you mean 'validate' using two different train and test sets? Classifiers are trained on a train set, and validated on a test set, what do you mean you're validating on two different train/test pairs?


How to normalize test data according to the training data if the normalization on the training data is performed row wise? I read on several places about the normalization of features in the machine learning method. But I normalize my training data row-wise as shown in the following code. I showed only two samples of training data. My question is that while performing the normalization on test data, should I choose the minimum and maximum value of each test sample to normalize each test data or should I uses minimum and maximum values from the training data?
As an explanation in first row -3 is one feature, -2 is second 0 is third and 3 is fifth feature. And second row is second sample comprising of 5 features from -4 to 2. Similar to all other machine learning algorithms each row correspons to one sample consisting of 5 features.

data = np.array([[-3,-2,0, 2,3],[-4,-1,0,3,2]])
print(data)

print(data.shape)
for i in range(len(data)):
    print("i: ",i)
    old_range = np.amax(data[i]) - np.amin(data[i])
    new_range = 2
    new_min = -1
    data_norm = ((data[i] - np.amin(data[i])) / old_range)*new_range +
new_min
print(data_norm)

Result

[-1.         -0.66666667  0.          0.66666667  1.        ]
[-1.         -0.14285714  0.14285714  1.          0.71428571]

You usually want to normalize features as you also pointed out. In case of tabular data almost every machine learning implementation will expect you to provide the features as columns and observations as rows. In your case if you have a feature in a row you may want to transpose it or if they are not the same features than you may apply a different transformation.

If you do normalization on features you have to use the same transformation on test data that you used on train data. (if you use i.e. sklearn implementations they will take care of it for you)


Are there any algorithms for a classification problem involving unlimited classes, and only a few instances per class The Scenario: A group of people must summarize specific parts of speeches they hear. They hear a new speech every day, and it's possible that multiple members of group are listening to the same speech.

The Question: What ml algorithm or approach could you take to link summaries that refer to the exact same part of the exact same speech?

Why this question is interesting: most classification problems deal with a large amount of instances and only a few classes. This problem is about dealing with very few instances (1-5) for a growing number of classes (The specific part of the specific speech).

Yes there are couple of methods. I recommend reading the Siamese Networks paper. From there you can check more modern approaches like Prototypical Networks and so on. In general the keyword you are looking for is few shot learning.
All this approaches basically depend on finding similiarities between two or more inputs. One input is your unknown input and the others are your known samples. Your network will predict a class corresponding to maximum similarity.


How effective would a data-driven vaccination program be over a simple rules based approach The rollout of vaccination programs tends to be based on a given set of rules (advice) devised by government UK example. Such a rule-set are generally limited in complexity in order to be consistently applied by humans administering the roll-out.
There is a strong association between age and risk nature but it appears below the age of 60 that other factors such as sex/deprivation/ethenicity predominate. There is also mounting evidence towards mental disorders holding a significant risk factor risk factors and mental disorders
Is there any openly available papers or modelling showing a data-driven prioritization of vacination (for example using machine-intelligence), especially in the scenario of limited vaccine availability, would be more effective than simply using age alone as the predicate.

The reason for the rules based approach is the simpson paradox.

In short, the way a test group is sampled can easily overturn the conclusion of a study. It is very important to include sufficient patients from a particular risk group in the study.
This is why rules are set.
This approach however is the basis of causality theory. A data driven-approach would be as followed:

experts flag expected causal patterns, i.e. draw causal graph of the study. [This is the mathematical formulation of risk factors]
Causal graph generates ideal distribution of study and tools to draw conclusion


How to handle columns with large/infinte values in dataset for ML classification Computed a column using a formula (formula does't involve any log functions, just a group by with .sum()), but as expected this column would result in infinite/exponential values like below:

-inf
 nan
 inf
-3.000e+32
 7.3297+23 ...etc (similar data)

My doubt is how should I handle this column as I could also not avoid this column for prediction for a binary classification problem. What are the techniques to use before using this column in model classifier.

Thanks!

Depends a bit on the model you are going to run. I will explain a bit for Linear Models and for Decision trees ensembling (gradient boosting and random forest)

Decision tree

Not much to do, when the tree is built each branch will choose a split. If there is any information gain with a large value, it will choose it and make a split.

Some implementations, as catboost have a quantization in it. The split could be done eventually in something like 95% quantile and that will work with large values.

Generalized Linear Models

You will need to change this. My suggestion is creating a feature based in the values of the column (if df.col1> 9999: 1 else 0).
And then Winsorizing by a certain threshold.


Exponential Linear Units (ELU) vs $log(1+e^x)$ as the activation functions of deep learning It seems ELU (Exponential Linear Units) is used as an activation function for deep learning. But its' graph is very similar to the graph of $log(1+e^x)$. So why has $log(1+e^x)$ not been used as the activation functions instead of ELU?

In other words what is the advantage of ELU over $log(1+e^x)$?

ReLU and all its variants ( except ReLU-6 ) are linear i.e $ y = x $ for values greater than or equal to 0.

This gives ReLU and specifically ELU an advantage like:

Linearity means that the slope does not plateau or saturate when $x$ becomes larger. Hence, the vanishing gradient problem is solved.

Now, the graph $ y = log( 1 + e^x ) $ isn't linear for values > 0.

For larger negative values, the graph produces values which are very close to zero. This is also found in sigmoid where larger values produce a fully saturated activation. Hence, $ y = log( 1 + e^x ) $ can raise problems which sigmoid and tanh suffer.

About ELU:

ELU has a log curve for all negative values which is $ y = \alpha( e^x - 1 )$. It does not produce a saturated firing for some extent but saturates for larger negative values.

See here for more information.

  Hence, $ y = log( 1 + e^x ) $ is not used because of early saturation
  for negative values and also non linearity for values > 0. This may
  produce problems and even bring down some features which ReLU and
  variants exhibit.


How to cluster label (in a multilabel classification problem) which mostly appear together in a class To cluster label (in a multilabel classification problem) which mostly appear together in a dataframe? For example, I have this dataframe:
text     |   genre
===========================
text 1   | [action,mistery,horror,thriller]
text 2   | [drama,romance]
text 3   | [comedy,drama,romance]
text 4   | [scifi,mystery,horror,thriller]
text 5   | [horror,mystery,thriller]

How can I cluster the tag that often appear together? For example, genre "mystery","horror","thriller" often appear together (3 times), genre "drama","romance" often appears together (2 times).

The difficulty here is the design: if one just represents every "class" as an instance with a vector of its co-occurrences with other "classes", then the clustering will only group the full group of instances/classes by how similar they are with other groups.
Instead I think that an instance should represent a pair of "classes", using the PMI as a measure of how strongly they are associated.
An advanced option would be to represent each "class" as a node in a graph and add an edge between any two classes which occur together, weighted by their PMI value. Then use a graph-clustering algorithm to group the "classes".


is there any whatever2vec to generate one vector for one document? Word2Vec generates a feature vector for each word. I was wondering if exist something2vec that generates one feature vector for one document. In this way I could compute the distance between two documents by only comparing two feature vectors.

Thanks.

Yes, there are such things, but they aren't really related to word2vec.

First of all, Bag of Words is the old "something2vec" approach of representing documents in a vector space. Of course, the resulting vectors are very sparse, and distance does not always behave well in such cases, and it is better to use dot-product based similarities like cosine.

But you can reduce the dimensionality of this vector space with SVD or NMF (Non-Negative Matrix Factorization), and this will give you what you want: dense vectors of small dimensionality for which you can compute the distances.

Then you can go further and use Topic Modelling, which also will give you a vectorial representation of documents, and you can use it for comparing them.


How to render environment in Tensorforce? How can one render the environment using the Tensorforce library?

I've tried calling environment.render, but it says that the function does not exist. This is my code:

from tensorforce.agents import Agent
from tensorforce.environments import Environment
from tensorforce.execution import Runner

# Create an OpenAI-Gym environment
environment = Environment.create(environment='gym', level='MountainCarContinuous-v0')

agent = Agent.create(agent='random', environment=environment)

# runner = Runner(agent=agent, environment=environment) # Initialize the runner
# runner.run(num_episodes=NUM_EPISODES) # Start the runner
# runner.close()

# Train
for ep in range(NUM_EPISODES): # Number of episodes

    print('********Episode ' + str(ep) + '********')

    # Initialize episode
    states = environment.reset()
    done = False
    step = 0

    while not done: # Episode timestep
        actions = agent.act(states=states)
        states, done, reward = environment.execute(actions=actions)
        agent.observe(terminal=done, reward=reward)
        environment.render() # Gives error

environment.close()
agent.close()

And this is the error I get:

Traceback (most recent call last):

  File "c:\users\user\reinforcement learning\rl.py", line 179, in <module>
    environment.render()

AttributeError: 'OpenAIGym' object has no attribute 'render'

In case you use
https://github.com/tensorforce/tensorforce/blob/master/examples/act_observe_interface.py ,
the following modification works.
Import gym:
import gym
from gym import wrappers

then define a Tensorforce environment with a Gym environment:
    env = gym.make('CartPole-v1')
    env = wrappers.Monitor(env, 'tmp', force=True)

    environment = Environment.create(environment=env, max_episode_timesteps=500)
    # environment = Environment.create(environment='/Users/rondelion/git/tensorforce/benchmarks/configs/cartpole.json')

and render the Gym environment:
            ....
            num_updates += agent.observe(terminal=terminal, reward=reward)
            env.render()
            sum_rewards += reward
            ....


Train at 99% and Validation split accuracy is not more than 70% I am training a model that get accuracy on train set up to 99%
but the validation split test not increase more than 70-72%
this is how my model is configurated:
model = Sequential()
#model.add(TimeDistributed(Dense(64, activation='linear')))
#model.add(GRU(512, return_sequences=True, activation='linear',))
model.add(LSTM(128, return_sequences=True, activation='linear'))
model.add(Conv1D(64, 64, strides=1, padding='same'))
model.add(Dense(128, activation="linear"))#, kernel_regularizer=regularizers.l1_l2(l1=1e-4, l2=1e-6)
model.add(Dropout(0.2))

#model.add(Conv1D(16, 16, strides=1, padding='same'))
#model.add(TimeDistributed(Dense(32, activation='linear')))
#model.add(GRU(512, return_sequences=True, activation='linear'))
model.add(LSTM(128, return_sequences=True, activation='linear'))
model.add(Conv1D(64, 64, strides=1, padding='same'))
model.add(Dense(128, activation="linear"))#,kernel_regularizer=regularizers.l1_l2(l1=1e-4, l2=1e-6)
model.add(Dropout(0.2))
#model.add(BatchNormalization())

model.add(Dense(1,activation='linear'))

model.compile(optimizer='adamax',loss="mse",metrics=['accuracy'])
model.fit(X_train,y_train,epochs=8000, batch_size=256, verbose=1, validation_split=0.1, callbacks=callback)

what can be the issue?

This means your model is overfitting the train set. Try simplifying the model by reducing the number of neurons in each layer. That will reduce your train accuracy but may allow your validation accuracy to increase.
You know that you are not overfitting when both train and validation accuracy hover around the same value.


How to calculate classification accuracy with confusion matrix? I have Train and Test data, how to calculate classification accuracy with confusion matrix ? Thanks

@attribute outlook {sunny, overcast, rainy}
@attribute temperature {hot, mild, cool}
@attribute humidity {high, normal}
@attribute windy {TRUE, FALSE}
@attribute play {yes, no}

Train:

1   sunny       hot     high    FALSE   no
2   sunny       hot     high    TRUE    no
3   overcast    hot     high    FALSE   yes
4   rainy       mild    high    FALSE   yes
5   rainy       cool    normal  FALSE   yes
6   rainy       cool    normal  TRUE    no
7   sunny       cool    normal  FALSE   yes
8   rainy       mild    normal  FALSE   yes
9   sunny       mild    normal  TRUE    yes
10  overcast    mild    high    TRUE    yes
11  overcast    hot     normal  FALSE   yes
12  rainy       mild    high    TRUE    no

Test:

overcast    cool    normal  TRUE    yes
sunny       mild    high    FALSE   no

Rules found:

(humidity,normal), (windy,FALSE) -> (play,yes) [Support=0.33 , Confidence=1.00 , Correctly Classify= 4, 8, 9, 12]
(outlook,overcast) -> (play,yes) [Support=0.25 , Confidence=1.00 , Correctly Classify= 2, 11]
(outlook,rainy), (windy,FALSE) -> (play,yes) [Support=0.25 , Confidence=1.00 , Correctly Classify= 3]
(outlook,sunny), (temperature,hot) -> (play,no) [Support=0.17 , Confidence=1.00 , Correctly Classify= 0, 1]
(outlook,sunny), (humidity,normal) -> (play,yes) [Support=0.17 , Confidence=1.00 , Correctly Classify= 10]
(outlook,rainy), (windy,TRUE) -> (play,no) [Support=0.17 , Confidence=1.00 , Correctly Classify= 5, 13]

A confusion matrix is a cross tabulation of your predicted values against the true observed values, and (test) accuracy is emperical rate of correct predictions. So in this case you'll need to

Predict the 'play' attribute for your test set.  (Currently you don't have a method to predict your second test case, so for the sake of argument let's assume your model would predict yes for the sunny example.
The following method of keeping track of your predictions is reffered to as a confusion matrix.  The top labels are prediced

               Predicted
         +----------------+
         ¦     ¦ yes ¦ no ¦
Oserved  ¦ yes ¦ 1   ¦ 1  ¦
         ¦ no  ¦ 0   ¦ 0  ¦
         +----------------+

Here the first 1 is from your first test case and the second 1 is from the misclassified second test case.

Calculate accuracy,

Accuracy = (# correct predictions)/(# total predictions) = 1 / 2 = .50.


Alternatives with better GPU than Google Colab Pro I am currently running/training MAchine learning models that are very GPU expensive, Google Colab Pro is not giving me enough GPU/RAM
Is there any alternatives with better GPU  and more RAM than Google Colab Pro??

Here are a few options:

Commercial cloud provider: AWS is the go-to cloud provider for virtually any need, they have options for everything. Check out their options for Deep Learning.
You could also buy your own hardware
Finally from a data science perspective there is also the option of modifying the parameters of the model and/or data so that the training fits with the hardware limitations.


Confusion about Keras' skipgram and sampling table utilities I'm fairly new to ML, so as a learning exercise to get familiar with Keras I'm trying to learn some word2vec style embeddings from a dataset. I'm confused about the behavior of the skipgram utility, in particular the sampling table argument that's supposed to be populated from make_sampling_table. I understand the idea behind it, but I'm seeing some strange outputs.

As a toy example, let's pretend that we have the following parameters:

vocab_size = 3
sequence = [0,1,0,1,1,0]

So a small vocabulary with 3 words in it, only 2 of which show up in the sequence. Generating the skipgrams & labels works fine without a sampling table at all:

pairs, labels = skipgrams(sequence, vocab_size)
print("pairs: " + str(pairs) + " labels: " + str(labels))

OUTPUT: pairs: [[1, 1], [1, 2], [1, 1], [1, 2], [1, 1], [1, 1], [1, 1], [1, 2], [1, 1], [1, 1], [1, 1], [1, 1]] labels: [0, 0, 1, 0, 0, 1, 1, 0, 1, 0, 1, 1]

But when I try to pass in a sampling table so as to not oversample the common words, I get no outputs at all:

pairs, labels = skipgrams(sequence, vocab_size, sampling_table=make_sampling_table(vocab_size))
print("pairs: " + str(pairs) + " labels: " + str(labels))

OUTPUT: pairs: [] labels: []

My question is: what's going on here? Why does the act of passing a sampling table into the skipgrams method make it produce no output?

Thanks for the help!

After looking through the code for the skipgram method, I figured it out: it has to do with the sampling_factor parameter. It defaults to 1e-5, which means that for such a small sampling factor it's almost never going to actually pick a sample unless your dataset & vocabulary size is big enough to actually get some samples. If you have a small vocabulary (like in the toy example), you should set that to something much higher: when I set it to 1.0 (i.e. take a sample as often as possible) I got lots of pairs, even for the toy example.


Clusters: how to improve results for text classification I am trying to classify texts using kmeans, TfidfVectorizer, PCA. However, it seems that many texts are not correctly classified as you can see: I have texts in cluster2 that should be in Cluster 0 or 1.
My question is on how to improve the results, if increasing the number of clusters or adding more constraints (like specific words to look at for clustering texts).

Help and suggestions will be greatly appreciated.

Code:

def preprocessing(line):
    line = re.sub(r"[^a-zA-Z]", " ", line.lower())
    words = word_tokenize(line)
    words_lemmed = [WordNetLemmatizer().lemmatize(w) for w in words if w not in stop_words]
    return words_lemmed

vect =TfidfVectorizer(tokenizer=preprocessing)
vectorized_text=vect.fit_transform(df['Text'])
kmeans =KMeans(n_clusters=n).fit(vectorized_text)

cl=kmeans.predict(vectorized_text)
df['Predicted']=pd.Series(cl, index=df.index)

df.groupby("Predicted").count()

kmeans_labels =KMeans(n_clusters=n).fit(vectorized_text).labels_

pipeline = Pipeline([('tfidf', TfidfVectorizer())])
X = pipeline.fit_transform(df['Text']).todense()
pca = PCA(n_components=n).fit(X)
data2D = pca.transform(X)

kmeans.fit(X)
centers2D = pca.transform(kmeans.cluster_centers_)
labels=kmeans.labels_
cluster_name = ["Cluster"+str(i) for i in set(labels)]

It can be improved in many ways. However, the best start would be if you elaborate on your pipeline together with choosen parameters, as well as your data.


Is it possible to have a default class in multi class classification? In the general text classification problem, training a machine learning model to detect if a text belongs to one of N number of classes always yields a value in N. Even if the text that was passed to the model is very far outside any of the N classes (like say you trained it on categories for movie genres, but someone passed a recipe to the classifier), it will always choose one of N as the output even though the text passed to it doesn't belong to any of the classes.

Is there a method to have an additional class for 'unknown' so that if the model output is one of the N types but matches it with very low probability then assign it to the default or unknown class?

EDIT: we are using LinearSVC from sci kit learn

Depends on the model but this would be a post-processing move.

E.g. in a recent problem I trained a random forest classifier and then classified any data point where the probability of the likeliest class was less than 50% as unknown.


what if conda has no package? I just installed tensorflow environment with conda.
but problem is conda does not have some packagefor example 'fastai'
pip3 only has this package
as you know pip install and conda install has different package directory
so if conda does not have the package what you have to do?
 I dont want to move pip package to conda manually....

Activate your conda environment, then use the pip that will also be in your environment (just like the Python interpreter is that of you activated environment).

For example:

source activate your_tf_env          # now we are in the conda env
which -a pip                         # should list all pip executables

The first pip one should be in your_tf_env somewhere.

Now you can simply install your package, e.g. using pip install fastai, which will use the pip of your environment and the package will also be installed in your environment. You do not need to manually copy the files anywhere.

With your conda env still activated, you can then confirm the package is installed by running:

conda list                     # list all installed packages
conda list | grep fastai       # grep filters results, e.g. for  "fastai"


How to define a suitbale number of centroids for fuzzy c-means? In Fuzzy c-means, we have to put how many centers(centroids) in the code. I am wondering how many centroids are suitbale ?

How to define a suitbale number of centroids for fuzzy c-means?

You can try the exact same heuristics you would use for k-means.

But usually, you would do this:

try some parameters by experience
run the algorithm
carefully inspect the result
go back to 1 and try other parameters until you have found something interesting or are tired of trying

There is no "correct" solution in clustering.

It's about discovering some new pattern and you cannot out the "new" into an optimization equation. The point is to try, and try again. It is not a drop-in replacement for classification.


Solve a pair of coupled nonlinear equations within certain limits This answer to this question works only for situations in which the desired solution to the coupled functions is not restricted to a certain range.

But what if, for example, we wanted a solution such that 0 < x < 10 and 0 < y < 10?

There are functions within scipy.optimize that find roots to a function within a given interval (e.g., brentq), but these work only for functions of one variable.

Why does scipy fall short of providing a root solver that works for multi-variable functions within specific ranges?  How might such a solver be implemented?

As a workaround, you could minimize another function that includes both the objective and the constraints, then check if sol.fun is (numerically) equal to zero.

from scipy.optimize import minimize
import numpy as np

f = lambda x: np.sin(x).sum() #the function to find roots of
L = np.array([-1,-1]) #lower bound for each coordinate
U = np.array([1, 1])  #upper bound for each coordinate
g = lambda x: f(x) **2 +  max(0, (L - x).max(), (x - U).max())
sol = minimize(g, [0.5,0.5])

Also, scipy.optimize seems to have some optimisers that support rectangular bounds, i.e. differential_evolution (since version 0.15.0).


SVM radial basis generate equation for hyperplane I need to generate an equation for hyperplane, I have two independent variables and one binary dependent variable.

Regarding this following equation for svm , $f(x)=sgn( sum_i alpha_i K(sv_i,x) + b )$

I have two independent variables (say P and Q) with 130 point values for each variable. I used svm radial basis function for binary classification (0 and 1) and I calculated for radial basis kernelized case, and now I have

One column of 51 y (i) alpha (i) or (dual coefficients).
Two columns of 51 sv (support vectors)for P and Q.
One single value for b .

I received these using scikit SVC.

So, how can I generate the equation now?

Can I multiply those 51 y (i) alpha (i) or (dual coefficients) with 51 sv (support vectors) for each variable P and Q so that I have two coefficients for P and Q so that finally my equation appears as f(x)=sgn( mP + nQ +b) where m = sum of the (product of 51 sv of P with 51 dual coefficients) and n = sum of the (product of 51 sv of Q with 51 dual coefficients)?

I'm not sure if I've fully understood you. Radial basis kernel assumes that you transform your features into an infinite space and the dot product of your transformed vectors is exactly the radial basis kernel.

$k(x,y)=\phi(x)\cdot \phi(y)$

$\phi(x)$ - mapping

The main reason for using a kernel trick is the ability to transform features into higher dimensions without knowing the map function explicitly. Your hyperplane has infinite number of coefficients. You can always expend the radial basis kernel into Taylor series and get some of the initial coefficients.


Image dataset with timestamps? I am starting an exploratory project to determine the time (of day) of which a photo is taken based on image features. I've done some research and downloaded a few databases hoping that "creation date", "last modified", or the file names will properly reflect the time when the original photo was taken, but I have had no luck.

So far I will be resorting to all of my personal photos, which I do not think will be sufficient enough in quantity or diverse enough in quality.

Are there any natural image datasets out there with timestamps or even timestampped file names?

I found a dataset with EXIF data! It's the MIRFLICKR dataset. I was looking for more amateur-style images, but this is a great start.

I'll post if I make any more findings. Other datasets that I've checked have some images with EXIF data, but they've ranged from 1 in 100 to less than 1 in 1000 images with useful EXIF data.


Tokenizing words of length 1, what would happen if I do topic modeling? Suppose my dataset contains some very small documents (about 20 words each). And each of them may have words in at least two languages (combination of malay and english, for instance). Also there are some numbers inside each of them.

Just out of curiosity, while usually customizable, why are some tokenizers choose to ignore tokens that are just numbers by default, or anything that doesn't meet certain length? For example, the CountVectorizer in scikit-learn ignores words that do not have more than 1 alphanumeric. And the tokenizer utility in gensim ignores words with digits.

I used CountVectorizer in the end and made it to accept words containing digits and words with length 1 as well. This is because I needed exact match, as slight differences in those words of length 1 may point to a different document.

I am currently trying topic modeling (gensim's LSI) to perform topic analysis, but the main intention of doing that is to be able to reduce the dimension so I can feed it to spotify's annoy library for quick approximate searching (from 58k features, to 500 topics). Also I expect it to reduce the time and memory taken to compute classifier models.

So the question is really, if I tokenize words of length 1, would it makes sense to perform topic modeling? Even if I do a brute force search by just comparing cosine similarity (no fancy ANN), would it affect the precision or accuracy (i.e. would it be able to recognize the slight change in words of length 1 in query and retrieve the right document)?

The libraries usually exclude 1-length tokens and tokens with no alpha-numeric characters because typically they are noise and do not have any descriptive power. That is, these tokens are usually not helpful, say, in distinguishing between relevant vs not relevant documents.

However, if in your domain you feel like 1-length tokens can be helpful, feel free to use them as well. For example, if all the document that contain 1 belong to the same topic, it may be a good idea to preserve this token. 1 has descriptive power in this case: it can help distinguish between one particular topic and the rest.

Now, your next question is about LSI. For LSI there is no difference if a column in the document-term matrix corresponds to a 1-char token or to a 5-char token. So you can use LSI in your analysis.


Selecting features for malware analysis I am trying to build a classifier that detects if I have a malaware by predicting the provenance compiler. To do so I have a dataset composed of assembly code in json format :

In particular, I want to select as features the instructions, so the push, mov, jmp,..etc and create a feature vector that contains the number of times a feature appears. So, I want to apply the bag of words. To do so my code is the following:

#libraries
import numpy as np
import pandas as pd
import json as j
import re
import nltk
from nltk.tokenize import word_tokenize

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import *
from sklearn.metrics import confusion_matrix, classification_report
from sklearn import svm

#for visualizing data
import matplotlib.pyplot as plt
import seaborn as sns; sns.set(font_scale=1.2)

%matplotlib inline

json_data = None;
with open('training_dataset.jsonl') as data_file:
    lines = data_file.readlines()
    joined_lines = "[" + ",".join(lines)+"]"

    json_data = j.loads(joined_lines)

data = pd.DataFrame(json_data)
data.head()

vect = CountVectorizer()

data['instructions'] = data['instructions'].apply(lambda x: ' '.join(x))

vect.fit_transform(data['instructions'])

a =vect.vocabulary_
a

and from this I obtain a dictionary:

with more and more key, value pairs. At this point what I tried to do is to eliminate the registers so my idea to do this is to iterate the dictionary and eliminate the keys that have numbers, and so I have written the following:

def hasNumbers(inputString):
    return any(char.isdigit() for char in inputString)

for k in list(a.keys()):
    if hasNumbers(k):
        del a[k]

but actually it wasn't a really good idea because I have also registers without numbers, for example rdi.

At this point, I don't know how to move, and I am not sure I am going in the correct way. Can somebody please help me? Thank's in advance.

[EDIT 2]
Now I am trying to take the json file, not opening it with pandas, but open it as a dictionary, so:

and my intenction is try to split, as suggested, each key in such a way to eliminate the registers. So, my code for this is :

for key in json_data:
    splitted = key.split[0]

but I recieve the following error message:

AttributeError: 'dict' object has no attribute 'split'

I am getting really confused about how to operate now.

[EDIT 3] or I tried to do the following:

for v in json_data.values():
    splitted = v.split[0]

but it gives me the following error message:

AttributeError: 'list' object has no attribute 'values'

a problem I think is the fact that the various push r12,.. etc are in a list, so I don't know how to do.

[EDIT 4] I actually tried to solve the above problem this way:

for v in json_data[0].values():
    splitted = v.split[0]

but again I get the error message:

AttributeError: 'list' object has no attribute 'split'

[EDIT 5] Now, I am trying to do this using the file opened with pandas, so:

data['features'] = [i.split[0] for i in data['instructions']]

but still I get the following error:

AttributeError: 'list' object has no attribute 'split'

or I tried also:

feat = [i.split[0] for i in json_data[0].values()]

but get the same error as above.

In particular, I want to select as features the instructions, so the push, mov, jmp,..etc and create a feature vector that contains the number of times a feature appears.

Instead of reading each instruction entirely for a sample program, you can read the first word of each instruction to get the opcode and then create a feature vector out of these opcodes.

Suppose,

sample_program[i] =  ['push r12', 'push rbp', 'xor edx edx', ...]

i.e. a list of instructions, then define opcodes[i] as

opcodes[i] = [instruction.split[0] for instruction in sample_program[i]]
#['push', 'push', 'xor']

Now you can create feature vectors out of collections of such opcodes.


What if irrelevant features impact the outputs? We know that weather conditions in China is not very useful to predict price of a house in Spain (as life experience). However, when we drop that China's weather condition, the accuracy is reduced largely. Will we keep it?

It would mean that the model trained with some irrelevant features is overfit: a good model uses only features which have some predictive power on the target variable, so if the sample is representative enough any irrelevant feature is ignored or assigned very little weight. However if the training dataset is too small or the model too complex then it uses details which happen by chance, so it could use some irrelevant features: this is overfitting.
In case the performance on the test set was high despite the model being overfit, it's likely that there is some data leakage additionally.


find most dense neighborhood of points in high dimensional space I'm working on a project where I have many high-dimensional points and I want to find the most dense neighborhood of them. Ideally, out of my ~500 points that are each a 4x300 matrix (300 ms time series of four variables) I want to find the ~30 points that are the most similar. I've looked into k nearest neighbor methods but those are all finding the smallest neighborhoods of a certain point, I want to be able to specify the size N of the neighborhood (how many data points) and get the subset N points that are the closest together.

I think(?) that this is a clustering problem so I've looked at soft subspace clustering which clusters using the more relevant dimensions and also I've looked into biclustering which was developed specifically for time series data.

Any help would be greatly appreciated! I have a problem that I'm sure someone else has thought of but I don't think I know exactly how to word it and everything I've looked at so far has been close but not exactly what I need. So any papers/code/tutorials/etc that give me any information on how I might solve this would be fantastic :D

Update: I've looked into using Frechet distance to use as a distance between time series. Would that be an appropriate metric to use for this type of data?

Look into anomaly detection algorithms. Scikit-learn has a variety of built-in ones in its outlier detection module. Finding 30 most similar points among 500 is the same as finding 470 outliers. Most of these algorithms have a parameter contamination which specifies what fraction of your dataset you generally expect to classify as outliers, so manipulating the value of this parameter can possibly give you what you want to achieve.


Ranking Bias in Learning to Rank Users tend to click on results ranked highly by search engines much more often than those ranked lower. How do you train a search engine using click data / search logs without this bias? I.e. you don't want to teach the search engine that the results that are currently ranked highly should necessarily continue to be ranked highly just because they were frequently clicked.

Have you seen this paper?
Optimizing Search Engines using Clickthrough Data

I stumbled upon this the other day, and I'm still reading through it, but the author attempts to deal with the problem you describe. You may also find Improving Web Search Ranking by Incorporating User Behavior Information useful.


What is the difference between Concatenate() and concatenate() layers in Keras / TensorFlow 2.0? I am learning TensorFlow 2.0, whose layer functions are based on Keras. What is the difference between the Concatenate() and concatenate() layers?

Keras provides 2 kinds of API i.e. Sequential and Functional. And because of this 2 kind of API, the difference is there in them.

Concatenate is used when you are using Sequential API
concatenate is used when you are using Functional API

Just look into the following documentation:
https://keras.io/layers/merge/


Visualising a stream of emotions I have a stream of emotions (from some audio recordings) extracted by a speech emotion recogniser.  My questions now are how to best display these emotions to the end users?

What is the best way to visualise this stream of emotions?
Is there any tool/library/package/software that I can use a stream of ekman emotions (e.g. happy->surprise->sad->...) as input and get some visualisation as output?

Thanks !!

I would recommend using an Alluvial Diagram (Sankey Diagram). You can actually show multiple stages between emotions. So while if you google Sankey you will likely mainly see one two stage demonstrations...that is from A to B, you can easily code it to show from A to B to C, etc.


How to handle “not label Y” in a multi class machine learning problem? I have a train data set that comprises information in the form:

  feature 1, ..., feature N, label
1 x1,        ..., xn,        A
2 x1,        ..., xn,        B
3 x1,        ..., xn,        C
  ...
4 x1,        ..., xn,        not A
5 x1,        ..., xn,        not B
6 x1,        ..., xn,        D
7 x1,        ..., xn,        E
  ...

In the test dataset, the labels A, B, C, D, E should be predicted.

I would like to use the information in row 4 and 5 in order to avoid overfitting.

Which techniques / tools can I use to utilize the "not A" and "not B" information in rows like 4 and 5?

I think you should start by looking into Multi-label Classification, as your problem seems to be a subset of MLC, where one example can have multiple correct labels. If one class is "Pictures with people in them" and another is "Pictures with guitars in them", one picture could certainly belong to both - this is an example of multi-label classification.

In your case, you could formulate your problem by considering that most examples have only one appropriate label, but 'Not A' has four: 'B', 'C', 'D', and 'E'. (If you have more specific information about the possible distributions of the correct label, you could use that information here too)

There are several methods for MLC, not all of which I'm familiar with, but I know plenty of research has been done on this using the CelebA image dataset, for example (it comes along with a number of potential label vectors).

Good Luck, and welcome to the site!


LSTM -RNN : How to get continuous range output instead of categorical? I am trying to solve a problem predicting a value between a range for a sentence:

The dataset looks like this:

Index_no               text_sentence                             value

01                     yes I like riding a bike.
                       I was 4 when I learned                   4.2311
                       to ride a bike. the colors of my
                       bike is yellow and black.

02                     i like riding my bike, i learnt
                       riding a bike when i was 8 or            -2.11
                       9 years old ,my bike is sparkling
                       pink with white marks

Range of values is -7 to 7. Now, I am thinking about using a LSTM for text, but I am confused about the continuous output.

I was thinking about two methods:

Converting (normalizing the data between 0 and 1) and then after getting the output from network, denormalize the data, will this work?

Second approach, using a custom activation function?

Or how can I get output between a range?

Using Sigmoid Activation function (standard, no need for customization), you will get outputs in the range [0,1]. If you pre-scale your labels accordingly to fit in [0,1], the network will output values which you can manually re-scale to match the original amplitude. Hope it helps :)


Data binning - Why we need to transform Categorical Variables? Having a lot of categorical features and other numerics why we need to transform the categorical to binary values? Is it for using the values in mathematics functions of the algorithms?

Thanks!

Yes. If you need the information of your categorical variables then you need to represent them by numbers as most of machine learning algorithms are math-based.


Is it ok to trust regression predictions when none of the coefficients are statistically significant? Background to the problem: I am estimating individual treatment effects using double machine learning model. I do not know true treatment effects for my problem.

Double ML: Given Y (outcome), T (treatment) and X ( features)

Y = aT + bX + error

coefficient a is of interest (measures treatment effect) .

Double ML procedure:

Fit Y ~ X  => Compute residuals (Y* = Y – Y’) – Residuals are
treatment effects to be estimated
Fit T ~ X => Compute residuals (T* = T- T’) – This model captures
variation in T explained by X
Fitting a model (Y* ~ T* ) on residuals will give the average
treatment effects

I am fitting a linear regression model (Y* ~ T* ) and none of the coefficients are statistically significant. Instead of relying on point estimates, I am computing prediction confidence intervals and p-value to check if the predicted value is statistically significant or not.

Is this approach good?

It depends on the goal of the project.
Statistical significance is more important for publishing in academic journal. For applied projects, statistical significance is less important.
Even if a model is not statistical significant, it can have value. The model may summarize the data and identify relationships. The model maybe practically important, the effect maybe of value to practitioners.


Ordering of standardization, pca, and/or tfidf for neural network I have 60k rows of text data. I have tokenized it into 55k columns. I am using a neural network to classify the data but have some questions about how to order my preprocessing steps. I have too much data for my hardware (doesn't fit in memory/too slow) so I am using PCA to reduce dimensions.

Obviously, I need to scale before PCA. I am currently standardizing the columns, but I am wondering if I can use tfidf instead of standardization. Some rows have 50k+ tokens while others have <1k tokens so I am worried these rows have undue influence on the outcome of scaling which will trickle down the pipeline. Is this a good/bad idea? Would I maybe use tfidf then standardize before PCA?

Generally neural nets prefer standardized data. After PCA the first few columns have much greater magnitude than the rest b/c they capture so much variance. Should I standardize after PCA and before training? The reason for standardizing before training is so no feature has bigger influence on the model just b/c the scale is bigger, but isn't PCA telling me that the first few features are actually more important? FWIW, I've tried both and not scaling seems a little better.

What about performing tfidf after PCA and before training? Again, rows with 50k+ tokens will prefer a network with orders of magnitude larger weights than rows with <1k tokens. Wouldn't it be hard for the network to set weights for both types of rows?

Diagram for clarity: data -> tokenize -> ?standardize/tfidf? -> PCA -> ?standardize/tfidf? -> neural net

I would go for this:
data -> tokenize -> tfidf* -> neural net

But in tfidf vectorizer, you could actually regularize the number of terms used, say for example restricting the minimum number of occurrences of a term and/or defining the max_number of features so that you only keep the ones that have the highest importance according to Tfidf.
If you want to reduce the number of features via some decomposition technique PCA won't be adequate since the term-frequency matrix is sparse, so you could, for example, using NMF (Non-negative matrix factorization instead)
So:
data -> tokenize -> tfidf->NMF -> neural net

This time the regularization on tfidf is not necessary since you have an additional step.
In the end, metrics on CV will guide you on what the best strategy is


CNN training data size for determing the winner of tic-tac-toe I'm trying to learn machine learning with tensorflow and wrote a program that uses CNNs to determine game results for a given tic-tac-toe board. Its inputs and outputs are -

Input  - An array of 9 elements that represents the board (0=empty, 1='X', 2='O')

Output - An array of 3 elements ([1,0,0]=draw, [0,1,0]=player1 won, [0,0,1]=player2 won).

Below is the tensorflow graph part of the program (it is a modified version of tensorflow tutorial at - https://www.tensorflow.org/get_started/mnist/pros).

baseFeatureSize = 64
x = tf.placeholder(tf.float32, shape=[None, 9])
x_image = tf.reshape(x, [-1, 3, 3, 1])
W_conv1 = nn_utils.weight_variable([2, 2, 1, baseFeatureSize])
b_conv1 = nn_utils.bias_variable([baseFeatureSize])
h_conv1 = tf.nn.relu(nn_utils.conv2d(x_image, W_conv1) + b_conv1)
W_conv2 = nn_utils.weight_variable([2, 2, baseFeatureSize, baseFeatureSize * 2])
b_conv2 = nn_utils.bias_variable([baseFeatureSize * 2])
h_conv2 = tf.nn.relu(nn_utils.conv2d(h_conv1, W_conv2) + b_conv2)
W_fc1 = nn_utils.weight_variable([3 * 3 * baseFeatureSize * 2, baseFeatureSize * 4])
b_fc1 = nn_utils.bias_variable([baseFeatureSize * 4])
h_pool2_flat = tf.reshape(h_conv2, [-1, 3 * 3 * baseFeatureSize * 2])
h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)
keep_prob = tf.placeholder(tf.float32)
h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)
W_fc2 = nn_utils.weight_variable([baseFeatureSize * 4, 3])
b_fc2 = nn_utils.bias_variable([3])
y_ = tf.placeholder(tf.float32, shape=[None, 3])
y_conv = tf.matmul(h_fc1_drop, W_fc2) + b_fc2

cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_conv))
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y_conv,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

def conv2d(x, W): return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')
def max_pool_2x2(x): return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')
def weight_variable(shape):
    initial = tf.truncated_normal(shape, stddev=0.1)
    return tf.Variable(initial)
def bias_variable(shape):
    initial = tf.constant(0.1, shape=shape)
    return tf.Variable(initial)

Questions - The above model works well but requires to train with over 700k inputs to get output prediction accuracy up to 80%. But the total possible permutations of the game board are less than 300K. If I feed just all of the unique permutations, then the accuracy worsens.

Is it normal for CNNs to require training data size to be much larger than all possible permutations of the inputs? I'm assuming this is not true or else they'd not be able to play chess and go etc, so what am I doing wrong?
Based on your experience, how big a training sample size would typically make sense for this setup?

I experimented with some other approaches as well and discovered that MLPs produce equivalent or better results for this problem. Thanks to the commenters Neil Slater and Student T for pointing me in the right direction.

For anyone interested, this is the model that eventually worked for me

boardSize = 3*3
featureSize = #I experimented with 500-1500 with varying results here
output = 3
x = tf.placeholder(tf.float32, [None, boardSize])
y_ = tf.placeholder(tf.float32, [None, featureSize])
learning_rate = 0.01
seed = 128
weights = {
    'hidden': tf.Variable(tf.random_normal([boardSize, featureSize], seed=seed)),
    'output': tf.Variable(tf.random_normal([featureSize, output], seed=seed))
}
biases = {
    'hidden': tf.Variable(tf.random_normal([featureSize], seed=seed)),
    'output': tf.Variable(tf.random_normal([output], seed=seed))
}
keep_prob = tf.placeholder(tf.float32)
hidden_layer = tf.add(tf.matmul(x, weights['hidden']), biases['hidden'])
hidden_layer = tf.nn.relu(hidden_layer)
hidden_layer2 = tf.add(tf.matmul(hidden_layer, tf.Variable(tf.random_normal([featureSize, featureSize], seed=seed))),
                tf.Variable(tf.random_normal([hidden_num_units], seed=seed)))
hidden_layer2 = tf.nn.relu(hidden_layer2)

y_conv = tf.matmul(hidden_layer2, weights['output']) + biases['output']
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=y_conv, labels=y_))
train_step = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y_conv,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))


How do I label images faster I have around 1600 images extracted from videos shot at night time. I am labeling each image and trying to be as accurate as I can in assigning bounding boxes. I am labeling vehicles and traffic light/traffic signs. This is very time-consuming, I am wondering if someone have experience or done this before and can advise me on some automated methods of labeling night time images. The objects of interests are usually appear to be quite small. An example labelled image is given below

On the internet, LabelImg is pretty popular. It's a Python program that you can use to automatize drawing of bounding boxes. Alternatively there is HyperLabel, on their website they say it's free so it.s worth to give it a try.

The options available are so many I completely get lost while searching. I suggest you to read this nice review of tools first.

With these tools, labeling 1600 images is not going to be a problem IMHO. Good luck.


Is it possible to train a neural network to solve polynomial equations? I randomly generate millions groups of triplet $\lbrace x_0, x_1, x_2 \rbrace$ within range $(0,1)$, then calculate the corresponding coefficients of the polynomial $(x-x_0)(x-x_1)(x-x_2)$, which result in triplet groups normalized in a form of $\lbrace { {x_0+x_1+x_2 \over 3} , {\sqrt{x_0x_1+x_1x_2+x_0x_2 \over 3}} , {\sqrt[3]{x_0x_1x_2}}} \rbrace$;
After that, I feed the coefficient triplets in to a 5-layered neural network $\lbrace 3,4,5,4,3 \rbrace$, in which all the activation function is set to sigmoid and the learning rate is set to 0.1;
However, I only get a very poor cross validation, around 20%.

How can I fix this?

BackGround

My original problem is a dynamic inverse problem.
In that problem, I have hundreds of thousands of observations $O$, from these observations, I need to recover several hundred parameters $P$.
The simulation process from $P$ to $O$ is very easy and cheap to calculate, but the inversion from $O$ to $P$ is highly nonlinear and nearly impossible. My idea is to train a neural network getting $O$ as inputs and $P$ as outputs. To check the feasibility of this idea, I employ a 3-ordered polynomial equation to do the validation.

update half a year later

With more nodes per layer, I have successfully trained a neural network. The topology is set to $\lbrace 3, 64, 64, 64 \rbrace$. And the most important trick is, sorting the generated triplet $\lbrace x_0, x_1, x_2 \rbrace$, ensuring $x_0 <= x_1 <= x_2$ always holds.

You're trying to fit a very complicated function.  There is no reason to expect that neural networks will be very good at this.  Neural networks aren't magic pixie dust.  They can do some things well, but don't expect a silver bullet.

You're trying to get the neural network to learn to compute some complicated function.  We do know that given sufficiently nodes and sufficiently many layers, you can get an arbitrarily good approximation to this function, but there's no a priori reason to expect it should be possible with the particular (small) number of nodes and layers you chose.  In fact, two layers suffices, if you have sufficiently many nodes -- but we have no way to compute how many nodes are needed.

In particular, the function you are trying to compute amounts to computing $x_0,x_1,x_2$ from $a,b,c,d$, where

$$\begin{align*}
x_k &= - {1 \over 3a} \left(b + \eta^k C + {\Delta_0 \over \eta^k C}\right)\\
\Delta_0 &= b^2 - 3ac\\
C &= \sqrt[3]{\Delta_1 \pm \sqrt{\Delta_1^2 - 4\Delta_0^3} \over 2}\\
\Delta_1 &= 2b^3 - 9abc + 27a^2d\\
\eta &= -{1 \over 2} + {1 \over 2} \sqrt{3} i
\end{align*}$$

(This comes from the general formula for solving a cubic equation.)

That's an extremely messy function, and thus it might be challenging to approximate using a neural network of the architecture you present.  You can always try increasing the number of nodes and/or number of layers, but there's no a priori theory to tell you what the right number is.


When, if ever, is nearest neighbor classification the best choice? I've been researching the history and use of k-nearest neighbor classification and regression, and various tweaks including k-d trees and LAESA.

I understand that it is useful because it is simple and flexible, but can be computationally expensive and requires a lot of data storage.

But here's what I don't know:

  Is there any class of problems for which nearest neighbor classification is the best or one of the best algorithms to use?

By 'class of problems' I mean either a class based on data structure (for instance, maybe KNN is great for low-dimensional data with a mix of nominal and numerical data), or a class of real-life problems (maybe KNN is useful in predicting diseases for insurance holders).

One situation where NN would be ideal is if the data are sample points of a piecewise constant function. In this case, the true function is composed of a tesselation of its domain, with a constant value for all points within each division.


Monte Carlo Dropout as Uncertainty predection I am pretty new to Python and this board so I am not sure, if I am at the right place for my question since it doesn't include any code. If not so, please give my a hint for a better way/place to ask.

I am struggling with using Monte Carlo Dropouts for determine Uncertainty for my image classificator using ResNet18.

I have read several papers to this topic and I am still kinda confused about this topic.

I know already to use dropout like

def dropout2d(input, p=0.5, training=True, inplace=False):

multiply times for getting a variance which can be interpreted as the uncertainty. I am pretty sure, that I already understood how MC-Dropout works in general.

So coming to my questions:

Do I use MC-dropout while training or testing and why? I feel like I have read different ways.

I have searched for a coding example from the dropouts but didn't find anything. Do you know Paper/ example code/ searchlink for a proper code?

Is

nn.dropout2d(imput, p=x, ...) the right way to use dropout for a image classificator?

Thanks for your help.

Monte Carlo Dropouts (MCDO) is used during the prediction / inference phase to provide an estimate of uncertainty for the model's predictions.
Regular dropout during the training phase is a regularization technique.


Inductive Bias in Gaussian process All supervised learning techniques have some kind of an inductive bias. What is the inductive bias in Gaussian process models ?

The inductive bias of a Gaussian process (GP) is encoded in the covariance kernel. A GP is a distribution over functions — when we choose a kernel, we are specifying characteristics that we expect the solution function to have, e.g., smooth, linear, periodic, etc.

For example, a common covariance kernel is the squared exponential:

$$k_{SE}(x, x') = \exp \left( - \frac{||x - x'||_2^2}{2l^2} \right)$$

where $l$ is the characteristic length-scale. This kernel specifies that an unobserved point will be similar to each observed point as an exponentially decaying function of the Euclidean distance between the two points.

More detailed explanations can be found in section 2.4 of Andrew Gordon Wilson's thesis or in the book Gaussian Processes for Machine Learning.


Linear Regression of Sine wave using Gradient descent Not working I am writing an algorithm to fit a sine wave. I want to have 4 parameters ( amplitude, frequency, phase & centre position). When I tried to program with all 4 parameters I wasn't able to find a good fit. So I tried to first make it work with each parameter separately and making the other 3 constant hardcoded values. I managed to make this work for all parameters except for the frequency. So I think the problem is the derivative of the error function w.r.t the frequency parameter. As seen below all the other parameters of the sine are hardcoded values except the frequency parameter

%This is iterated for a large number of times
for i = 1:length(x)
%computing hypothesis and error
    TrialY(i) = 4* sin(w2(end) *x(i) + 4) -2;
    Error(end) = Error(end) + (TrialY(i) - y(i))^2;
end
Error(end) = Error(end) / (2*length(x));

dw2 = 0; %derivative value

%derivative of error function with respect to frequency (w2)
for i = 1: length(x)
    dw2 = dw2 + (4* sin(w2(end) *x(i) + 4) -2 - y(i)) * 4 * x(i) * cos(w2(end) * x(i) + 4);
%hardcoded derivative
end
dw2 = 2 * dw2 / length(x); %normalising error

w2(end + 1) = 0;

w2(end) = w2(end - 1) - alpha * dw2; % alpha is the update constant

It seems like the parameter is not converging on the right value with whatever alpha and initial values I give it. While the plot of the Error with time is decreasing but converging on a value > 0.

The problem seems to be that in the case of frequency parameter, the error function is not convex, and has multiple local minima of value greater than 0. The gradient descent is converging to one of these minima. The error will reduce to 0 only if the starting estimate lies in the right convex basin.

For the amplitude and center value parameters, the error functions are convex, and for phase parameter, there are multiple local minima all of value 0.


How can one quickly look up people from a large database? Vocabulary

Face detection: Finding all faces in an image.
Face representation: The simplest way to represent a face is as an image (pixels / color values). This is not very space efficient and likely makes follow-up tasks hard. Face embeddings are one other representation. In this case a face is a point on the unit-sphere in $\mathbb{R}^{128}$, IIRC.
Face verification: Given two face representations, deciding if they are the same

Question

I was just wondering how identifying a person with many potential people can work. So finding a face in an image works quite well and fast enough for most applications. Face verification as well. But I'm not sure how to scale this if you don't compare 1 face against 1 other face, but 1 against millions / billions of faces.

Suppose you have a lot of examples of faces with the identity of the person. Think of Facebook, where many people tagged friends. Or of countries with biometric passports.

In the real applications, the face verification task is easy because you can just brute-force compare against all candidates:

Facebook: Only candidates are your friends, so ~200 candidates.
Airport EU fast entry: Your face is compared against the passport. So only one candidate.

But then think about some dystopian books / movies, where cameras identify anybody. While tracking helps to reduce that problem, finding a match from millions / billions of examples is computationally super heavy. Assuming a single face verification takes ~200ms, for a million candidates it would already take 60h. For a billion users it would already be 6 years. For all people on earth who currently live it is 48 years.

So with that many candidates, you don't want to compare against all candidates.

When you use the face-embedding, it becomes a nearest neighbor search in $\mathbb{R}^{128}$.

Calculating the euclidean distance of two vectors in $\mathbb{R}^{128}$ takes roughly 15μs (see "timing" below). This means a single check over $7.5 \cdot 10^9$ people would take 31h. Way better, but still pretty long.

While the face-embedding approach pre-computes a good face representation, going over all examples is still a pretty dumb approach. If it was only $\mathbb{R}^1$, one could make a simple binary tree. For few dimensions, I think something like a k-d-tree might work. But what about 128 dimensions?

Is there another approach to get the person quicker?

Timing

import numpy as np
durations = timeit.repeat('np.linalg.norm(a-b)',
                          setup='import numpy as np;a=np.random.random(128);b=np.random.random(128)',
                          repeat=1000,
                          number=3)
print('min: {min:5.1f}μs, mean: {mean:5.1f}μs, max: {max:6.1f}μs'
      .format(min=min(durations) * 10**6,
              mean=np.mean(durations) * 10**6,
              max=max(durations) * 10**6,
              ))

That is problem is call identification, mapping a percept to a specific entity.
One common option is hashing, take a percept and map it to a specific, unique integer. If two different percepts map to the same integer, they are the same entity. If two different percepts do not map to the same integer, they are different entities. Hashing takes constant time to look-up an entity no matter how many entities.
In the case of facial recognition, the hashing function is best learned through Deep Learning.


Analysis of Alternating Decision Tree on Weka I am applying the AD Tree algorithm & this is the tree visualization of the output:

I can't understand the values in the decision nodes (-0.4,0.541,-0.882...), How are these calculated? & how did we calculate the root node's score?

Are predicate conditions (<127.5..) formed by entropy splitting mechanism?

This is an image of the output:

Any help is appreciated, cannot find any AD Tree output analysis document!!

As per the original paper here on Alternative Tree it says,

"We give a different representation of the same classification rule....
The decision node is identical to what we had before , while the prediction node is associated with a "real valued number"......
The classification that is associated with the path is NOT the label of the leaf, instead that is the sign of the sum of the predictions along the path.for example, the classification of the instance a=b=0.5 is sign(0.5-0.7-0.2)=sign(-0.4)=-1..."

Now there is optimization of it too here which is available here and is implemented by WEKA and JBoost . And of course there is always -Wiki