Matlab feature selection I am trying to learn relevant features in a 300*299 training matrix by taking a random row from it as my test data and applying sequentialfs on it. I have used the following code:

>> Md1=fitcdiscr(xtrain,ytrain);
>> func = @(xtrain, ytrain, xtest, ytest) sum(ytest ~= predict(Md1,xtest));
>> learnt = sequentialfs(func,xtrain,ytrain)

xtrain and ytrain are 299*299 and 299*1 respectively. Predict will give me the predicted label for xtest(which is some random row from original xtrain).

However, when I run my code, I get the following error:

Error using crossval>evalFun (line 480)
The function '@(xtrain,ytrain,xtest,ytest)sum(ytest~=predict(Md1,xtest))' generated the following error:
X must have 299 columns.

Error in crossval>getFuncVal (line 497)
funResult = evalFun(funorStr,arg(:));

Error in crossval (line 343)
    funResult = getFuncVal(1, nData, cvp, data, funorStr, []);

Error in sequentialfs>callfun (line 485)
    funResult = crossval(fun,x,other_data{:},...

Error in sequentialfs (line 353)
                crit(k) = callfun(fun,x,other_data,cv,mcreps,ParOptions);

Error in new (line 13)
    learnt = sequentialfs(func,xtrain,ytrain)

Where did I go wrong?

func = @(xtrain, ytrain, xtest, ytest) sum(ytest ~= predict(Md1,xtest));

The function "func" is called by sequentialfs, which does not pass "Md1" to the "predict" function. Therefore, Md1 is unknown to that "cost" function. You should include "Md1=fitcdiscr(xtrain,ytrain);" within your "func" function.

Beside, "sequentialfs" does a 10-fold CV. Why are you trying to pass only 299 samples instead of all 300?


How to display an Heat Map (using Tableau or other BI tool)? I've a simple dataset:

Id
Longitude
Latitude
Value

What can I obtain at the moment are "one map point" in every place.. Sum of Value don't affect "density" color.
I would like to display a Geographic Heat Map like this (i copied from Google image). How ?

Set the latitude and put it under rows, select longitude and put it under cols.
Select value and drop it to the center of the sheet.
And under the marks dropdown select filled maps.

Make sure you see filled maps in the dropdown and that the value column is selected for color.

Edit: Do this for gradual color in heatmap-
click on color tab.
Select edit Color-now choose stepped color. This should do the work of filled maps with gradual colors.

Hope it was helpful.


Installing skmca python package I can't figure out how to install the python package named skmca ( MCA for sickit-learn ) : https://github.com/TomAugspurger/skmca

There is no pip or conda command available.

Thanks for your help.

Check their reference to the following page for PIP install:

https://github.com/MaxHalford/Prince


Does Sampling size matters in Multi classification Model I am working on a multi class classification model where few of the class are with less data compare to other classes.
I used random sampling technique to create a sample from the population keeping the proportion of each class equal to that of population.
For example, class A has 400 records in the population and class B has 100 records in the population then when doing random sampling I am creating a sample where records of class A and class B are in proportion of 4:1.
The trend I have observed is by changing the sample size (keeping inter class proportion constant) of one class leads to change in model performance (accuracy,precision,recall).

What technique do i need to apply in order to make my model stable irrespective of sample size?

It is always better to keep sample sizes close each other. The problem you are facing is Imbalanced Classification.
There are lots of methods you can apply such as upsampling/downsampling, synthetic data generation (check SMOTE).

Model:

I would first convert the model to binary classification such that:

model 1 predicts: A or not A
model2 predicts B or not B in "not A" group

Note: Another option three x or not x classifier for three classes)

Then I would apply those sampling techniques (see above) to models. If you have high number of observations try to use downsampling to make class weights as 50:50, if you cant do that try SMOTE to generate data from minority class.

Metrics:

Note that if you don't have balanced dataset, try not to use accuracy as a performance metric. Consider area under ROC, F1 and precision/recall according to your case.

Hope it helps!


Activation function vs If else statement The question is very naive and most of us may know the answer. I have googled it but was not able to find a satisfactory answer so posting it here. Can someone please put the right words on this question.

Activation functions like ReLU, Sigmoid etc generally work like if else statements. e.g. if value of input is less than 0 then 0 else same value ( max(0,a)  in case of ReLU)  etc. So the question is why if else statements are not being used? Are these functions less compute intensive or work better in case of multidimensional data?

The argument in favor of if else statements is, the programming of neural networks is also being done in high level languages like python, C etc. In that case what compels the use of these activation functions rather than programmatic oriented If Else syntax?

Activation functions in general cannot be implemented with if-else statements.

ReLU is a particularly simple activation function, and can thus easily be implemented with an if-else block. For example (in Python):

def relu(x):
    return x if x > 0 else 0

But this only works because ReLU is linear for $ x \geq 0$.

What about sigmoidal activation functions?

There's no if/else implementation for this function that's analogous to that of ReLU.  In fact, this is a one-liner with no if/else involved:

def approximate_logistic(x):
    return 1 / (1 + 2.71828**-x)

Same goes for lots of other activation functions like Softplus, CELU, Tanh.

If/else blocks simply aren't powerful enough to capture the wide variety of activation functions that we might want to use.


Heat maps in R with more than 2 categorical variables I have a dataset which looks like-

Variable value year Quarter Location
A 48.235 2011 Q1 North
B 65.444 2011 Q2 North
C 77.453 2011 Q3 North
D 44.678 2011 Q4 North
A 88.542 2012 Q1 South
B 66.566 2012 Q2 South
C 55.443 2012 Q3 South
D 78.990 2012 Q4 South

Can anybody help me with a code to generate heatmap with more than 2 categorical variables? I need 'variable' and 'Location' to be on y axis and 'Quarter'&' year' to be on x axis. The shades of the data inside heat map should correspond to 'value'.

Here is one way of visualizing the data you have presented. However, I have used some liberties and assumptions to create this plot.

First, create a year_quarter variable that simply concatenates the year and quarter to present the time on the X axis. The following piece of code can do this in R

year_quarter = paste(dat$year, dat$Quarter, sep="-")

Now, the dataset you have will look like:

> dat

Variable  value year Quarter Location year_quarter
1        A 48.235 2011      Q1    North      2011-Q1
2        B 65.444 2011      Q2    North      2011-Q2
3        C 77.453 2011      Q3    North      2011-Q3
4        D 44.678 2011      Q4    North      2011-Q4
5        A 88.542 2012      Q1    South      2012-Q1
6        B 66.566 2012      Q2    South      2012-Q2
7        C 55.443 2012      Q3    South      2012-Q3
8        D 78.990 2012      Q4    South      2012-Q4

Finally, using ggplto2, you can create the plot such that the colour represents the value, the shape represents the Location and the size represents the Variable.

This simple one liner can help you produce such a plot:

p = ggplot(dat, aes(x = year_quarter, y = value, colour = value)) + geom_point(aes(shape = Location,size = Variable))

This is how the output plot looks like:

Note that you can also add geom_line with interaction if you would like the lines to be connected based on Location and Variable.


Stratified Sampling Variable Choice I am trying to do stratified sampling in R to sample from my data and one of the parameters is group, which takes variable names to sample from keeping same initial distribution of the data set.

Is it common to include the response as a grouping variable or not and what are the implications of including/excluding it?

Yes, it is very common and sometimes necessary to use the target variable for stratified sampling.

Consider the case of fraud detection, given a bunch of features about a person (e.g. income, gender, position etc) we want to predict the likelihood of that person has committed the crime (a boolean value indicating whether the person is a suspect). This dataset is likely to be very asymmetric with very few positive examples.

Now if we want to use k-fold cross-validation, we must stratify the samples using the target variable. If we don't, we might end up with a fold without any positive example at all and no metrics can be calculated from that fold.


How to extract numerical data from a matplotlib scatter plot in python? I have a scatter plot with about 19,000 data points. By visual inspection, I noticed some points for which I want to look at the corresponding numerical data from the data frame (basically a subset of the original data whose scatter plot we are looking at).

Is there a way to isolate the data corresponding to the point(s) [when the coordinates of the data points are not apparent from the figure] ?

Below is an example. How do I get data corresponding to the circled point on the plot?

Any response would be of great help.

Thanks.

You could also show the values for each point by using matplotlibs annotate command:

import numpy as np
import matplotlib.pyplot as plt

x = np.arange(5)
y = np.random.randint(10, size=5)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.set_ylim(0,10)
plt.plot(x,y)
for i,j in zip(x,y):
    # xytext and textcoords are used to offset the labels
    ax.annotate("({},{})".format(i, j), xy=(i, j), xytext=(5, 5), textcoords='offset')
plt.show()


Forget Gate in Long Short-Term Memory What value does actually LSTM forget in a training phase?
for example, I do have a surface temperature data for 10 years. Then I made them as a training data for building my neural network using lSTM algorithm. Then what value or which time step in my data that LSTM forget?

The way LSTMs are capable of learning long term dependencies is by keeping a cell state which serves as a memory of sorts.
This cell state is updated based on the values of different gates within the cell, forget gate being one of them.
The forget gate looks at the previous hidden state and the current input and outputs a number between 0 and 1 for each value in the cell state - deciding which values should be discarded and which saved.
During training, the cell learns which values in the cell state it should keep at different time steps in order to yield better results for the task at hand.
Now, saying exactly what are those values are is hard since it changes between samples and time steps. For example, let's assume that your objective is to predict the temperature for tomorrow based on the previous 10 days. If you had an outlier on the 3rd day of the previous 10 days, that might hinder the prediction, and the cell could decide to forget that value. In other cases, it might decide to keep the value of the 3rd day since it helps prediction.
If you want to understand LSTMs better I suggest this great post by Chris Olah.


Matching new set of data with pre-defined sets I have sets of data describing sets of levels of requirements needed for certain sets of tasks. The following is a tabulated example:

Note that the data values are on a scale from 0 to 10.

My problem here is that I have a set of employees whose skills (analysis, patience, comprehension ...) have been analyzed, like the following employee:

Analysis --> 8.5
Patience --> 5
Comprehension --> 7
Communication --> 7.5
Creativity --> 8

How to match this employee to the best task according to his skillset and the required ones for each task and also find a matching percentage.

Please note that the number of tasks is much bigger (around 1000 tasks) with more requirements that have been found statistically.

You don't need Machine Learning for doing this.

You can subtract the vector describing the user to the vector of the task, and calculate the magnitude of those vectors.

The vector with the smallest magnitude will come from the most matching task.


Timestamps in Ridge Regression Scikit Learn I am trying to transform data for use in regression, most likely the Ridge or Lasso technique implemented in sklearn.linear_model.

My training data contains time stamps , which I believe may have predictive power. The time stamps reflect the time that a user placed an order for pizza. Here is an example:

Edit: Including labels in field elapsed_time, which is in seconds.

import pandas as pd
import sklearn.linear_model as linear_model

delivery_data = {
    'order_time' : ['2018-09-12 21:43:08', '2018-09-13 06:33:04', '2018-09-13 09:12:18'],
    'price' : [34.54, 8.63, 21.24],
    'miles' : [6, 3, 7],
    'home_type' : ['apartment', 'house', 'apartment'],
    'elapsed_time' : [2023, 1610, 1918]
}

df = pd.DataFrame(delivery_data)
df['order_time'] = pd.to_datetime(df['order_time'])

The resulting DataFrame looks like this:

           order_time  price  miles  home_type  elapsed_time
0 2018-09-12 21:43:08  34.54      6  apartment          2023
1 2018-09-13 06:33:04   8.63      3      house          1610
2 2018-09-13 09:12:18  21.24      7  apartment          1918

I am trying to predict the time to deliver pizza (elapsed_time) given timestamp, quantitative, and categorical data.

I suspect that time of day is predictive but that date is less predictive.

So far, I am considering extracting only the hour from the time stamp. In this example, order_time would become [21, 6, 9]. My first concern is that 23:59 has an hour of 23 and 00:01 has an hour of 0. The two values are far apart, even though the order times are two minutes apart.

Is there a better way to transform this datetime data?

Does it make a difference that the dataset contains other quantitative data (price, miles_from_store) and categorical data (home_type)?

The datetime type has arithmetic operations available. If you have two datetime types, you can find the delta between them - the result will be a datetime.timedelta class. The other quantitative data is easily incorporated into a linear regression model, either lasso or ridge. Pretty much all those scikit-learn models can use a vector of features.


How to pass features extracted using CNN into RNN? I have word images as below:

Let's say it's a 256x64 image. My aim is to extract the text from the image as 73791096754314441539 which is basically what an OCR does.
I am trying to build model which can recognise word from images.
When I am saying word it can be any of the follwing:

Any dictionary word, non-dictionary word
a-z,A-Z, special characters including spaces

I have built a model (snippet because of company policies) in tensorflow as below:

inputs = tf.placeholder(tf.float32, [common.BATCH_SIZE, common.OUTPUT_SHAPE[1], common.OUTPUT_SHAPE[0], 1])
# Here we use sparse_placeholder that will generate a
# SparseTensor required by ctc_loss op.
targets = tf.sparse_placeholder(tf.int32)

# 1d array of size [batch_size]
seq_len = tf.placeholder(tf.int32, [common.BATCH_SIZE])

model = tf.layers.conv2d(inputs, 64, (3,3),strides=(1, 1), padding='same', name='c1')
model = tf.layers.max_pooling2d(model, (3,3), strides=(2,2), padding='same', name='m1')
model = tf.layers.conv2d(model, 128,(3,3), strides=(1, 1), padding='same', name='c2')
model = tf.layers.max_pooling2d(model, (3,3),strides=(2,2), padding='same', name='m2')
model = tf.transpose(model, [3,0,1,2])
shape = model.get_shape().as_list()
model = tf.reshape(model, [shape[0],-1,shape[2]*shape[3]])

cell = tf.nn.rnn_cell.LSTMCell(common.num_hidden, state_is_tuple=True)
cell = tf.nn.rnn_cell.DropoutWrapper(cell, input_keep_prob=0.5, output_keep_prob=0.5)
stack = tf.nn.rnn_cell.MultiRNNCell([cell]*common.num_layers, state_is_tuple=True)

outputs, _ = tf.nn.dynamic_rnn(cell, model, seq_len, dtype=tf.float32,time_major=True)

My current approach is to use take input a word image pass it through a CNN extract high level image features, convert the image features to sequential data as below  [[a1,b1,c1],[a2,b2,c2],[a3,b3,c3]] -> [[a1,a2,a3],[b1,b2,b3],[c1,c2,c3]]
then pass it through a RNN(LSTM or BiLSTM), then use CTC (Connectionist Temporal Loss) to find the loss and train network.
I am not getting results as expected, I wanted to know if:

There is some other better way to do this task
If I am converting features to sequence correctly
Any research paper where something like this is done.

1 and 2. You are in the right direction, you need to extract the features using a CNN, then instead of predicting the class you want to reshape the last layer of features and feed it directly into the RNN.

A couple of things to pay attention to:

With a rather shallow CNN you aren't taking advantage of the higher-level feature extraction these architectures can offer. If all your images are as simple as the example you've shown it is adequate.
If you are considering a larger CNN, along with the RNN there is a substantial number of parameters to be trained. For this you need lots of data and lots of computational resources (either very strong GPUs or time).
In order for you to get the best of the two, I would suggest incorporating a pre-trained CNN into your model (and just fine tune the latter layers). This  pre-trained model can even be trained in generic images (i.e. ImageNet) and will substantially increase the performance of the CNN without computational cost. You then can train the latter layers of this CNN jointly with the RNN.

3.This is a good example of what you are trying to do. They basically try to recognize text from street photographs among other things with the same methodology you describe.

Similar methodologies can be found in other research domains such as multi-label image classification, sequence labelling, facial expression recognition, etc


Guidelines for Machine Learning model I'm trying to build a model of Machine Learning for the first time, and I'm looking for some guidelines.
My final purpose is to identify customers who are planning to flight abroad.
From my existing data, I identified customers who flew in the past, and I've flagged some potential variables (features) that might indicate that one is about to flight abroad.

My questions are:
1. What techniques should I use, in order to choose the right features for my model?
2. What approaches (models) would you use in such a problem?

because this is the first time I'm experiencing ML, I'll be happy to hear any further suggestions you have to give me.

Your help would be appreciated.

Which algo you should use is up to debate and sometimes comes down to the person

If you're going for accuracy then random forests are a great place to start (as mentioned in the comments) but personally I like logistic regressions as my starting place for this type of problem as they have decent accuracy and are comparatively explainable.

But if this is your first Machine Learning problem I'd recommend doing some wider reading first on how to formulate and approach machine learning problems.

It'll give you the perspective to decide which model to use on your own. Furthermore, you'll likely need to explain your model choices further down the line and using the "some dude from stack overflow told me to use it" excuse won't cut the mustard.

Links:

How to approach a ML problem

Intro to ML in Python


Deploying the prediction model under missing values for test data I have successfully built a logistic regression prediction model based on data set that is complete and clean, i.e., there is no missing values and the data is consistent.
Now, for deploying the model and testing it for online use, there is missing values in the inputs, i.e., not all inputs are available to predict the target value.

Is there a standard way to deal with this?

I can think of three ways to deal with the problem:

Treat "missing value" as another feature: Imagine you have a feature like "date of graduation". One possible (likely?) reason why this value is missing might be that the person did not graduate. So you could build a model which as a binary feature "graduation date is available" and the actual graduation date as another feature.
Predict the missing values: If data is missing because of your lack of knowledge of it (in contrast to the first point), then you might think about trying to predict the missing value. You could  also add a feature which encodes the certainty of the predicted value being correct.
Skip the feature: If it is missing very often and if it doesn't add much value to your prediction, you might simply want to remove it.


A data set to distinguish man from woman? Is there a set and/or a model that is available freely in the Internet that can tell man from woman based on input image?

You can use deepdetect which provides image classification functionality over API.

There is a gender classification model available on the models page. I have used it without problems before.


What features used by CNN model should a feature store actually store? According to MLOPs principle, it is recommended to have a feature store. The question is in the context of doing image classification using deep learning models like convolutional neural networks which does automatic feature engineering(using convolution layers) as part of the training process.
Questions

Does it make sense to have a feature store for pure image classification/segmentation models?
What features should be stored in the feature store? Output of convolution layers? but then they cannot be reused during the training since during training these will be rebuild by the convolution layers.

Disclaimer: i am one of the authors of the referenced link.

If you are doing pure image classification/segmentation, I would say no.
I would not store the output of convolution layers, noramlly. In principle, the output of models can be stored in the feature store, however.

Feature stores store data in tabular file formats (like parquet) and are used to store both immutable data (like images) and features that change over time (number of clicks on the website in the last 1 hour). These latter historical/contextual features need to be constantly updated by feature pipelines. Typically, you cache them in an online feature store, so that your model can retrieve them to build feature vectors.
You could store image data in the feature store, but I would normally just store the path to an image (on a distributed file system or object store), as well as all the metadata about the image that i need to make predictions. By storing image paths, it is easy to retrieve them to build training datasets in file formats designed for efficient training of DNNs on image data, like Petastorm.


Method to filter out objective statements (or say facts) I have millions of lines of statements containing both subjective (like I prefer the red skirt) and objective (Washington was born on February 22, 1732) statements or opinions. How can I separate them? Not manually.
By "objective", I mean if the object or predicative of the sentence changed, it would be in conflict with the truth. For illustration, in "She prefer red skirt.", if we change "red" or "skirt", the (new) statement would remain right, which can not apply to statements like "Washington was born on February 22, 2016".
The statements which should be pushed out:

Washington was born on February 22, 1732
Washington was born on February 21, 1731
Obama was born on February 22, 1732
Nobody A was born on February 22, 2016
Red is blue
Color is a day
I'm a robot
You are a person

Statements which should be kept:

I like coffee
Red is my favorite color
I hate him
You are awesome
You are clever
I was born years ago
You are very old
Tomorrow will be better

There isn't really an "objective" statement.

In your example: what if he wasn't born that day? Same statement, but is it still objective?

You may assume that "1 + 1 = 2" is an objective statement. But what if I'm doing binary math, and 1+1=0 then? So even that is subjective.

How would a machine tell apart these things, where philosophers will disagree with each other?

All you can do is provide training data examples of "your" (subjective) idea of objectiveness.


Getting a transition matrix from a Adjacency matrix in python I have a 3*3 Adjacency matrix and I'm trying to sum the elements of each column and divide each column element by that sum to get the transition matrix. Can you please help me code this part? Thanks in advance.

Considering a is your adjacency matrix 2D numpy array :

a / a.sum(axis=0)

Should do the trick (divide all elements by columns sum)


difference between logistic regression and binary logistic regression In xgboost R package, there are two objectives given with booster gbtree.

reg:logistic
binary:logistic

See,page 22 (first 2 lines) https://cran.r-project.org/web/packages/xgboost/xgboost.pdf

I was wondering, what is the difference between these two methods?

binary:logistic is used for binary classification where the target variable takes binary output [0, 1]

reg:logistic is used for regression where the target variable is continuous between [0, 1]

Quote from xgboost doc:

We use linear regression here, if we want use objective = reg:logistic logistic regression, the label needed to be pre-scaled into [0,1]

https://github.com/dmlc/xgboost/tree/master/demo/regression


Precision and recall clarification I'm reading the book Fundamentals of Machine Learning for Predictive Data Analytics by Kelleher, et al. I've come across something that I think as an error but I want to check to be sure. When explaining precision and recall the authors write:

  Email classification is a good application scenario in which the different information provided by precision and recall is useful. The precision value tells us how likely it is that a genuine ham email could be marked as spam and, presumably, deleted: 25% (1 − precision). Recall, on the other hand, tells us how likely it is that a spam email will be missed by the system and end up in our inbox: 33.333% (1 − recall).

Precision is defined as: $TP \over {TP + FP}$. Thus:
$$1 - precision = 1 - {TP \over TP+FP} = {FP \over TP + FP} = P(\textrm{prediction incorrect}|\textrm{prediction positive})$$
So this should give us the probability that an email marked as ham (positive prediction) is actually spam. So precision and recall in the quote above should be switched?

It's very likely that the authors assume that the spam class is positive, whereas you intuitively associated the ham class with positive. Both options make sense in my opinion:

the former interpretation is based on the idea that the goal of the task is to detect the spam emails, seen as the class of interest.
the later interpretation considers that the ham emails are the "good ones", the ones that we want, hence the "positive" class.

There's no error when one reads the paragraph with the authors' interpretation in mind. This confusion illustrates why one should always clearly define which class is defined as positive in a binary classification problem :)


ValueError: Expected 2D array, got scalar array instead using predict method I am trying to get a predicted value instead of whole features for a particular level using predict method.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

#Importing Dataset
dataset = pd.read_csv('C:/Users/Rupali Singh/Desktop/ML A-Z/Machine Learning A-Z Template Folder/Part 2 - Regression/Section 7 - Support Vector Regression (SVR)/Position_Salaries.csv')
print(dataset)
X = dataset.iloc[:, 1:2].values
Y = dataset.iloc[:, 2].values

# Feature Scaling

from sklearn.preprocessing import StandardScaler

sc_X = StandardScaler()
sc_Y = StandardScaler()
X = sc_X.fit_transform(X)
Y = sc_Y.fit_transform(Y.reshape(-1,1))
#Fitting SVR model to dataset

from sklearn.svm import SVR

regressor = SVR(kernel='rbf')
regressor.fit(X,Y)

#Visualizing the dataset

plt.scatter(X, Y, color = 'red')
plt.plot(X, regressor.predict(X), color = 'blue')
plt.show()

# Predicting a new Result

Y_pred = regressor.predict(6.5)
print(Y_pred)

This is my dataset, here I am trying to predict value only for level 6

Position  Level   Salary
0   Business Analyst      1    45000
1  Junior Consultant      2    50000
2  Senior Consultant      3    60000
3            Manager      4    80000
4    Country Manager      5   110000
5     Region Manager      6   150000
6            Partner      7   200000
7     Senior Partner      8   300000
8            C-level      9   500000
9                CEO     10  1000000

This is the error message I am getting:

File "C:/Users/Rupali Singh/PycharmProjects/Machine_Learning/SVR.py", line 34, in <module>
    Y_pred = regressor.predict(6.5)
  File "C:\Users\Rupali Singh\PycharmProjects\Machine_Learning\venv\lib\site-packages\sklearn\svm\base.py", line 322, in predict
    X = self._validate_for_predict(X)
  File "C:\Users\Rupali Singh\PycharmProjects\Machine_Learning\venv\lib\site-packages\sklearn\svm\base.py", line 454, in _validate_for_predict
    accept_large_sparse=False)
  File "C:\Users\Rupali Singh\PycharmProjects\Machine_Learning\venv\lib\site-packages\sklearn\utils\validation.py", line 514, in check_array
    "if it contains a single sample.".format(array))
ValueError: Expected 2D array, got scalar array instead:
array=6.5.
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

I would be really grateful for any kind of help.

Try:

Y_pred = regressor.predict(np.array([6.5]).reshape(1, 1))

Scikit does not work with scalars (just one single value). It expects a shape $(m\times n)$ where $m$ is the number of features and $n$ is the number of observations, both are 1 in your case.


Derivation of CNN math equations in Matrix format I've gone through jefkine's website and Jae Seo's articles to get a hold of math behind the famous CNN architecture. Although I understand it in theory, I'm unable to implement in matrix format or to put it straight...  In numpy format. After digging through internet like a broken web scraper, I could only understand FC ANN after Andrew Ng tutorial and this,  sudeepraja.github.io/Neural/. Can someone please help me understand CNN architecture in similar way?
I'm expecting downvotes and closed as off topic by admins for being naive and dumb. If that's the price I gotta pay to learn...  So be it!

I can't explain to you exactly what you want. However, if you got in this link and work on Q4, maybe this is what you are looking for.


Series data structure in pandas In the overview page of the pandas documentation the Series data structure is described as 'homogeneously-typed'.

Data Structures
Dimensions  Name    Description
1   Series  1D labeled homogeneously-typed array
2   DataFrame   General 2D labeled, size-mutable tabular structure with potentially heterogeneously-typed column

However it is possible to create Series objects with multiple data-types.

pd.Series(data=[1,2,3,4,5,'x'], index=['a','b','c','d','e','f'])
#=>a    1
#  b    2
#  c    3
#  d    4
#  e    5
#  f    x
#  dtype: object

So what could be the meaning of homogeneously-typedmentioned in pandas documentation?

Your series is indeed homogeneously-typed and you can check it's type:

s = pd.Series(data=[1,2,3,4,5,'x'], index=['a','b','c','d','e','f'])
s.dtype
> dtype('O')

where 'O' is for "object". However, if you check the type of the individual elements of your series, they are different:

type(s['a'])
> int
type(s['f'])
> str

I think the key thing to remember is that if your Series or DataFrame Column is non homogenously-typed (eventhough technically homogeneously-typed of type "object"), then there are certain pandas functions that won't work.

Here is a better explanation from the pandas documentation's Essential Basic Functionality:

  If a DataFrame or Panel contains homogeneously-typed data, the ndarray
  can actually be modified in-place, and the changes will be reflected
  in the data structure. For heterogeneous data (e.g. some of the
  DataFrame’s columns are not all the same dtype), this will not be the
  case. The values attribute itself, unlike the axis labels, cannot be
  assigned to.

  Note: When working with heterogeneous data, the dtype of the resulting ndarray will be chosen to accommodate all of the data
  involved. For example, if strings are involved, the result will be of
  object dtype. If there are only floats and integers, the resulting
  array will be of float dtype.


How to make an effective sampling from a database of text documents? Problem: I want to know methods to perform an effective sampling from a database.
The size of the database is about 250K text documents and in this case each text document is related to some majors (Electrical Engineering, Medicine and so on). So far I have seen some simple techniques like Simple random sample and Stratified sampling; however, I don't think it's a good idea to apply them for the following reasons:

In the case of simple random sample, for instance, there are a few documents in the database that talk about majors like Naval Engineering or Arts. Therefore I think they are less likely to be sampled with this method but I would like to have some samples of every major as possible.
In the case of stratified sampling, most of the documents talk about more than one major so I cannot divide the database in subgroups because they will not be mutually exclusive, at least for the case in which each major is a subgroup.

Finally, I cannot use the whole database due to expensive computational cost processing. So I would really appreciate any suggestions on other sampling methods. Thanks for any help in advance.

Off the top of my head, I would consider, at least, two approaches, as follows.

Graph sampling. If you can (and it makes sense to) model your population (database of text documents) as a graph, then consider graph sampling. Check this relevant answer of mine on the topic on Cross Validated site (contains many references - I would start with this nice overview paper).
Topic modeling. Another alternative is topic modeling (also referred to as topic models). See this introductory paper as well as this more detailed paper on the subject. Also, take a look at this interesting paper on topic modeling of streaming documents (or other situations, such as large databases, when unsupervised solutions make more sense). Finally, speaking about software for topic modeling, while many options do exist, I suggest investigating MALLET, which is an interesting Java-based software package for "statistical natural language processing, document classification, clustering, topic modeling, information extraction, and other machine learning applications to text".


Dealing with unbalanced error rate in confusion matrix Here is the confusion matrix I got when I was playing with Forest Type Cover Kaggle dataset : Link.

In the matrix, light color and higher numbers represent higher error rates, so as you can see, lots of mis-classification happened between class 1 and 0 .

I wonder what kind of methods I can use to reduce these two error rates though improvements have been made through combing two classifiers, Random Forest and Extra Tree. Will stacking help in this case?

Data can be found on
https://www.kaggle.com/c/forest-cover-type-prediction/data

Welcome to the site!

I think Ensemble Method is very tricky. when one of the model doesn't work well then the accuracy of the Ensemble also goes down.

For instance let us consider that you are using RandomForest(RF) and Rpart for classification and RF accuracy is 90% and Rpart accuracy is 60%. If you ensemble these 2 models then the Ensemble accuracy goes down.

Coming to your scenario, you need to be very careful at the time of stacking, you need to select the models that are performing moderately and then stack them to improve the accuracy.

How is the distribution of 0/1's, if they are imbalanced then you need to balance to improve the accuracy of the model. To handle imbalance data situation we use packages like SMOTE,ROSE etc.

Feature Engineering like adding external factors or adding new features, might help you to improve your models accuracy.

Do let me know if you have any additional questions.


Mahout Similarity algorithm comparison Which of the following is best (or widely used) for calculating item-item similarity measure in mahout and why ?

Pearson Correlation
Spearman Correlation
Euclidean Distance
Tanimoto Coefficient
LogLikelihood Similarity

Is there any thumb-rule to chose from these set of algorithm also how to differentiate each of them ?

For those not familiar, item-item recommenders calculate similarities between items, as opposed to user-user (or user-based) recommenders, which calculate similarities between users. Although some algorithms can be used for both, this question is in regard to item-item algorithms (thanks for being specific in your question).

Accuracy or effectiveness of recommenders is evaluated based on comparing recommendations to a previously collected data set (training set). For example, I have shopping cart data from the last six months; I'll use the first 5 months as training data, then run my various algorithms, and compare the quality against what really happened during the 6th month.

The reason Mahout ships with so many algorithms is because different algorithms are more or less effective in each data set you may work with. So, ideally, you do some testing as I described with many algorithms and compare the accuracy, then choose the winner.

Interestingly, you can also take other factors into account, such as the need to minimize the data set (for performance reasons), and run your tests only with a certain portion of the training data available. In such a case, one algorithm may work better with the smaller data set, but another may work with the complete set. Then, you get to weigh performance VS accuracy VS challenge of implementation (such as deploying on a Hadoop cluster).

Therefore, different algorithms are suited for different project. However, there are some general rules:

All algorithms always do better with unreduced data sets (more data is better).
More complex algorithms aren't necessarily better.

I suggest starting with a simple algorithm and ensuring you have high quality data. If you have additional time, you can implement more complex algorithms and create a comparison which is unique to your data set.

Most of my info comes from This study. You'll find lots of detail about implementation there.


How does " Sparsity of connections" in CNNs causes the network to have less parameters? I am studying Andrew NG's lectures on Convolutional Neural Network and he had provided two reasons for CNNs having less parameters compared to Non-Convolutional networks . They are :

Parameter Sharing
Sparsity of connections .

While I can make sense of the first reason causing CNN to have less parameters . I don't understand why Sparsity of connections , that is , " Each output in a layer comes from a small number of inputs " cause the network to have less parameters .
Isn't the second reason a bit redundant ?
Can someone please explain ?

In fact these involve different aspects of parameters in a CNN.

Parameter sharing, means one parameter may be shared by more than one input/connection. So this reduces total amount of independent parameters. Parameters shared are non-zero.
Sparsity of connections means that some parameters are simply missing (ie are zero), nothing to do with sharing same non-zero parameter. Parameters in this case are zero, ignored. That means that not necessarily all (potential) inputs to a layer are actually connected to that layer, only some of them, rest are ignored. Thus sparsity of connections.


Number and size of dense layers in a CNN Most networks I've seen have one or two dense layers before the final softmax layer.

Is there any principled way of choosing the number and size of the dense layers?
Are two dense layers more representative than one, for the same number of parameters?
Should dropout be applied before each dense layer, or just once?

First of all:

  There is no way to determine a good network topology just from the
  number of inputs and outputs. It depends critically on the number of
  training examples and the complexity of the classification you are
  trying to learn.[1]

and Yoshua Bengio has proposed a very simple rule:

  Just keep adding layers until the test error does not improve anymore.[2]

Moreover:

  The earlier features of a ConvNet contain more generic features (e.g.
  edge detectors or color blob detectors) that should be useful to many
  tasks, but later layers of the ConvNet becomes progressively more
  specific to the details of the classes contained in the original
  dataset.[3]

For example, in a method for learning feature detectors:

  first layer learns edge detectors and subsequent layers learn more complex features, and higher level layers encode more abstract features. [4]

So, using two dense layers is more advised than one layer.

Finally:

  The original paper on Dropout provides a number of useful heuristics to consider when using dropout in practice. One of them is:
   Use dropout on incoming (visible) as well as hidden units. Application of dropout at each layer of the network has shown good results. [5]

in CNN, usually, a Dropout layer is applied after each pooling layer, and also after  your Dense layer. A good tutorial is here [6]

References:

[1] https://www.cs.cmu.edu/Groups/AI/util/html/faqs/ai/neural/faq.html

[2] Bengio, Yoshua. "Practical recommendations for gradient-based training of deep architectures." Neural networks: Tricks of the trade. Springer Berlin Heidelberg, 2012. 437-478.

[3] http://cs231n.github.io/transfer-learning/

[4] http://learning.eng.cam.ac.uk/pub/Public/Turner/Teaching/ml-lecture-3-slides.pdf

[5] https://machinelearningmastery.com/dropout-regularization-deep-learning-models-keras/

[6] https://cambridgespark.com/content/tutorials/convolutional-neural-networks-with-keras/index.html


What software is being used in this image recognition system? I was wondering if anyone knew which piece of software is being used in this video? It is an image recognition system that makes the training process very simple.

http://www.ted.com/talks/jeremy_howard_the_wonderful_and_terrifying_implications_of_computers_that_can_learn#t-775098

The example is with car images, though the video should start at the right spot.

I'm pretty sure that the software you're referring to is a some kind of internal research project software, developed by Enlitic (http://www.enlitic.com), where Jeremy Howard works as a founder and CEO. By "internal research project software" I mean either a proof-of-concept software, or a prototype software.


Is it worth to upgrade CUDA and cuDNN while having older GPUs? New CUDA 11.x versions add support for TF32 format, other new features for newer cards (RTX30xx, A100 etc).
Is it worth upgrading to CUDA 11.x if you have GTX 1050 or RTX 2080 (having tensor cores)?
Could it be that new features only add computational overhead (at least in the size of installation file, they do), and an older GPU won't be able to use the new features?

There is no right answer to this question, because there are many factors to consider, for instance:

Indirect dependency on specific/minimum CUDA version: with deep learning frameworks like pytorch and tensorflow, a specific version of the framework depends on a specific version of CUDA/cuDNN, so the decision of which (minimum) CUDA version to have is directly dependent on the framework version you need.

Bug fixes: new versions or old versions may introduce unexpected bugs that make upgrading/not upgrading not an option.

Performance regressions: some new CUDA versions may introduced performance regressions for older cards (e.g. this, this).

About support for new features, these are normally tied to new hardware (i.e. tensorcores), so it is not frequent to gain new features just by upgrading. But it is not impossible either ¯\(ツ)/¯
So, the answer is: it depends on the case.


What's the best way to generate similar words? Hi all I'm fairly up to date with all the NLP tasks out there (nlpprogress.com, paperswithcode.com) and great tools like (nltk, flair, huggingface etc). I want to take a single word, and predict a similar word, a little like the old "Google Sets" feature except extrapolating from a single example. I'm thinking GPT-3 might be the best bet with some seed text like
here is a list of similar things: banana,

and ask it to predict the next word.
transformer.huggingface.co is promising enough (though hilariously inadequate in itself) that I'm thinking GPT-3 indeed may well be the answer.
But the alternative is to navigate a treebank, through "type of" relationships… much, much faster and cheaper.
I've tagged this "semantic similarity" but really I don't want the relationship to be "similar", rather "is part of same set of".
thoughts most appreciated from actual practitioners in this space rather than hobbyists like me :)

But the alternative is to navigate a treebank, through "type of" relationships… much, much faster and cheaper.

WordNet provides exactly this: it is a lexical database in which words are grouped by synonyms, with several types of relations between groups in particular hypernyms/hyponyms (more general/more specific).
The database can be downloaded and there is a library to use it through nltk.


Why BERT tokenizers function differently? While experimenting with transformers' TFBertForSequenceClassification and BertTokenizer, I noticed that BertTokenizer:
transformer_bert_tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")

tokenizes the text differently from the tokenizer that I use to construct for my BERT models in this way:
!wget --quiet https://raw.githubusercontent.com/tensorflow/models/master/official/nlp/bert/tokenization.py
import tokenization
FullTokenizer = tokenization.FullTokenizer

and then
BERT_MODEL_HUB = 'https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/2'
bert_layer = hub.KerasLayer(BERT_MODEL_HUB, trainable=True)
to_lower_case = bert_layer.resolved_object.do_lower_case.numpy()
vocabulary_file = bert_layer.resolved_object.vocab_file.asset_path.numpy()

tokenizer = FullTokenizer(vocabulary_file, to_lower_case)

as an example:
sequence = "Systolic arrays are cool. This  is cool too."
transformer_bert_tokenizer .tokenize(sequence)
# output: ['s', '##ys', '##to', '##lic', 'array', '##s', 'are', 'cool', '.', 'this', '[UNK]', 'is', 'cool', 'too', '.']
tokenizer2.tokenize(sequence)
# output: ['sy','##sto','##lic','arrays','are','cool','.','this','[UNK]', 'is','cool','too','.']

Does anyone know why there is a difference? Aren't both tokenizers using the same vocabulary? which way is preferred?

Each BERT variant is trained with text that has been prepared differently, e.g. as the name implies, BERT uncased is trained with text where all letters are lowercase. This means that the vocabulary extraction process has also use lowercase text as input, and therefore gives as result a different vocabulary than the same vocabulary extraction process used with text in its original casing.
Note that, as the vocabularies are different, each model should be used with the tokenizer it was used to train it. Using a model with a different tokenizer may lead to bad results.
The choice of tokenizer, therefore, is tied to the choice of BERT model. The criteria to use one or the other (e.g. uncased vs. cased), depends on the case. For instance, for doing named entity recognition (NER) in English, it may be important to keep the original casing so that the model can more easily distinguish proper nouns.


Versatile data structure for combined statistics Not sure if this is Math, Stats or Data Science, but I figured I would post it here to get the site used.

As a programmer, when you have a system/component implemented, you might want to allow some performance monitoring. For example to query how often a function call was used, how long it took and so on. So typically you care about count, means/percentile, max/min and similiar statistics. This could be measurements since startup, but also a rolling average or window.

I wonder if there is a good data structure which can be updated efficiently concurrently which can be used as the source for most of those queries. For example having a ringbuffer of rollup-metrics (count, sum, min, max) over increasing periods of time and a background aggregate process triggered regularly.

The focus here (for me) is on in-memory data structures with limited memory consumption. (For other things I would use a RRD type of library).

It sounds like you would like the Boost Accumulators library:

  Boost.Accumulators is both a library for incremental statistical
  computation as well as an extensible framework for incremental
  calculation in general. The library deals primarily with the concept
  of an accumulator, which is a primitive computational entity that
  accepts data one sample at a time and maintains some internal state.
  These accumulators may offload some of their computations on other
  accumulators, on which they depend. Accumulators are grouped within an
  accumulator set. Boost.Accumulators resolves the inter-dependencies
  between accumulators in a set and ensures that accumulators are
  processed in the proper order.


LDA for sentiment analysis As far as I understand it, LDA works by assuming that a corpus was written by a set of topics ands words corresponding to that topic by a specific distribution. I'm however not enterely sure what the output of the LDA process is and more importantly, how one can (for example) model a corpus of political speeches to their sentiments.

LDA is a unsupervised technique which identifies the set of words that compromise of a topic.
This is done by assuming that there is a generator that generates the sentences using the words.
To simplify the assumptions on the number of topics are made which ideally has to be learnt from the data.

The output of LDA is the words that represent a topic and probability of a sentence/ text observation being a mixture of the N topics we initially assumed.

We will get a 1xN row vector for each text observation that has probability of it containing topic n (which is one of the N topics)


what does random seed value mean in weka? I use weka to make a dataset classification but there is an option in classifier evaluation ( random seed for XVAL/% split). What does this option mean? ,and what is the seed value ? ,and What is the effect of changing the value of this option from one to two or three or other values?.

I read that the value of the seed is the starting point, but what is the difference if it is the starting point(seed value) 1 or 2 or 10 , for example.

Seed is just a value by which you can fix the Random Numbers that are being generated in your task. Also, this is a general concept and not just for weka.

So, here random numbers are being used to split the data. Now if you run the code without fixing any seed, you will get different splits on every run. But if you fix the seed to some specific value, you will get the same split every time. This is useful when you want to make your scores reproducable.


How does ResNet bottleneck architecture's input size is possible to change from 56x56x64 to 56x56x356? In ResNet papaer, First residual block's input size is 56x56x64 caused by 7x7x64 filter in first layer. But, in the paper, they showed residual block that has 56x56x256 input size. How does it is possible to change from 56x56x64 to 56x56x256?

I do not know the answer for sure but I assume that the "256-d" refers to the shortcut connection and not the input. Since the output dimension of the 3 conv. layers in your right hand picture (fig. 5) is 256 the shortcut is transformed from depth 64 to depth 256.

How this is being done is explained in the following paragraph on p. 4:

  Residual Network. Based on the above plain network, we
  insert shortcut connections (Fig. 3, right) which turn the
  network into its counterpart residual version. The identity
  shortcuts (Eqn.(1)) can be directly used when the input and
  output are of the same dimensions (solid line shortcuts in
  Fig. 3). When the dimensions increase (dotted line shortcuts
  in Fig. 3), we consider two options: (A) The shortcut still
  performs identity mapping, with extra zero entries padded
  for increasing dimensions. This option introduces no extra
  parameter; (B) The projection shortcut in Eqn.(2) is used to
  match dimensions (done by 1×1 convolutions). For both
  options, when the shortcuts go across feature maps of two
  sizes, they are performed with a stride of 2.

For ResNet-34 this not required for the first building block which is shown in fig. 5 but for later blocks too. This is shown by the dotted lines in figure 3:


Strings/ features in Turicreate decision tree I am trying to create a prediction model by using a decision tree with Turicreate. While my problem does involve numbers, it also involves strings and ultimately I want it to return the string 'true/false'. Are Turicreate decision trees able to process strings as input and output?

I'm not familiar with the package, but the documentation suggests that strings as input will be handled internally by dummy-encoding (with a reference level).  The linked section is for linear regression, but is linked to from the Advanced Features section of the decision tree page, so I assume that applies.


Identifying top predictors from a mix of categorical and ordinal data I have a dataset with 261 predictors scraped from a larger set of survey questions. 224 have values which are in a range of scale (some 1-10, some 1-4, some simply binary, all using 0 where no value is given), and the rest are unordered categories.

I'm trying to perform classification using these predictors and identify the top n predictors. Am thinking of the following approach:

convert the 224 ordered predictors into numeric, centered, and scaled.
Run separate modeling (I use caret from R): one for using the numeric predictors, another using the  remaining 37 categorical predictors (both cross-validated within each modeling exercise).
Choose the respective best-fitting models modelN and modelC for the numeric and categorical predictors.
Choose top n (say 10) predictors from model N and model C.
Combine them in an ensemble model that can handle both numeric and categorical data (say, random forest).
Choose top n predictors in the ensemble model.

I am going through this a roundabout way rather than directly fitting all predictors into an ensemble model to try and reduce the complexity of the problem first (and because in R, I'm having a problem with too many levels from the predictors).

Would this be a valid approach to identifying the n most salient predictors?
Any possible issues to mitigate?

Ricky,

Loose thoughts:

Depending on the algorithm you intend to use, centering might not be a good idea (e.g. if you go for SVM, centering will destroy sparsity)
I would suggest not to handle ordered / unordered separately, as you are likely to miss interactions that way. If the categorical ones don't have too many possible values, randomForest in R can handle factors.
if that is an issue (as you seem to hint), I think you have two possibilities: binary indicators or response rates
if it's feasible in terms of computational cost, i would convert all factors to binaries (use sparse matrices if necessary) and then try a greedy feature selection. caret, if memory serves, has rfe or somesuch.
if that's too much trouble, try calculating response rates / average values per factor level (I don't see any info whether your problem is classification or regression): you split your set into folds, and then for each fold fit a mixed effects model (e.g. via lme4) on the remainder, using the factor of interest as the main variable. It's a bit of a pain to setup all the cv correctly, but it's the only way to avoid leaking information.

Hope this helps,
K


Computer science corpus for training a language model I am looking for a domain specific computer science corpus of at least 20M words (preferable >50M words), for the purpose of training a language model in it.

Is there anything out-of-the box that I could use?
*I tried to look for the sciBERT corpus, can not find how to access it.

Thanks!

Depends on the domain and language, but I'll share an adaptive example.

The wikipedia corpus's English version contains more than 1.9 billion words from 4.4 million articles.

You can create create virtual corpora from the full corpus to contain only topics of interest, such as biology, investments, Buddhism, psychology, cars, basketball, etc.


Is time series multi-step ahead forecasting a sequence to sequence problem? I'm using the keras package in order to train an LSTM for a univariate time series of type numeric (float). Performing a 1-step ahead forecast is trivial, but I'm not sure how to perform a, let's say, 10-step ahead forecast. Two questions:

1) I read about sequence to sequence NNs, but can barely find anything of it in the context of time series forecasting. Am I right with the assumption that the forecasting of more than 1 time step in advance is a seq2seq problem? That makes sense to me because each forecast depends on its predecessor.

2) An intuitive solution without seq2seq would be: Perform 1-step ahead forecast, then append this forecast to the series and use it to obtain the next forecast, and so on. How would this differ from a seq2seq approach?

Seq2Seq architecture can definitely be used for time series problem. The only twist is that you will need a linear layer on top of your decoder to project the outputs to the required size (for example, 1 for univariate).

The stepwise forecast approach can be used for short sequences but because any biases are compounded using this approach, it is not good for longer sequences.

For example, if you have a sequence where the value is constant at each time step $x_{i+1} = x_i$, but you model learned to do $x_{i+1}=1.01x_i$ instead (which is highly likely with gradient descent algorithm). For $t=10$, the target value will be $1^{50} = 1$, however you model will predict $1.01^{50}=1.64$.

Thus a 1% single step error results in a 64% difference in 50 steps.


Classify graphs using machine learning I am trying classify an input graph (a 2D point sequence) into one of the predefined graphs (A,B,C etc) using machine learning. The goal is to identify which type of graph the input graph belongs to.

I have done classification of single data points before, but I have never classified sequences of data like graphs before. The only way I could think of is calculating 'mean squared error' between input & each of A,B,C graphs and choose the category with the lowest error.

3 example outputs might look like this.

input graph belongs to type A (confidence: 82%)
input graph belongs to type C (confidence: 68%)
input graph doesn't belong to any type (max confidence: 12%)

How can I achieve this using classification techniques or any other accurate way?

The wording of your question makes it seems as if you only want to answer the question is the current graph to a specific graph A,B or C. If this is the case, machine learning might not be the best approach and in fact I would say your MSE based approach is probably a good starting place. If this is the case You may want to calculate the performance of this approach and see if it meets your needs. If you have a bunch of examples of graphs that are class A,B, and C then machine learning a probably good approach.

If you have a bunch of examples of each class then you'll first need to calculate a set of features on each graph before you can apply machine learning techniques. After calculating the value of each feature on the graph you'll have a feature vector. From this point, the classification problem should be the same as the "single" data point problems you've already encountered.

Example features:

Min or max x or y value
The average slope of the series
Path length of the series
Maximum curvature of the series
Coefficients from a polynomial fit

The tsfresh package will calculate a bunch of features for time series and help you sort out the relevant ones so its probably a good place to start.

Good luck!


Is it better to use separately regularization methods for Neural Networks (L2/L1 & Dropout) I have been exploring different regularization approaches and observed the most common to be using either Dropout Layers or L1/L2 Regularization. I have seen many debates of whether it is of interest to either combine or seperate regularization methods.
In my case I have implemented/integrated both approaches (combined and separate). For which I have seen promising results when actually combining as it has helped me not to always overfit my models entirely while generally improving the r2 score of my model.
Question:
Is it preferable to combine L1/L2 Regularization with Dropout Layer, or is it better to use them separately?
Example Code:
def model_build(x_train):
    # Define Inputs for ANN
    input_layer = Input(shape = (x_train.shape[1],), name = "Input")
    #Create Hidden ANN Layers
    dense_layer = BatchNormalization(name = "Normalization")(input_layer)
    dense_layer = Dense(128, name = "First_Layer", activation = 'relu', kernel_regularizer=regularizers.l1(0.01))(dense_layer)
    #dense_layer = Dropout(0.08)(dense_layer)
    dense_layer = Dense(128, name = "Second_Layer", activation = 'relu',  kernel_regularizer=regularizers.l1(0.00))(dense_layer)
    #dense_layer = Dropout(0.05)(dense_layer)

    #Apply Output Layers
    output = Dense(1, name = "Output")(dense_layer)

    # Create an Interpretation Model (Accepts the inputs from branch and has single output)
    model = Model(inputs = input_layer, outputs = output)

    # Compile the Model
    model.compile(loss='mse', optimizer = Adam(lr = 0.01), metrics = ['mse'])
    #model.compile(loss='mse', optimizer=AdaBound(lr=0.001, final_lr=0.1), metrics = ['mse'])

I am unsure there will be a formal way to show which is best in which situations as it depends on many factors like your dataset, architecture of the your ANN  - simply trying out different combinations is likely best.
It is worth noting that Dropout* is actually doing more than just regularization, it makes the model more robust,allowing it to try different nodes for prediction.
As for L1/L2 it just reduces overfitting by penalizing the higher weights.


What is a "residual mapping"? A recent paper by He et al. (Deep Residual Learning for Image Recognition, Microsoft Research, 2015) claims that they use up to 4096 layers (not neurons!).

I am trying to understand the paper, but I stumble about the word "residual".

Could somebody please give me an explanation / definition what residual means in this case?

Examples

  We explicitly reformulate the layers as learning
  residual functions with reference to the layer inputs, instead
  of learning unreferenced functions.

  [...]

  Instead
  of hoping each few stacked layers directly fit a
  desired underlying mapping, we explicitly let these layers
  fit a residual mapping. Formally, denoting the desired
  underlying mapping as $\mathcal{H}(x)$, we let the stacked nonlinear
  layers fit another mapping of $\mathcal{F}(x) := \mathcal{H}(x)−x$. The original
  mapping is recast into $\mathcal{F}(x)+x$. We hypothesize that it
  is easier to optimize the residual mapping than to optimize
  the original, unreferenced mapping

It's $F(x)$; the difference between the mapping $H(x)$ and its input $x$. It's a common term in mathematics (DE).


Word analysis in Python I have a list of documents which look like this:
["Display is flickering"]
["Battery charger is broken"]
["Hard disk is making noises"]

These text documents are just free text.
I have processed with tokenization, lemmatization, stop words removal, and now I want to assign tags based on a list of words. Example:
{"#display":["display","screen","lcd","led"]}
{"#battery":["battery","power cord","charger","drains"]}
{"#hard disk":["hard disk","performance","slow"]}

After text normalization I have:
["Display is flickering"] -> ["display","flicker"]

What technique is recommended to compare document: ["display","flicker"]
with my dictionary of words and return which value matches the best?
In this case I would like:
["display","flicker"] = "#display":"display"
["battery","charger","broke"] = "#battery":"charger"

Basically it compares Document A in tokens with a list B of other documents and return which document in list B with more common matches.
I'm using TF, but want to know if there are other techniques, code samples to use.

You can use word embedding in order to compare whole phrases. I am aware about two models: Google's word2vec and Stanford's GloVe. Now, word embedding works best with, well - words. However, you could concatenate every word in your phrase and re-train the models. Afterwards, you could calculate their similarity (say, with cosine similarity) and see how similar your whole phrases are semantically.

Hope this helps.


Classify multivariate time series I have a set of data composed of time series (8 points) with about 40 dimensions (so each time series is 8 by 40). The corresponding ouput (the possible outcomes for the categories ) is eitheir 0 or 1.

What would be the best approach to design a classifier for time series with multiple dimensions ?

My initial strategy was to extract features from those time series : mean, std, maximum variation for each dimension. I obtained a dataset which I used to train a RandomTreeForest. Being aware of the total naivety of this, and after obtaining poor results, I am now looking for a more improved model.

My leads are the following :
classify the series for each dimension (using KNN algorithm and DWT), reduce the dimensionality with PCA and use a final classifier along the multidimensions categories.
Being relatively new to ML, I don't know if I am totally wrong.

If you're in Python, there are a couple of packages that can automatically extract hundreds or thousands of features from your timeseries, correlate them with your labels, choose the most significant, and train models for you.

https://github.com/blue-yonder/tsfresh

https://github.com/rtavenar/tslearn


Finding a pattern in reponses in R programming I did a study in which around 1000 participants took a test (100 questions). In this study participants were asked in each question to choose between two texts (text 1 and text 2) and decide which text is easier for them. Now in R I want to check if there are any participants who followed a pattern. For example, he or she have only chosen texts 1 or text 2. I also want to examine response string screening for participants that alternated left/right/left/right etc... 20, 30, 40, 50, 60, 70, 80, 90, 100 times in a row. Can anyone help me that how I can do it in R?

Generally I would suggest to look for differences with the general pattern, the general pattern being the answers from most users. A very basic way to do that:

Calculate for every question the proportion of text 1, store it as a vector. This distribution is the "mean vector".
For every player represent the vector of their answers: for every question the value is 1 if they chose text 1, 0 if they chose text 2.
For every player compare the vector of their answer with the "mean vector", for instance with cosine similarity (or any other distance/similarity measure).

Then look at the distribution of the similarity scores: normally most players have a high similarity scores, so if some of them have very low similarity it's likely that they didn't do the test seriously.
If you want a more advanced method, you could cluster the players' vectors or do anomaly detection.

[edit based on comment]

Take the proportion of "text 1" for every participant and compare to the mean proportion across all participants. This would catch the most obvious outliers, but probably not any subtle pattern.
Measure how much correlation there is between the question number and the answer: normally the two are independent so the correlation should be very low, if not there is pattern. Note that I think there would be a better statistical test than Pearson correlation, but I don't remember which one would suit this case (Spearman correlation might be relevant, I'm not sure).


Feature Extraction Using A CNN I want to use a CNN to extract features from a dataset. My questions are:

What is the stopping criteria when training a CNN to extract features?
What is the difference between extracting features after training for 50 epochs or 100 epochs?
How should I choose the best layer for feature extraction?
Does it depend on the accuracy of the model?

my question is what the stopping criteria of train CNN to extract features?

Generally speaking, the best features are the ones that, when classified, should have the best performance. So train your model, while monitoring a metric of your choice on a validation set. Stop the training phase when that metric stops improving.

  What the difference between extract features after train 50 epoch and 100?

Assuming no overfitting has occurred between epochs 50 and 100, the features of the latter will be more sophisticated and better equipped for classification on the given dataset.

If the model has started to overfit between these two epochs, then the latter model will have begun to memorize the training set. The features of this model will have begun to extract features tailored to the training set images.

  How to choose the best layer for extraction?

You should get the highest-level features available from the CNN. The most usual case is taking the previous layer of the first fully connected one. However, if that layer is a pooling one, you might want to take the previous one (which contains more information).

  Are it depend on the accuracy of model?

This comes back to the first question. If you think that accuracy is a good enough metric to represent your model's performance, then yes. There are many cases, however, where this is not the case (e.g. imbalanced dataset). Whichever metric you choose, it should be measured on the validation set.


Ways of calculating the area of colored regions in a map Background
I am a PHD student trying to improve my data science. One of my research projects, has me tasked with determining the size of the clusters in a colored image of regions. Here is an example image I am using. The coloring is natural as it represents the orientation of the microscope light. The light hits the surface in different ways resulting in the different colors. But I'm not trying to sum regions of similar colors, but instead just determine the area of each of those polygon regions. I don't have ground truth or labeled images.
Problem
I am trying to calculate the size of each of these colored regions.

Attempts to solve the problem
I don't have labeled images so I have not attempted any supervised approaches. And I don't know if my approach is wise, but my current approach has me solving two problems at once. First is detecting the edges, second is counting the pixels. Maybe there is a better approach?
To detect the edges I am using K means as provided by weka. The parameters I used are below.

weka.clusterers.SimpleKMeans -init 0 -max-candidates 100 -periodic-pruning 10000 -min-density 2.0 -t1 -1.25 -t2 -1.0 -N 70 -A "weka.core.EuclideanDistance -R first-last" -I 500 -num-slots 1 -S 40

Another way I tried solving this problem is by graphing the RGB values of the pixels to determine the co-variance of the pixels in order to determine the boundaries. Then I would add the pixels between the boundaries to determine the size of the region.
Issues with my approach
By using k means, I have to know ahead of time how many colored regions are in the image and I don't know this. By using the co-variance method, it determines the boundaries well in the color array, I don't have a way of knowing the area of the region.
Question
What would a more enlightened way of going about this be, that I could try next? What method would work best here?
More information
Image format: TIF
Compression: None
Size: 2000 x 2000 Pixels (4 MPixels)
Colors: 16.7 Million (24 BitsPerPixel)
Number of Unique Colors: 382,887

Finding the size of colored regions might be better framed as a convex hull  problem since the borders are distinct. After each convex hull is defined, the region can be defined in pixels.


2D matrix for labelbinarizer There is one behavior of labelbinarizer

import numpy as np
from sklearn import preprocessing
lb = preprocessing.LabelBinarizer()
lb.fit(np.array([[0, 1, 1], [1, 0, 0]]))
lb.classes_

The output is array([0, 1, 2]). Why there is a 2 there?

I think the documentation is kind of self-explanatory here. Fit takes in array of size n_samples in which each element is the class of the datum or if the data point belongs to multiple classes, the input would be obviously of size n_samples x n_classes. That is what you gave in as input in your example. Each point can belong to any of the three classes. That is why you have [0, 1, 2] as number of classes. So as mentioned in the documentation if you try

>> lb.transform([0, 1, 2, 0])
[[1 0 0]
 [0 1 0]
 [0 0 1]
 [1 0 0]]

and if you try a class that is non-existent after fit like

>> lb.transform([0, 1, 2, 1000])
[[1 0 0]
 [0 1 0]
 [0 0 1]
 [0 0 0]]

No class named 1000 exists, so multi-targeted conversion for 1000 class case is plainly [0, 0, 0]. Hope this helps.


Machine Learning: creating a prediction without previous trends Is there a machine learning algorithm that allows one to give an input a classification without a grouping or previous classification? Example: with a dataset of 5 digit numbers, the user inputs a random 5 digit number (12345), can the machine output "alteration" when 12345 is similar to something in the dataset, or output "completely new" when 12345 or anything similar doesn't exist? To clarify, the machine would do something like "alteration" when it finds something similar to 12345 such as 12445 where its only number difference, and output "completely new" when no similar numbers exist.

At first sight I don't see the need for ML, you just need a similarity measure which would return a score between two objects:

Given the input object, say "12345", compare it against any known object using the similarity measure. The result is a similarity score for every known object.
Rank the known objects according to their similarity against the input object, or just obtain the object with the highest similarity. for example "12445" is the most similar with score 0.8.
Apply a threshold $t$: if the max similarity is higher than $t$, then the corresponding known object is returned; otherwise the system says that the input object is "completely new".

Of course ML could still be used, for instance to tune the similarity measure or to find the optimal threshold.


XG Boost Regression to model Log of Dependent Variable I'm working on a data set that has continuous dependent variable. I used XG Boost to model the dependent variable. However, when I transformed the dependent variable by applying Log transformation and then modeled it using XG Boost, the results were drastically improved. I am getting results close to 100% of actual on Test Data. Is there any explanation to this ?

It's actually good practice to perform some type of scaling of your continuous variables. The box-cox transformation (or positive variable) or the yeo-johnson transformation (which adapts the box-cox transformation) are in the class of power transorms implemented in scikit-learn.
I you look at the implementation, you see that the log transformation that you applied is a particular case of box-cox.
Gradient boosting includes an optimisation step that will rely on computing gradients of your loss function. A recurrent theme in ML is how gradient descent have a harder time finding optimal solution when the data has a high order of magnitude. Essentially the information is there at high order of magnitude but the optimisation space is "stretched" - scaling your data brings everything closer if you will (that's a really gross explanation of what happens).
So essentially you've done a great job at making the most out of your data by transforming it to get a better model.
And this is actually an excellent case because, you often hear that when using decision tree you should not scale the data, and by that what is meant are the features. When using a decision tree model for regression you should absolutely consider scaling the target.


Is TensorFlow a complete Machine Learning Library? I am new to TensorFlow and I need to understand the capabilities and shortcomings of TensorFlow before I can use it. I know that it is a deep learning framework, but apart from that which other machine learning algorithms can we use with tensor flow. For example can we use SVMs or random forests using TensorFlow? (I know this sounds crazy)

In short, I want to know which Machine Learning Algorithms are supported by TensorFlow. Is it just deep learning or something more?

This is a big oversimplification, but there are essentially two types of machine learning libraries available today:

Deep learning (CNN,RNN, fully connected nets, linear models)
Everything else (SVM, GBMs, Random Forests, Naive Bayes, K-NN, etc)

The reason for this is that deep learning is much more computationally intensive than other more traditional training methods, and therefore requires intense specialization of the library (e.g., using a GPU and distributed capabilities). If you're using Python and are looking for a package with the greatest breadth of algorithms, try scikit-learn. In reality, if you want to use deep learning and more traditional methods you'll need to use more than one library. There is no "complete" package.


Linear Regression - Metrics to consider? While evaluating the Linear regression model, which metric do we need to consider R square or Mean Squared Error (or which needs to be given more importance)?

Thank you.

Aditya already mentioned rightly so in the comments RMSE, MSE or MAE is preferred while evaluating a linear regression model. So to answer your questions I will provide with the pros and cons of using RMSE and MAE and further more put forward some points to why R-squared is not preferred.

1. MAE vs. RMSE - Which metric is better ?

Ideally this depends on the loss function of your model, the most common one being L2 - euclidean distance (you can create your own loss function too). RMSE gives relatively higher weight to large errors. It does not necessarily increase with the variance of the errors but increases with the variance of the frequency distribution of error magnitudes. While, MAE is comparatively steady.

typically, it makes more sense to give a higher weight to points away from the mean in which RMSE would be preferred. In a scenario where being away from the mean by any magnitude would be penalized equally, MAE would be preferred.

RMSE can be troublesome when calculating on different sized test samples (As is the case when training a model). RMSE has a tendency to be increasingly larger than MAE as the test sample size increases ([RMSE] ≤ [MAE * sqrt(n)].

2. Is R-squared useless ? (Kinda harsh but sorry not sorry)

To keep it short and simple.

R-squared does not measure goodness of fit.
R-squared does not measure predictive error.
R-squared does not allow you to compare models using transformed
responses.
R-squared does not measure how one variable explains another.


Scalable way to group users with similar titles purchased I'm trying to figure out the best way to group customers based on checkout items in their shopping cart. I have the basket, and what's in the basket, but am at a complete loss on how to group all the similar baskets. I have a group of users I believe shouldn't be counted in my overall metrics (or at least acknowledge them). These users create a new account, place 4-5 items in their cart, and check out. Then a new account is created, and the process repeats. They seem to repeat this process for up to 12 hours and then change the items, and I'd like to group those customers vs "regular" customers.

My issue is that it's not always the same 4-5 items -- and I can't figure out how to scale up the 'search'. If I knew it was Item1-Item5, it would be an easy search, even comparing everyone's cart to that, but comparing everyone's cart to everyone else's cart seems like it won't scale at all. (Also the fact that I'd like to have the 'marginals' as well if I'm looking for 5 titles, but they got 4 of them, or all 5 and one new one, etc.)

I'm completely new to this, and even pointers to what terms I should be searching for, or what software packages I should aim to be learning a bit of would be very appreciated.

This issue seems somewhat similar to a recommendation system problem, if I understood correctly, in which you have several titles you want to recommend to users based on their previous interactions.

Maybe you could read on that? You can search for algorithms such as Matrix Factorization/Collaborative Filtering.

A good resource for that: https://course.fast.ai/videos/?lesson=4


What is the difference between Pytorch's DataParallel and DistributedDataParallel? I am going through this imagenet example.

And, in line 88, the module DistributedDataParallel is used. When I searched for the same in the docs, I haven’t found anything. However, I found the documentation for DataParallel.

So, would like to know what is the difference between the DataParallel and DistributedDataParallel modules.

As the Distributed GPUs functionality is only a couple of days old [in the v2.0 release version of Pytorch], there is still no documentation regarding that. So, I had to go through the source code's docstrings for figuring out the difference. So, the docstring of the DistributedDataParallel module is as follows:

  Implements distributed data parallelism at the module level.
      This container parallelizes the application of the given module by
      splitting the input across the specified devices by chunking in the batch
      dimension. The module is replicated on each machine and each device, and
      each such replica handles a portion of the input. During the backwards
      pass, gradients from each node are averaged.
      The batch size should be larger than the number of GPUs used locally. It
      should also be an integer multiple of the number of GPUs so that each chunk
      is the same size (so that each GPU processes the same number of samples).

And the docstring for the dataparallel is as follows:

  Implements data parallelism at the module level.
      This container parallelizes the application of the given module by
      splitting the input across the specified devices by chunking in the batch
      dimension. In the forward pass, the module is replicated on each device,
      and each replica handles a portion of the input. During the backwards
      pass, gradients from each replica are summed into the original module.
      The batch size should be larger than the number of GPUs used. It should
      also be an integer multiple of the number of GPUs so that each chunk is the
      same size (so that each GPU processes the same number of samples).

This reply in the Pytorch forums was also helpful in understanding the difference between the both,


Are mainstream pre-trained models useful as discriminators? In the context of GANs I see many papers designing new discriminator networks.

I'm curious about the usefulness of designing discriminators as modified versions of mainstream models like Inception, MobileNet, EfficientNet etc. My intuition is that the mentioned image classification models are way more refined and standardised than a custom discriminator for a specific paper. In addition, I think their pretrained weights should be helpful.

And if they aren't useful, I'm curious why not. Any links about this topic are much appreciated.

Using a standard network architecture is perfectly reasonable. Most discriminator architectures are trivially different variants of well-known architectures anyway.
Depending on the GAN loss, starting with a pretrained network as a discriminator may be dangerous. For instance, the classical GAN loss minimizes the Jensen-Shannon divergence, so having a powerful discriminator right from the start will mean little support overlap and no useful gradients from the discriminator. This is a major reason for training instability in GANs. Yet, more recent losses (e.g., the various WGAN losses) do not have this shared support problem. However, losses like WGAN assume certain properties of the discriminator (i.e., Lipschitz-$k$), which pretrained networks are quite unlikely to satisfy out of the box. So from a purely theoretical perspective, I can see why it's not very appealing.
Another reason, I suspect, is that the discriminator's job is quite specialized and requires it to be dynamic; i.e., it has to be able to "change its mind" quickly as the generator adapts. It's not clear to me that a pretrained network, which people have worked to place in a high-quality and robust part of weight space, is good for this. Furthermore, since a discriminator's job is a little easier than e.g. ImageNet classification I suspect that the massive deep networks often used for transfer learning are simply unnecessarily large for the task (the backward or even forward passes being unnecessarily costly, I mean; GANs already take enough time to train).
That being said, it is actually common to use pretrained networks for the discriminator for perceptual feature matching losses. (You may be familiar with the even more common "perceptual losses" commonly employed in autoencoders).
The idea is to match feature statistics, rather than optimize a scalar probability (this idea comes from Salimans et al, I believe).
Note that the pretrained networks used for this (usually VGG) are not trained or even fine-tuned most of the time though; they're usually left alone.
Some relevant papers: [1], [2], [3].
I suspect that you can always use a pretrained network in a setup similar to these or in the manner employed by McGAN, and it will be helpful.
Despite all this, I am fairly sure that using a pretrained network as a starting point will still work fine in many cases, though I'd suggest starting with a WGAN variant for it to work. It may even do very well, depending on the data and the other training details. Let me know if you try it out =)


How is Importance-Sampling Used in Off-Policy Monte Carlo Prediction? In the section, "Off-Policy Prediction via Importance Sampling", found in the chapter on monte carlo methods of the second edition of Sutton and Barto's, "Reinforcement Learning: An Introduction", the importance-sampling ratio $\rho_{t:T-1}$ of a sequence of action-state transitions $S_t, A_t, S_{t+1}, A_{t+1}, ..., A_{T-1}, S_T$ (where $T$ is the final time step, that is, $S_T$ is the terminal state of the episodic task) is defined as the ratio of the probability of that particular sequence of state-action transitions occurring under the target policy $\pi$ to the probability of that sequence occurring under the behavior policy $b$:

where $p$ is the state-transition probability. The author introduces the importance-sampling ratio as a means of obtaining the state-value function of the target policy $v_{\pi}(s)$ from an episode generated according to $b$, as explained in the following excerpt from the text.

My question is this: how exactly does multiplying the return under the behavior policy by the importance-sampling ratio in equation 5.4 produce the correct expectation for the state value function of the target policy? It is unclear to me how the ratio "transforms the returns to have the right expected value".

After some head-scratching, I think I was able to make sense of the "transformation" that the author uses in equation 5.4 to yield the correct expectation for $v_{\pi}(s)$. I've introduced some notation to help clarify what's going on. In particular, let $G_t^{(\pi)}$ denote the return after time step $t$ by following policy $\pi$. Then our goal is to estimate $v_{\pi}(s)$ for the target policy $\pi$ from an episode generated under behaviour policy $b$. Note that the state-value function for $b$ is defined as

$$v_b(s) = E[G_t^{(b)} | S_t = s]$$

We now apply the multiplication by the importance-sampling ratio proposed in equation 5.4 and expand using the definition of the expected value of  $G_t^{(b)}$ as follows.

$$E[\rho_{t:T-1}G_t^{(b)} | S_t = s] = \sum_{g \in supp(G_t^{(b)})} \rho_{t:T-1} \cdot p_{G_t^{(b)}}(g) \cdot g$$

However, note that the probability of a particular return occurring under $b$ is just the probability of its sequence of state-action transitions (called the "trajectory" of a return in the text) occurring under $b$. Thus $p_{G_t^{(b)}}(g)$ is simply the probability of a particular trajectory
occurring under $b$, as given in the following excerpt from the text for the example of an arbitrary policy $\pi$.

Recall $p$ here is the state-transition probability. We now substitute this expression for $p_{G_t^{(b)}}(g)$ into our expanded version of equation 5.4, and substitute the definition of $\rho_{t:T-1}$ to obtain:

$$E[\rho_{t:T-1}G_t^{(b)} | S_t = s] = \sum_{g \in supp(G_t^{(b)})} \left(\frac{\prod_{k=t}^{T-1}\pi(A_k|S_k) \cdot p(S_{k+1} | S_k, A_k)}{\prod_{k=t}^{T-1}b(A_k|S_k) \cdot p(S_{k+1} | S_k, A_k)} \cdot \prod_{k=t}^{T-1}[b(A_k|S_k) \cdot p(S_{k+1} | S_k, A_k)] \cdot g \right) $$

Canceling like terms gives us

$$= \sum_{g \in supp(G_t^{(b)})} \prod_{k=t}^{T-1}[\pi(A_k|S_k) \cdot p(S_{k+1} | S_k, A_k)] \cdot g$$

Note that, as discussed earlier, $\prod_{k=t}^{T-1}[\pi(A_k|S_k) \cdot p(S_{k+1} | S_k, A_k)]$ is the probability of a given return trajectory occurring under the target policy $\pi$. In other words, we have:

$$ = \sum_{g \in supp(G_t^{(b)})} p_{G_t^{(\pi)}}(g) \cdot g$$

The support of $G_t^{(\pi)}$ is the same as the support of $G_t^{(b)}$: the support of either random variable is the set of all state-action sequences (trajectories) possible after a particular time step t. We conclude that

$$E[\rho_{t:T-1}G_t^{(b)} | S_t = s] = \sum_{g \in supp(G_t^{(\pi)})} p_{G_t^{(\pi)}}(g) \cdot g = E[G_t^{(\pi)} | S_t = s] = v_{\pi}(s)$$


What's my target variable? I am beginner in data science. I have this "aids" dataset from "mdhglm" package in R.

dataset = aids, info = Repatead Measures on AIDS Data

data("aids", package = "mdhglm")

Here, I want to know what is my target variable?
I am sorry, if the question is too basic.

Obtaining a dataset is an important part of defining a ML problem but it's not the only one. Typically this involves the following steps:

Define the goal of the problem. Example: predict AZT level of tolerance among AIDS patients.
Obtain appropriate data for the problem.
Design the formal setting of the experiment:

what kind of problem is it (e.g. classification)
what is the target variable and what are the features in the data
how to evaluate the quality of the results (performance measure, e.g.f1-score)
Experimental setup: ML method(s), use of cross-validation etc.


Loading Data into DW: Direct Insert from PreProcessing vs PreProcess and then loading from CSV files I have a preprocessing script (google cloud function) that generates files (stored in Google Drive). I want to load those files in my DW (Big Query).

What are the pros and cons of:

1) Running the preprocessing script, generate the files and then loadthose files

vs

2)  Loading data directly from the preprocessing script (skip the file generation, just do a direct insert in the DW from the preprocessing script)

?

I am interested in focusing the question not only in terms of technical stuff and costs, but also in terms of data processing methodology.

I think the question could lead to the dilemma loading online or more in a batch process.

I have added some conclusions of mine as answers.
Still, it would be great to have more comments about the technical perspective on when to use direct transfer and when to use a file for staging preprocessing results.

Thanks!

Loading the data directly from the preprocessing script to the DW means not storing the results of your processing on storage.

Not storing those results on storage may imply:

Needing to do the processing again if for some reason the data is needed again, for a new DW or for a new research.
Not saving the data as a data sink and so not contributing to a data lake architecture, where data sinks are just saved for future reusing on "unpredicted" situations.

From a technical perspective, some tradeoffs are mentioned on the Google BigQuery docs on Streaming data, from which the following could be highlighted:

Consistency management, as for errors or duplicates.
Waiting time until data is available for copy and export operations

Still, it would be great to have more comments about the technical perspective on when to use direct transfer and when to use a file for staging preprocessing results.

Thanks


If I have to recommend 10 movies to the users Let's say I have some information about a user and movie data similar to the following:

user: age, gender, height, nationality, etc
movie : actor, genre , director info
userwatch : userA watch movieA at 2009,,.......
userComment : userA comments about movie

In this case, if I would like to recommend 10 movies to each user, which machine learning technique would I use?

I was thinking about making a data frame and running RandomForestClassifier, but in this case, it's hard to recommend exactly 10 movies to each user.

I was also thinking about using K-means clustering, but when the movie number is over 200000, i'm not sure k-means is the best choice.

Any help is greatly appreciated.

Use a recommender system.

I suggest you begin by reading an introduction to recommender systems.


Context classification problem I have a bunch of  articles about science from a certain website. When a new article is published, I want to determine if that article is really talking about science (and not politics for example). How can I do that? What machine learning technique shroud I use? I'm thinking at using something similar to spam detection. Is that ok? Thank you!

Spam detection can be done with many different methods, the same goes for your task. They do share the similar idea of processing a given text and classifying it to be one of 2 classes (science/not-science or spam/not-spam).

What you first need to do is to turn the articles into a vector of constant size
(for example with Word2vec which takes as its input a text and produces a vector space).

Ones you have a vector representing each article, you can start training your classifier and feature extractor (these days they are trained together).

As for determining which machine learning approach to take, you can try first using an SVM, it will probably be good enough.

You can follow one of the following tutorials (there are many more), just replace their dataset with yours :

Email Spam Filtering: An Implementation with Python and Scikit-learn

Spam Classifier in Python from scratch


Should I concat multiple stock timeseries datasets into one? I have several timeseries datasets of stock data, with fundamental indicators. I would like to build a model that selects stocks for buy and hold.
I understand that to perform this task I have two options:

Train a model for each stock: This way, I understand that it is the most practical, however, the amount of data for each model will be very reduced (Each dataset has less than 1000 lines).

Putting all the data together in a single dataset: I didn't find anything on the internet to support this idea, however, I understand that the model would be more robust and would have a much larger amount of data to be trained.

So, what would be the correct way to perform this type of analysis? Any of you would suggest another way?
Thannk you in advance!

I would suggest option two. Because this way your model would have the chance to learn something for one stock, which it can apply for other stocks as well. If you provide the type of stock as an input feature, it should be able to distinguish between the specialities which only occure within one stock and the common things. So it is kind of able to transfer knowledge from one stock to the other.
But at the same time, I would suggest to try out both and choose whatever performs the best. This way you can also justify your choice in the end.


Transform Categorical Variables into Numerical I'm very new to machine learning approaches. I'm reading a tutorial for build a predictive model using random forests.
One of the transformations implemented was transform categorical variables to binary.
Imagine (short sample):

Field_Desc  Field_Value
A               32
A               100
B               1

And then the developer pass this dataset into:

Field_A1    Field_B1    Field_Value
1               0                32
1               0                100
0               1                 1

What is the advantage to make this transformation for Random Forest Prediction? And for K-Means there will have any advantage?

Thanks!

Suppose that you want to have k-means algorithm, in the formulation of average, you have to take the average of each cluster, and then reassign the centers. If you have categorical data, how do you want to take mean? Changing categorical data to numeric data is for translating situations which don't have numerical features to be suited to be used for such algorithms.