Which is first ? Tuning the parameters or selecting the model I've been reading about how we split our data into 3 parts; generally, we use the validation set to help us tune the parameters and the test set to have an unbiased estimate on how well does our model perform and thus we can compare models based on the result of the test set. However, i've also read that model selection shoud be  done before tuning the parameters. I'm getting confused. Which one must be done before the other ? Is the validation set used for tuning ? If true, how are we supposed to do model selection before tuning the parameters ?

You can tune parameters only if you have already trained the model, otherwise there is nothing to tune.

  However, i've also read that model selection shoud be done before tuning the parameters.

Before tuning you need to do some kind of pre-processing before tuning the parameters.
Usually your pipeline will consist of:

Get Data and Clean It.
Do some EDA ( Exploritary Data Analysis) do get better understanding of you data.
Do some feature engineering - roughly, if possible, you would like to transform current data, so it would be more suitable for your ML algo.
Train and evaluate your model on validation data.
Tune your model`s params to get better performance ( fight with overfitting alos included).
Evaluate on Test set

  how are we supposed to do model selection before tuning the parameters

About Model Selection - I think even without having deep inshights about your data, you can already choose some baseline models (I think its a good rule to start with simple models like Linear Regression or K-NN to get the "feel of the performance"), because in most cases you know that you are probably dealing with regression/classification/clustering task, so you can already specify the set on models to try - but before trying the horde of ML models, you should get your data ready.

Some posts like this one can give you the feels of available ml algos, but there are a lot more to try.

In Addition, I think this paper might be very usefull if you are starting to work with ML


Bootstrap clarification for datasets with multiple columns How do I bootstrap a Dataset/DataFrame with multiple continuous and categorical columns?
For eg: Say I'm trying to bootstrap the colour distribution of M&M's and I have 50 bags (samples) each with recorded counts of 6 colours, but the samples/bags randomly come from two factories, and have 2 different time stamps on the sample/bag packets.
What would be a good method of bootstrapping this kind of dataset?
Using loops that resample slices by categorical values? eg: Split the dataset into serial_no groups E2 and E1, resample those with replacement and combine, then do the same for serial_time etc...
Or can the whole dataset be resampled at once?
If there happens to be subtle patterns within 'E2' data for eg, how is that preserved in bootstrapping?

# create dataframes for serial_no columns.
E1_bs = pd.DataFrame()
E2_bs = pd.DataFrame()

# proportinal sample with replacement
E1_df = df[df['serial_no'] == 'E2']
sample_size = len(E1_df)*10
for i in range(len(E1_df)):
    E1_swr = pd.DataFrame(E2_df.sample(sample_size, replace=True))
    E1_bs = pd.concat([E1_bs,E1_swr],axis=0)

E2_df = df[df[['serial_no'] == 'E1']
sample_size = len(E2_df)*10
for i in range(len(E2_df)):
    E2_swr = pd.DataFrame(E2_df.sample(sample_size, replace=True))
    E2_bs = pd.concat([E2_bs,E2_swr],axis=0)

bootstrap = pd.concat([E2_bs,E1_bs],axis=0)

If I do this for all of the categorical columns (serial_no and serial_time in this case) and add them together as one dataset, I get different means (significantly) compared to if I just sample the whole thing at once using the same total amount of samples).
# Compared to bootstrapping the whole thing
sample_with_replacement = pd.DataFrame(df.sample(len(bootstrap), replace=True))

Here you have an example on how to carry out a bootstrap selection from any given set of data, based on (as you suggested) using loops and randomly selecting indexes to recreate a simulated new bag from the whole population of M&Ms. The criteria should be pretty the same for different kind of data, since what you actually do is simulate the selection of samples from a population.
It could be something like:
-bags you have from different factories:
bag_1 = ['blue']*15 + ['red']*15 + ['green']*10 + ['orange']10
bag_2 = ['blue']*14 + ['red']*16 + ['green']*9 + ['orange']*11
bag_3 = ['blue']*16 + ['red']*15 + ['green']*11 + ['orange']*8

-color distributions per bag:
import seaborn as sns
import pandas as pd

bags_df = pd.DataFrame({'bag': [1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3],
                     'color': ['blue','red', 'green', 'orange', 'blue','red',
                     'green', 'orange', 'blue','red', 'green', 'orange'],
                     'count': [15, 15, 10, 10, 14, 16, 9, 11, 16, 15, 11, 8]})

ax = sns.barplot(x="bag", y="count", hue="color", data=bags_df)

-now your population would be all the bags together inb a big bag:
super_bag = bag_1+bag_2+bag_3

-and here is where you do the actual bootstrap:
### GENERATE 10 BOOTSTRAPED BAGS
from numpy.random import seed, rand, randint, randn

# seed random number generator
seed(1)

bootstrap_bags_dict = {}
for i in range(10):
    bootstrap_bags_dict['sub_bag_'+str(i)] = []
    # bootstrap sample
    boots_indices = randint(0, 50, 25)
    for idx in boots_indices:
        bootstrap_bags_dict['sub_bag_'+str(i)].append(super_bag[idx])

and you get a dictionary with 10 bags like:

In case you want to see if there is any difference in, for instace, the red color distribution coming from factory 1 or 2, I would:

split the dataset by factory1 and factory2, get the red M&Ms count for each one
then shuffle all the M&Ms in a big bag as coming from the same factory; here, your initial hypothesis is that there is no difference in receiving red M&Ms from both factories
as explained here , you can compare by permutation and hypothesis testing if the difference between bootstrapped samples from your big bag (as considering both bags from the same factory) is statistically different compared to the difference between the 2 initial factories bags (it might sound hard to know, some code about it can be provided)


Collaborative Social Network Visualization I do movement building work for Effective Altruism (http://en.m.wikipedia.org/wiki/Effective_altruism), and would like to level up our growth strategy. It occurred to me that a social network visualization tool which allowed us to strategically find and recruit new influencers/donors would be mega useful. I'd love to find something (preferably free), similar to InMaps, which would allow us to:

Combine all of our social media connections into a single map
Easily see who the superconnectors are
Weight each person by their degree of social influence (perhaps some function of things like Klout score * amount of social media connections * number of Google mentions, etc)

Does such a thing exist? If not, is anyone interested in pro bono work for an amazing cause? =)

Disclaimer: I am a data science noob, so preferably the solution would be one with a nice GUI and minimal involvement of R or Python.

I think Gephi, an open-source visualization tool, would help you a lot. Actually, as I know, the InMaps and its community detection algorithm are same as the Gephi's.


Can you sum up gradients and apply in batch? I'm following this Reinforcement Learning Tutorial. There, training data is collected during an episode. When the episode is done, the data is used to do backpropagation. However, instead of applying the calculated gradients to the model, they are stored in a buffer. The gradients of multiple episodes get summed up before they are applied.

I wonder if that is a reasonable thing to do. Usually, if the model is very good after training with the first episodes data, the gradients of the other episodes will be close to 0 (no change). If you used the tutorials code, the gradients of all episodes would be large, and the model would move away from the optimum. Am I missing something?

Edit: I tried buffering gradients with Q-Learning and compared it to directly updating the weights. It is working, but buffering is definitely worse.

Reading the relevant paragrpah, we can gain a little more insight. It seems to come down to the idea of delayed reward. By only updating the model (i.e. the agent) every once in a while, we are inherently inducing the agent to make moves which are benficial over longer periods of time.

Here is the paragraphs from the post:

  Delayed reward — Keeping the pole in the air as long as possible means moving in ways that will be advantageous for both the present
  and the future. To accomplish this we will adjust the reward value for
  each observation-action pair using a function that weighs actions over
  time.

  To take reward over time into account, the form of Policy Gradient we
  used in the previous tutorials will need a few adjustments. The first
  of which is that we now need to update our agent with more than one
  experience at a time. To accomplish this, we will collect experiences
  in a buffer, and then occasionally use them to update the agent all at
  once. These sequences of experience are sometimes referred to as
  rollouts, or experience traces. We can’t just apply these rollouts by
  themselves however, we will need to ensure that the rewards are
  properly adjusted by a discount factor

  Intuitively this allows each action to be a little bit responsible for
  not only the immediate reward, but all the rewards that followed. We
  now use this modified reward as an estimation of the advantage in our
  loss equation. With those changes, we are ready to solve CartPole!

The agent might otherwise want to just do the optimal move to keep the pole upright for the next milllisecond, but with this buffer, we are essentially only allowing the agent to consider larger chunks of time.
This will take longer to train, but should in theory (well, the assumption behind these rollouts...) lend itself to creating a more thoughtful agent.

In other contexts, such as a standard CNN used to classify images, this approach would make less sense as there isn't a temporal dimension to the task.

Saving up the errors and making one big update can have several other explanations in general:

I can save compute, as we could formulate the graph to then only perform backprop after our buffer is full. For buffer size not close to 1, this would potentially halve the training time (making the gross simplication of an assumption that learning occurs at the same rate)
It may be equivalent to using batches, in that it could help smooth the learning process. If you have a batch size of 1 sample, the error that is measures and used to update weights may be quite erratic. Using a batch has a nice averaging/smoothin effect. Saving up errors in a buffer, one could argue, may have a similar effect.

I should add, I haven't personally seen this in practice outside of reinforcement learning.


Representation options of strings (keywords/topics) in models What are all the possible ways to represent keywords in a machine learning model?

The two I am aware of are:

one hot encoding, using a static index.
vector representation, using
an embedding layer.

We have a specific problem where we are doing client side (browser) ml and need to convert text data into something the model can consume without sending it over to the server.

EDIT: (comment clarification)
The text data is extracted from the page on which our script loads, we then want to run a model locally on the browser - using text based features from the page (ideally). We are minimising in all elements any data needed to sent to the server.

In terms of model, that is not defined at this stage, also this question primary concerned with representation of text based features.

Since (word-based) one-hot encoding and real-valued vector representations are already mentioned in the question, I would only add the n-gram representation, especially the character-based n-gram representation.

For word-based n-gram representations you consider not individual words, but their ordered combinations in the text and use the one-hot encoding for the combinations. E.g. for n=2 you might end up with the bigrams ["John likes", "likes to", "to watch", "watch movies"] and each of them would be assigned to some dimension using a static index.

This also works with characters, so you can represent the word "encoding" e.g. with those 3-grams: ["enc", "nco", "cod", "odi", "din", "ing"]. The one-hot encodings of n-grams are typically added, so multiple occurances of the same n-gram are recognizable in the resulting Bag-of-n-grams representation. This kind of representation is especially useful for languages with rich morphology and/or compound words. In a one-hot representation each single word form would be encoded in its own dimension whereas a character n-gram approach helps preserve similarity between different forms. An example in the English language would be the similarity between "encode", "encoded" and "encoding" which would stay preserved this way. Similar techniques are also used by some word embedding algorithms which consider subword information like e.g. FastText.

Also, although it's not directly an encoding, but depending on your use case and language it might be worth looking at different preprocessing options like lemmatization and stemming where you reduce different word forms to their base form. This would also affect the choice of representation, e.g. the word-based one-hot encoding might make more sense if you choose to use these preprocessing techniques.


What data treatment/transformation should be applied if there are a lot of outliers and features lack normal distribution? I am solving for a regression use case using tensorflow's DNNRegressor. For EDA purpose, I referred to this post and used pandas boxplot to plot my numerical predictors and target variable(here, pid demand) and scatter_matrix for plotting the distributions and here are the results :
predictor_target_boxplot ; features_label_pdf_scatter_matrix
.

I need help in interpreting these two plots, specifically on these fronts:

How come the boxplot shows so many points beyond whiskers (~10%), can there be so many outliers in a dataset?
How do I handle those outliers?
Based on the second plot (feature, label pdf), should I normalize my features to exhibit Gaussian distribution? If so, why?

This comments aren't all mine, I have asked on a slack forum,

boxplot is shouting at you:  skewness, and also high dispersion.  Can not ask much more to a boxplot than location, dispersion and skewness..

Also checkout this term heteroscedasticity(completely suits your case)

Try switching the transformation to a log plot or lower..

Also your eda can't be boxplots dependent as prices are involved here..one of the remedy would be doing box-cox transformation

https://www.differencebetween.com/difference-between-dispersion-and-vs-skewness/

http://www.statsmakemecry.com/smmctheblog/confusing-stats-terms-explained-heteroscedasticity-heteroske.html

To have a look at it,
https://datascienceplus.com/how-to-detect-heteroscedasticity-and-rectify-it/


Reducing MAE or RMSE of linear regression I am trying to guess a home price and at the end I intend to figure out a formula by using linear regression. As you can see here, I have 1480 samples with 45 features in which price(fiyat) is the target variable.
Do higher values for $RMSE$ and $MAE$ mean that the dataset cannot be trained in a good manner?
Is there a way to reduce these values?
As you can see $R2$ seems well.

How can we say that how much percentage of error occurs for the guesses on average? If we can how much could you say?
There is really substantial difference between the prices and guesses as being seen below:

You can try things like outlier removal, reducing data skewness, or stacking different models. For an advanced regression guide, checkout this kernel from Kaggle.


Discovering non-interesting attributes I would like to ask a question about recommender systems. We are showing some movies to users and they have to decide if they like them or not. These movies have only a few attributes

Title
Director
Category
Duration
Nationality

We show the attributes as a block. The users can decide if they like (the block) or not. We aggregate these decisions to recommend them more movies in the future. My question is: Are there any methods that can tell us whether a user is uninterested in a given attribute? For example, if a user is never considering duration when saying yes or no.

As a simple measure, you could simply take the point biserial correlation between duration and rating for a given user.  If they aren't correlated (positively or negatively), this variable is likely not important to that user.

Another way to determine the impact of a variable is by training another model excluding that variable and determining how that affects the quality of the predictions.


difference between feature interactions and confounding variables Let me define the problem space.

I am working a binary classification problem. I am trying to build a causal model as well as predictive model.

My aim is to find list of significant features (based on causal model) and use that to build a predictive model. I did refer the suggestions provided in this post and it was very much useful but I have few more questions due to my limitations with ML field.

I understood from literature that there are two ways to adjust/control for confounders. one is through study design phase and other is during modelling/analysis phase.

As I am working on retrospective data analysis, I can only adjust confounders during analysis phase.

We know that certain features like Age in a typical example like "gender causes heart disease" is a confounder.

1) So during analysis phase, we include age as a variable in our model. Similarly all the potential confounders that we could think of are put in the model as features. ex: X_train will have all the columns/features that I think of as potential confounders and then it is fed to the model (logistic regression). Am I right till here?

2) Does this mean our LR model is adjusted for confounders? How would you do confounding adjustment during logistic regression modelling phase? If we include all potential confounders in our model and if the coeff of already existing variables (gender) change by 10% or so, I understand that age is a confounder but does this also mean that LR is adjusted for confounders?

3) Then, why it is said that logistic regression doesn't consider feature interaction? Is feature interaction different from confounding? I understand that feature interaction is usually denoted as gender*age but does this mean both variables work together to influence the outcome? doesn't confounder mean the same?

4) What's the usefulness of having interaction variables? I mean if gender*age impacts the outcome, can I understand that gender (individually) and age(individually) impact the outcome?

5) I see that people usually create 2x2 tables called as strata for stratified analysis and compute risk ratio and compare it with crude risk ratio. But how can we do this for all variables which I think as confounders in my dataset? I know we can use tools like SPSS, STATA etc but is it the only way to do? But then can't we do using multivariate regression?

6) Is it mandatory that all our continuous variables be converted into some categorical variable for analysis/confounder adjustment?

7) Any simple examples/explanation would be helpful as I couldn't find any tutorial for adjusting confounding during logistic regression and finding significant variables. I have been referring this though it's useful, some links are broken. lot of questions arise because I am neither a stats or biostats person. I usually build models using classic ML algorithms and now trying to learn all this.

Some remarks (as far as I understand your questions):

In a causal model you need to reflect the "data generating process" (DGP). The DGP is a theoretical construct. You need to come up with an idea what is relevant for your research question, so what $X$ explain $y$ in a causal way. You can also include variables which are not so important. In tendence underspecification (exluding important variables) is a real problem. Overspecification can also be a problem but the consequences are less salient.
I think the wording is unclear here. In some fields "adjusting for confounders" means to "control" for all relevant variables (aka. confounders). First thing to do: get the wording right and understand what you mean when you say "confounders".
Where does this claim come from ("feature interaction"). In general, Logit is a model that is linear in the way you include variables, so if you think about interaction of two or more $x$ in your model, then no: Logit does not consider interaction unless you specify your model so to include interaction terms.
An easy way of thingking about interactions is the interaction of a continuous variable $x_1$ with a dummy variable (1 or 0) $x_2$. In the model $$y=\beta_0+\beta_1 x_1$$ you only have one intercept ($\beta_0$) and one slope ($\beta_1$). In a model with interaction between $x_1$ and $x_2$ you have $$ y=\beta_0+\beta_1 x_1 + \beta_2 x_2$$ where there is the same slope for $x_1$ but a different intercept in case $x_2=1$. Another way to think about interactions is to add "squared" terms because a squared term is just the interaction $x_1 * x_1$.
I don't understand this: needs a reference! I guess you speak about the inclusion of indicator variables (aka dummies) as interactions??!
No you can use any numbers as $x$.
Look here and read this book.


Standardization After PCA for Kmean clustering I want to apply Kmean for clustering after PCA dimensionality reduction. I have standardized data with StandardScaler before the PCA, then I want to train Kmeans for finding clusters. However, the variance among the PCA components could not be of the same order of magnitude.

It is a good practice to standardize the PCA components before the clustering?

Usually PCA already returns standardized components.

Did you compute the variance of each component? Usually, it will be 1.

The more tricky question is whether to use standardization before doing PCA. I don't think there is a general answer for that.


Linear Regression - finding thetha using Normal equation This is to find thetha which will give minimum cost function. Why is the x0 column required? why cant we assign size as x0? why do we need the feature count to be n+1?

Consider the 1-D equivalent of the table you have provided. In this case, you have one input x and one output y (see figure). If you try to fit the data points with y=mx equation, you can not fit the data successfully. You need an equation like y=mx+c to have a good fit. y=mx will be a good fit only if the line goes through the origin.
Now you can look at

y=mx+c

as

y = m * x + c * 1
y = m * X1 + c * X0
y = W1 * X1 + W0 * X0

I hope the figures below will clarify why we need the intercept term X0 and its weight W0=c.


outlier detection in time serie without using windows I would like to know if it's possible to detect outliers in a time-serie with an outlier score computed given the whole dataset and not given windows

You can try to remove trend and seasonality and check whether the time series is stationary. If it is, you can calculate mean ($\mu$) and standard deviation ($\sigma$). Now for each data point, you calculate the distance from the mean ($\|y-\mu\|$) and check if it exceeds some threshold. The default value for threshold is $3 \sigma$ what comes from three-sigma rule motivated by the assumption about the normality of the data (residuals after time series decomposition in our case). You can choose any threshold which is reasonable for you. One solution is to find the distribution of the data and derive a rule for this particular distribution.


Different accuracy for different rng values While tuning the SVM classification model in Matlab, I came across the rng function in matlab in which seed (stabilizes the random shuffling of the data in the algorithm) is changed. When the function called is rng(1) then I am getting one accuracy value (99%). When it is changed to rng(2) then I am getting another value (57%). So there is a huge change in accuracy as visible. What does this mean? Am I training it wrong?

The train and test set correct rate (in %) that I am getting with different runs without changing rng are(train,test)
(96, 82.8)
(94.6, 95.3)
(96, 85.9)
(96, 90)
(95, 95)

The training errors in this dataset has a huge difference (99% vs 57%). So, maybe the one with the rng(1) split has overfitted your dataset.

  So there is a huge change in accuracy as visible. What does this mean?
  Am I training it wrong?

The huge change might be due to overfitting. (Also, judge the model through validation curves, and then fit a model which balances the bias-variance plot.)


Difference between convolution structures I am having a hard time understanding the difference what is a multichannel CNN:
In the paper titled, "A Multichannel 2D Convolutional Neural Network Model for Task-Evoked fMRI Data Classification" (https://www.hindawi.com/journals/cin/2019/5065214/#sec2.1) Figure 1 is a multichannel CNN where the authors are using a channel to represent an input image type. Image is a 2D grayscale matrix. According to the diagram, it appears the number of images = number of channels = number of different CNNs. Is that true? I was under the impression that irrespective of the number of channels, we have one CNN that applies filters to each input kind separately. However, this is not the same as having multiple CNNs for each input type.
I am confused -- does multi channel CNN mean that there is a separate CNN for each input data Or a single CNN that applies filters to each input separately.

In absolute, it is one CNN wich takes 3 inputs images. You could see it as 3 separate features extractors (CNN) which merge their results while trained together.
The author obtain 3 2D input from a 3D images by keeping 3 2D images; one in each plane.
Each of these images has multiple channel because they slices the input among the respective axis.

It is called a "multichannel" CNN because it uses differente representation (3 2D images) of the input (3D image) and therefore extract 3 times the features in order to merge then. (like looking an elephant from each side and above)


How to find a changing relation between continuous variables? I have a dataset with 8 continuous variables describing the behavior of a machine:
- input a,b,c for both left and right: a_left, b_left, c_left, a_right, b_right, c_right
- output o, non-linearly related to the inputs (and almost certainly also influenced by other factors)
- timestamp t
I have some environmental data (e.g. temperature) as well but these are fairly constant and I suspect not of significant influence.

Theory:
My theory is that due to performance degradation of the machine due to fouling, the value of o will be lower for the same input values.

Occasionally, the machine is cleaned to restore performance again. I do not know when and how well (i.e. to a "perfect" state vs. to some intermediate state between "perfect" and the previous state). The rate of degradation is also not known and most likely not constant.

Furthermore, a change is one of the input variables has a delayed effect on the output due to inertia.

Actions I have tried:
- plotting the data for some different relations between the input and output
- k-means clustering for some different relations between input and output (without timestamps)

Questions:
- Is it at all possible to verify my theory with this data?
- If so, what techniques are most suited to be applied? I'd prefer anything in Matlab.

Besides doing initial EDA and plotting dependent vs independent variables, it would seem prudent to try to first identify the "maintenance" points. According to your description of the problem, we can expect to see a decline in output o punctuated by single spike-step indicating improved performance. In other words you should in general see a negative slope interspersed with very high positive slope. We see a similar type of spike behavior in EEG signal, where the "blips" or "spikes" correspond to unwanted artifacts (e.g., eye blinks). These can be identified relatively easily by running a sliding window over the data and looking for big changes in slope. The time segments marked in green below are identified as artifacts by monitoring for changes in slope. You can look into EEGlab and ERPlab -- which are Matlab based -- to find out more about this approach.

For feature engineering, because of the input-to-response lag, you will want to initially include variable lags. For example, for variable a_left and a corresponding change in output o, you would want to add as features a_left_1, a_left_2, ... a_left_n, where each feature refers to the input value at some lagged time prior to the observed change in behavior.


How to set and check normal distribution on a data set? Sorry if my question is simple, I have a data set with two class and want to check and set normal distribution on it(if it was necessary, in MATLAB).

But the question is that I should use it on every rows or every column, it's 200 rows and 30 columns.

It's hard to be sure of anything without having a glimpse at your data but what I assume from what you are saying is:

you have 2 classes which probably are  given as 1 columns

you have 29 features

you have 200 samples
Edit: Then while writing my answer I'm not too sure what you wanna do. If you have 2 classes it is likely that you are about to do some classification tasks so I don't really get what you want to modelize with a normal distribution.


Linear discriminant analysis in R: how to choose the most suitable model? The data set vaso in the robustbase library summarizes the vasoconstriction (or not) of subjects’ fingers along with their breathing volumes and rates.
> head(vaso)
 Volume  Rate Y
1   3.70 0.825 1
2   3.50 1.090 1
3   1.25 2.500 1
4   0.75 1.500 1
5   0.80 3.200 1
6   0.70 3.500 1

I want to perform a linear discriminant analysis in R to see how well these distinguish between the two
groups.
And I consider two cases:
ld <- lda(Y ~ ., data=vaso)

ld1 <- lda(Y ~ log(Volume)+log(Rate), data=vaso)

Please help me understand which model is better? What characteristics to look at?

I'm not familiar with LDA, but as far as I know you're not really changing the "model" (i.e. the way to measure impact) between the two versions, what you're changing is the features: in the 2nd version, instead of looking at whether the value of the feature impacts Y, you look at whether the log of the value of the feature impacts Y.
The first version is of course the most natural way to look at features, the second is common but usually this is used when we already know that the distribution of the feature (or the relation between the feature and the response variable) makes it relevant.


How to combine human-labelled data with user behavior data? I am working on a supervised learning problem for a web-search task, where I have access to a relatively small set of human-labeled examples and lots of user-behavior data.
Now, user behavior data is biased, because of presentation bias, position bias etc. So it's likely that its' distribution will be different from human-labeled data.
I am planning to use both to train a Neural Network model.
Now I am confused about how to combine both datasets?

That is a common scenario in a learning to rank problem. One heuristic is to separately model explicit (human-labeled) and implicit (user-behavior) features. Then combine the separate feature groups with a learned weight for their final relative contribution. Improving Web Search Ranking by Incorporating User Behavior Information by Agichtein et al goes into greater detail.
RankNet takes this approach using a neural network.


Keras Model question for Pre trained model extension I want to add a few more layers to a Resnet50 model and my question is - do I need to compile it and train it on new data or can I just use it as it is? Will it just give me the Resnet50 results?

Here is what I am trying:

def base_model():
resnet = resnet50.ResNet50(weights="imagenet", include_top=False)
x = resnet.output
x = Conv2D(128, (3, 3), activation='relu',padding='same')(x)
x = Conv2D(128, (3, 3), activation='relu',padding='same')(x)
x = MaxPooling2D((2, 2), strides=(2, 2))(x)
x = Conv2D(256, (3, 3), activation='relu',padding='same')(x)
x = Conv2D(256, (3, 3), activation='relu',padding='same')(x)
x = MaxPooling2D((2, 2), strides=(2, 2))(x)
x = Conv2D(512, (3, 3), activation='relu',padding='same')(x)
x = Conv2D(512, (3, 3), activation='relu',padding='same')(x)
x = MaxPooling2D((2, 2), strides=(2, 2))(x)
x = GlobalAveragePooling2D()(x)
x = Dense(4096, activation='relu')(x)
x = Dropout(0.6)(x)
x = Dense(4096, activation='relu')(x)
x = Dropout(0.6)(x)
x = Lambda(lambda  x_: K.l2_normalize(x,axis=1))(x)
return Model(inputs=resnet.input, outputs=x)

And then use it like this :

enhanced_resnet = base_model()
img = image.load_img(file, target_size=(224, 224))
img = image.img_to_array(img)
x = resnet50.preprocess_input(img)
x = np.array([x])
feature = enhanced_resnet.predict(x)

What I want as return value is the features from the image and not a prediction, as I am using a distance equation to tell the similarity between images.

Short answer - Yes.

The pretrained Resnet model has trained parameters (weights and biases). When you add extra layers on top of it, the parameters of these layers are randomly initialized. Therefore the final model will give random results.

You need to train the new model after adding extra layers on your new dataset. You don't need to train all the previous layers, just few of them.

For further reading search for - transfer learning. That's what you're trying to do.


Implementing Batch normalisation in Neural network I have implemented my own mini neural network program1.
Currently, it does not have batch updates, it only updates the parameters by simple backpropagation using SGD after each forward pass. I was trying to implement batch updates and batch normalisation2.

1) For simple batch updates, instead of updating parameters each time, for each image of the batch size of 'n' I should backpropagate and accumulate the deltas for all the parameters and finally update them once after the end of the batch.

2)For batch normalisation (BN), I went through the paper and I am sort of the clear with the idea but I am confused regarding how to implement it. Generally, I would multiply the matrices in the net one after the other for a single image to get final input, but with BN, do I need to feed forward for all the images in the batch till the first layer, then normalise the values, then fwd pass these values till second layer, then normalise again, and so on?
Once I reach the final layer, should I backpropagate the error for the corresponding input-output pair and update the parameters immediately as fwd pass for all the images in the batch has been done already?

Going by the way I have described, it seems to require a lot of parameter tracking throughout the batch.
It will be helpful if you can point out a better way to do it or anything that I have misunderstood so far.

Give the whole batch as an input, take the activated output of the first layer and pipe it the next until the final. .

calculate the gradient decent direction (derivative of the cost function) for each row. U will have the error for each row of ur training set.

Then add the errors and divide the result with number of the training set. Then finally update it weights.
I know from experience that tracking all these numbers is hard.

I don't know what language u use, but if it's Python, numpy will help u a a lot. For example, to add the errors, errors.sum(axis=0). X.dot(weight) to multiply ur entire dataset with ur weight.  So my suggestion is use some matrix library.

One thing I want to mention is batch processing is susceptible to local minima which u don't want...especially in complex functions. SDG is a nice algorithm. I'm not sure why u needed batch update. There are many interesting tweaks in NN, don't waste ur time on batch update, use SDG


Stopwords for programming languages (for, while, print,...) I am trying to do some analysis about user behaviour when typing (keystroke biometrics). Ideally, it will include traits extracted when people are writing code. Although not technically Natural Language, code also has some structured characteristics as language and I wanted to leverage that. I was wondering if there has been some research about performing language analysis focusing on programming languages instead of traditional spoken languages.

Mainly, I am interested in having a comprehensive list of stopwords for as many languages as possible. For example, stopwords will include: for, while, return, break, string, if, else, and so on. Although it would be nice to have them separated by languages, I wouldn't mind a list comprising several languages.

I know this could be done for example by getting some sample code and retrieving the most frequent terms, but I also wanted to know if there has been some research towards this direction.

Any ideas, papers, methods would be welcome.

Thanks!

Stop words are common words in a language and they are usually removed when there appearance is not indicative to the analysis goal.

Suppose our goal is to text mine and find out if a given text is about sports or politics. If we use a bag of words, "not" is probably a stop word we should remove since it is probably not indicative of either categories.

On the other hand, if we would like to differ between "This article will not discuss politics" and "this article will discuss politics", we must not remove "not".

The examples you gave look like programming languages reserved words. In most languages you can find this documentation and skip the learning phase. Looking at the frequent tokens will probably also give you the reserved words (and probably plenty of English stop words that will appear in the remarks).

However, before utilising such a list, be sure that neglecting them will serve your goal. In case that typing them doesn't differ from typing other words, you might better leave them. This way you will have more data the represent better the user behaviour.


Detect Objects on a table is it possible to train a model which detects and draws bounding boxes for objects on a table, when I use a dataset where objects on a table are labeled with boundingboxes?

Yes, it is possible. Provided the dataset contains the objects that you actually want your model to detect during run time.


Dealing with foreign film titles on IMDb I've managed to scrape the data from the Cinemetrics website, but as it is user-submitted, some of the data needed cleaning. I've been running the titles through the search function of the OMDb API, however this struggles with certain results.

For example, the film Meghe Dhaka Tara. When searching with this as the query on the OMDb API I get no results. However a quick Google search, or even IMDb search, will find the film with the English title The Cloud-Capped Star.

I was wondering if anybody else has ever run into a similar problem and how they go about working around it?

"Meghe Dhaka Tara" is a noun phrase. It makes sense for an API looking for movies to not translate the phrase as that is not its purpose. I suspect, that at most it will have some basic spell checks in place (for instance "did you mean `Megh Dhaka Tara'")

If "The Cloud-Capped Star" is indeed an english dub for the movie, then it suggests that the service provider has not linked movies to its translations.


How do different models take in account independence/dependence of features? How do different models take in account independence/dependence of features?

I've only found very rough "pointers" as to what assumptions different models make about the independence/dependence of features, but I've not found nothing holistic yet.

It depends on the different families of the learning algorithms. For example, Naive Bayes and all the semi-Naive Bayes models assume that the features are indpendent, which is not true in most real world problems. The latter is a disadvantage of Naive Bayes learning algorithm, although in some problems as in natural language process these models usually perform well. Most of the machine learning algorithms do not assume such thing (feature independence). Finally, there is no holistic rule for that.


Machine learning models for predicting good machine settings given bad machine settings I’ve a dataset about a machine that produces a product. Salient features about the dataset:

•   For each data point, some of the features are machine settings (calibration factor, grind time, pressure, water temperature) used to make the product, while few others are stochastic and external in nature, such as water hardness.

•   Output says whether the product produced was of good or bad quality.

Goal:

•   When I’ve a new data point which says that the product was of bad quality, I want to feed in the corresponding features(machine settings + stochastic external features) to an ML model and predict machine settings which will lead to good quality product. Idea is to have an ML based smart controller which would adjust the machine settings to reduce faulty products being produced.

Which ML models would suit this purpose?

Can you explain the problem more precisely? What are inputs (besides control settings), is output same if the inputs are fixed? Are there any stochastic variables(external factors prone to probabilistic behaviour), what is the desired outcome?( Minimizing faulty outcome by choosing a setting or something similar)


What is the best practice for tuning hyperparameters using validation data? I'm building a binary classifier, using task-transfer from resnet and a total training set of 300 images.
Initially I put aside 100 images as validation, and tuned the hyperparameters, each time training on 200 and testing on 100, until I got a validation accuracy of 93%.
Happy with this accuracy I tried the same parameters on the test set (another 170 images) and got really bad accuracy (around 65%).
What did I do wrong?

Should I have used cross validation?
What is the best practice here? How should I got about tuning my hyperparameters?
Can I repeat my process and check on the test set again? If so, how many times can I do this before it's "cheating"

If you're seeing performance that is much better on the validation than the unseen test data, then that is suggestive of some sort of overfitting or, if not, that the data do not come from the same distribution. That could mean that your test images are very different from the training and validation data, for example.
First, I'd double check the data to make sure the train, validation and test sets are definitely distinct, and that all three sets have roughly the same number of positive and negative examples. If this isn't obviously the problem, then most likely I'd guess that you are overfitting the hyperparameters to the validation dataset.
Using cross-validation would most likely help you to see whether this is the case, as you'd see more dramatic variation between each fold if the parameters were overfit towards one specific validation fold.
If you repeat the process on the test dataset, then that data is no longer "unseen" and is part of the model, even if you just use it to tune hyperparameters. You should keep the test dataset completely separate from the model building process so that when you come to measuring model performance, you get a true estimation of what would happen on unseen data.


How to use inverse_transform in MinMaxScaler for pred answer in a matrix I am working on a data, for preding output, I used SVR by bellow code:
from sklearn.svm import SVR
regressor = SVR(kernel = 'linear')
regressor.fit(trainX,trainY)

from sklearn.metrics import r2_score
pred = regressor.predict(testX)
print(pred)

The answer is : [0.58439621 0.58439621 0.58439621 ... 0.81262134 0.81262134 0.81262134]. I'm trying to inverse the scaling to real amount.
I search it in StackOverflow and reach this: https://stackoverflow.com/questions/49330195/how-to-use-inverse-transform-in-minmaxscaler-for-a-column-in-a-matrix, I implement every 2 answers in my code, but I get error yet. Can anyone help me with this?
I write this from above source:
import sklearn
from sklearn.preprocessing import MinMaxScaler
scale=sklearn.preprocessing.MinMaxScaler()
scale.min_,scale.scale_=scaler.min_[0],scaler.scale_[0]
scale.inverse_transform(pred)

but, I got same error as:

Blockquote
Expected 2D array, got 1D array instead:
array=[0.58439621 0.58439621 0.58439621 ... 0.81262134 0.81262134 0.81262134].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.
Blockquote

From my understanding you are working on a regression task in which you have applied MainMaxScaler to your target variable y prior modeling.
If so you have two options:

As the error message suggests, you can reshape the output with array.reshape(-1, 1)

Scikit learn has implemented a class to work with transformations on target:

So just try
from sklearn.svm import SVR
from sklearn.compose import TransformedTargetRegressor
from sklearn.metrics import r2_score
from sklearn.preprocessing import MinMaxScaler

regressor = SVR(kernel = 'linear')

model = TransformedTargetRegressor(regressor= regressor,
                                        transformer = MinMaxScaler()
                                        ).fit(trainX,trainY)

pred = model.predict(testX)
print(pred)

Every time you make a model.predict(X) it will apply an inverse transformation so that your predictions are in the same scale as prior MinMaxScaler
EDIT:
Working example of transformation without using Scikit-learn
# array example is between 0 and 1
array = np.array([0.58439621, 0.81262134, 0.231262134, 0.191])
#scaled from 100 to 250
minimo = 100
maximo = 250
array * minimo + (maximo - minimo)

Returns:
array([208.439621 , 231.262134 , 173.1262134, 169.1])


What are the points called in a dataset where the points have the same features but different labels What are the points called in a dataset where the points have the same features but different labels?

For example, I am trying to predict whether some object is a hot dog. But I only have access to features is_wrapped_in_bread and has_meat_in_the_middle.

| object        | is_wrapped_in_bread | has_meat_in_the_middle | is_hotdog |
| salad         | false               | false                  | false     |
| hotdog        | true                | true                   | true      |
| burger        | true                | true                   | false     |
| veggie burger | true                | false                  | false     |

In the example above, burger and hotdog are not distinguishable based on the available features. Is there a name for that?

I'm not aware of any specific term for this but in general I would call this either:

noisy data, typically in the case where these inconsistencies are due to mistakes in the annotation process (normally the proportion of mistakes is much lower than the proportion of correct instances).
poor data design, if the way instances/features are defined doesn't match the goal of the ML process (the example provided would fall into this category).


How to measure the information of covariates in a ML task? Background

Recently, I do 2 different ML projects.

One is lending club loan prediction, another is a pravite dataset in online experiment field to predict whether a customer will take the treatment.

Both 2 tasks are binary classification with 100+Million observations and a hundred covariates. However, my model of lending club have a very high PR-ROC(0.86), which shows the good performance of model. My model of online experiment suffers, only have a 0.03 PR-AUC. The model is somekind useless.

I tried to explain to my leader that the dataset is so uninformative and that's the real reason why my project failed. I use low metric scores(PR-AUC = 0.03) and high loss function values as proofs to show the model I build is useless.

Question

Later, I came across one question, how do we measure the information contains in the covariates regardless of the model we built?

If we perform regression tasks using linear model, we can use RMSE, AIC, BIC to select a model, choosing the best model who mined the covariates best. If we perform binary classification ML task like I did before, we can use F1, ROC and PR-ROC metrics etc. I think these metrics helps us to compare model performance instead of the potential data quality.

The solution I want is something like entropy. For instance, we can calculate the entropy between 2 probability distributions or between covariate and target label. Entropy shows us the relation intensity between covariates and label. Is there any better solution to measure the information contains in the covariates?

I am a graduate of statistics, thanks a lot if you guys can provide me any source to learn!

There exists a notion of entropy in information theory. It is in fact closely linked to covariance, and especially the covariance matrix determinant (I don't recall the actual mathematical link but I think this should be something that Google can find).

From then, you could use some metrics based on the dataset correlation matrix. You could use its determinant (low determinant means highly correlated data), but it's not always easy to analyse (2 perfectly correlated features will drop the determinant value to 0). The distribution of correlation matrix eigenvalues will surely give more detailed insight (multiple low eigenvalues will mean lots of correlated features).

Another powerful and less mathematical solution would be to compute PCA over the dataset. Give yourself a threshold of explained variance (for instance 95%), and count how many principal components you need to reach this explained variance. A low number of such would mean low "quality" data.


StackOverflow Tags Predictor...Suggest an Machine Learning Approach please? I am trying to predict tags for stackoverflow questions and I am not able to decide which Machine Learning algorithm will be a correct approach for this.

Input: As a dataset I have mined stackoverflow questions, I have tokenized the data set and removed stopwords and punctuation from this data.

Things i have tried:

TF-IDF
Trained Naive Bayes on the dataset and then gave user defined input to predict tags, but its not working correctly
Linear SVM

Which ML algorithm I should use Supervised or Unsupervised?
If possible please, suggest a correct ML approach from the scratch.
PS: I have the list of all tags present on StackOverflow so, will this help in anyway?
Thanks

This exact problem was a kaggle competition sponsored by Facebook. The particular forum thread of interest for you is the one where many of the top competitors explained their methodology, this should provide you with more information than you were probably looking for: https://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction/forums/t/6650/share-your-approach

In general, it appears that most people treated the problem as a supervised one. Their primary feature was a tf-idf, or unweighted BOW, representations of the text and they ensembled 1000s of single-tag models. Owen, the winner of the competition, noted that the title text was a more powerful feature than the content of the body of the post.


How to construct a self learning process for an ensemble model? I have two different (regression) models spitting out predictions on a daily basis for the same dependent variable. My intention is to assign weights to those two predictions and calculate a weighted average. To this end, I have developed a simple system where the MSEs of the models are calculated and used as weights everyday. As a result, the higher the MSE a model has, the lower the weight it is assigned to the model. However, this is a very lame approach and I haven't observed any improvement compared to taking a simple average of the predictions. So what are the ways that I can use to assign weights to those predictions dynamically. That is, I want to update the weights everyday. From where should I start?

Note: I am aware that with ensemble models, I can get the weights automatically (i.e. H2O stacked ensemble). However, I am not allowed to retrain the predictions everyday. So applying an ensemble method and retrain it everyday is not applicable in my case.

Thanks in advance!

It is not much more sofisticated than what you thought, but you can try training a linear regression (in the online setting, by stochastic gradient descent) whose inputs are the predictions spit by your models and the output is the real value of the predicted variable. In this case, you wouldn't need to retrain with the whole dataset.

In fact, I think that all online machine learning techniques can be applied in this setting.


What is disadvantage of using CART in regression problems? Why is CART hardly used for regression?

Is there any significant reason for its unpopularity in regression techniques?

Regression and classification trees are nearly identical in how they function. I'm not aware of any specific downside to regression trees specifically. However, trees in general do have the downside of being susceptible to overfitting:

  The main downside of decision trees is that even with the use of pre-pruning, they
  tend to overfit and provide poor generalization performance. Therefore, in most
  applications,...ensemble methods...are usually used in place of a single decision tree.

Muller & Guido, Introduction to Machine Learning with Python, 2017.


Which programming languages should I use to integrate different analysis software packages? I am currently writing a package to streamline data analysis for a research lab. There are several different analysis software packages that we use, based out of unix, matlab, and (rarely used) python. A typical data set is about 250GB (raw), and requires at least 4 different preprocessing steps before analysis. The finished product typically ends up taking up about 1TB. The goal of my package is to allow the user to pick and choose which existing package to use for each step before running the analysis, and then the program will execute it without further user intervention. Since the goal is to integrate these different packages, written in different languages, I decided to write the program in bash to make it easy to call the actual analysis scripts no matter what language they are written in.

The program is starting to come along, but it is getting very complex because of the various idiosyncratic expectations and conventions of each analysis package. I realize bash may not be the most suitable language for complex tasks, but I like that it's easy to call scripts in different languages from there, and that it's relatively simple. The program also does a lot of file handling, which bash is good at. On the other hand, I hear it's also very slow, and it gets clunky when things get more complicated.

I'm wondering if bash is the best choice for this task. Does anyone have suggestions for other languages, or combinations of languages, that might be better suited to my needs?

I should note that I am a self-taught programmer and this is my first real programming challenge. I am mostly familiar with bash, matlab, R, and a little bit of python, but I'd like to learning new things too (C maybe?). Also, this is all going to run on unix.

If you are mostly stitching up together calls into other software, like unix utilities (awk, grep, sed ...), python, and matlab scripts, bash is just fine or possibly even best for the job to construct simple pipelines and workflows.

It's easy in bash to read user input, store it in variables, then launch other software depending on the set variables. It's perfectly fast enough for that, and nothing else gets any easier.

If you, however, were to use bash for preprocessing itself, like looping through files line by line, packing and unpacking tab-separated values into arrays etc., that would be excruciatingly slow and not recommended.


Running Neural Network experiments in loop I need to run neural network training with different hyper parameters settings like this:

for seed in range(2):
        NN_inst = FeedForwardNN(seed)
        NN_inst.load_data()
        NN_inst.classification_model()
        NN_inst.run_model()

First run everything works. On the second run I get tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value training/Adam/Variable_39
     [[{{node training/Adam/Variable_39/read}} = IdentityT=DT_FLOAT, _class=["loc:@training/Adam/Variable_39"], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

It looks like I need to clear tensorflow memory, but

K.clear_session()
tf.reset_default_graph()

do not help. I am using Keras, error is on fit().
What should I do?

I using InteractiveSession() and closing it every time worked

sess = tf.InteractiveSession()
#code here ...
sess.close()


after augmentation validation accuracy going down? My main question is about augmentation.
if I process the augmentation I believe it always better than less data
but in my case the validation accuracy going down

train : 7000 images , validation: 3000 images : validation accuracy:0.89

train : 40000 images , validation: 17990 images : validation accuracy:0.85

my augmentation code

def data_augmentation_folder(trainImagesPath,saveDir):
    #X_train=load_training_data(trainImagesPath,"train")
    print("=====================================================")

    X_train = cleanData(trainImagesPath)
    X_train = np.array(X_train)
    print(X_train[0].shape)

    for i in range(5):

        #print(i)

        datagen = ImageDataGenerator(rotation_range=15,
                       width_shift_range=0.1,
                       height_shift_range=0.1,
                       shear_range=0.01,
                       zoom_range=[0.9, 1.25],
                       horizontal_flip=True,
                       vertical_flip=False,
                       fill_mode='reflect',
                       data_format='channels_last',
                       brightness_range=[0.5, 1.5])

        if i==1:
            datagen = ImageDataGenerator(
                       featurewise_center=True,
                                     featurewise_std_normalization=True,
                                     rotation_range=90,
                                     width_shift_range=0.1,
                                     height_shift_range=0.1,
                                     #zoom_range=0.2
            )
        if i==2:
            datagen = ImageDataGenerator(
                   featurewise_center=True,
                                 featurewise_std_normalization=True,
                                 rotation_range=100,
                                 width_shift_range=0.1,
                                 height_shift_range=0.1,
                                 #zoom_range=0.2
            )
        elif i==3:
            datagen = ImageDataGenerator(
                    rescale=1./255,
                    shear_range=0.2,
                    zoom_range=0.2,
                    horizontal_flip=True)
        elif i==4:
             datagen = ImageDataGenerator(
                    rescale=1./255,
                    shear_range=0.1,
                    rotation_range=80,
                    zoom_range=0.1,
                    horizontal_flip=True,
                    brightness_range=[0.5,1.5])

        datagen.fit(X_train)

        for x, y in datagen.flow(X_train, np.arange(X_train.shape[0]),shuffle=True, save_to_dir=saveDir,save_format='jpg',save_prefix='aug'):
            #print(y)
            assert x.shape[1:] == X_train.shape[1:]
            break

  questions

  in which case the validation going down even though I proceeded augmentation?
  what you have to worry about when You proceed the augmentation?

You can not compare validation accuracies if the validation datasets are fundamentally different.

What you were doing initially was to validate your model on a limited set of images. Augmentation is applied to make your dataset more representative of your deployment environment.

If in your deployment environment you expect images to be drawn from the same subspace as your augmented dataset, the validation accuracy of the first case will not be a true representative of your model's real world performance.


how to run my python code on google cloud without fear of getting disconnected - an absolute beginner? I have been trying to use python 3 for text mining on a 650 MB csv file, which my computer was not powerful enough to do. My second solution was to reach out to google cloud. I have set up my VMs and my jupyter notebook on google cloud, and it works perfectly well. The problem, however, is that I am in constant fear of getting disconnected. As a matter of fact, my connection with google server was lost a couple of time and so was my whole work.

My question: Is there a way to have the cloud run my code without fear of getting disconnected? I need to be able to have access to my csv file and also the output file.

I know there is more than one way and have read a lot of material but they are too technical for a beginner like me to understand.

UPDATE: here is how I get access to my jupyter notebook on google cloud:
1- I run my instance on google cloud
2-I click on SSH
3- in the window that appears, I type the following:

jupyter notebook --ip=0.0.0.0 --port=8888 --no-browser &

I have seen people recommend to add nohup to the beginning of the same commend. I have tried it and got this message:

nohup: ignoring input and appending output to 'nohup.out'

And nothing happens.

There are several options to keep your session running on the host machine even if it's disconnected, see for instance this: https://www.tecmint.com/keep-remote-ssh-sessions-running-after-disconnection/

Personally I use the screen command because I'm familiar with it but I think tmux is more recent.


Using Apache Spark to do ML. Keep getting serializing errors so I'm using Spark to do sentiment analysis, and I keep getting errors with the serializers it uses (I think) to pass python objects around.

PySpark worker failed with exception:
Traceback (most recent call last):
  File "/Users/abdul/Desktop/RSI/spark-1.0.1-bin-    hadoop1/python/pyspark/worker.py", line 77, in main
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/Users/abdul/Desktop/RSI/spark-1.0.1-bin-    hadoop1/python/pyspark/serializers.py", line 191, in dump_stream
    self.serializer.dump_stream(self._batched(iterator), stream)
  File "/Users/abdul/Desktop/RSI/spark-1.0.1-bin-    hadoop1/python/pyspark/serializers.py", line 123, in dump_stream
    for obj in iterator:
  File "/Users/abdul/Desktop/RSI/spark-1.0.1-bin-    hadoop1/python/pyspark/serializers.py", line 180, in _batched
    for item in iterator:
TypeError: __init__() takes exactly 3 arguments (2 given)

and the code for serializers is available here

and my code is here

Most often serialization error in (Py)Spark means that some part of your distributed code (e.g. functions passed to map) has dependencies on non-serializable data. Consider following example:

rdd = sc.parallelize(range(5))
rdd = rdd.map(lambda x: x + 1)
rdd.collect()

Here you have distributed collection and lambda function to send to all workers. Lambda is completely self-containing, so it's easy to copy its binary representation to other nodes without any worries.

Now let's make things a bit more interesting:

f = open("/etc/hosts")
rdd = sc.parallelize(range(100))
rdd = rdd.map(lambda x: f.read())
rdd.collect()
f.close()

Boom! Strange error in serialization module! What just happened is that we had attempted to pass f, which is a file object, to workers. Obviously, file object is a handle to local data and thus cannot be sent to other machines.

So what's happening in your specific code? Without actual data and knowing record format, I cannot debug it completely, but I guess that problem goes from this line:

def vectorizer(text, vocab=vocab_dict):

In Python, keyword arguments are initialized when function is called for the first time. When you call sc.parallelize(...).map(vectorizer) just after its definition, vocab_dict is available locally, but remote workers know absolutely nothing about it. Thus function is called with fewer parameters than it expects which results in __init__() takes exactly 3 arguments (2 given) error.

Also note, that you follow very bad pattern of  sc.parallelize(...)...collect() calls. First you spread your collection to entire cluster, do some computations, and then pull the result. But sending data back and forth is pretty pointless here. Instead, you can just do these computations locally, and run Spark's parallel processes only when you work with really big datasets (like you main amazon_dataset, I guess).


What is the ideal size to a binary CNN? Is my dataset long enough? I would like to know what is the ideal size to a CNN, or there's a mathematical function to determine it, or it change through the differents scopes?

And also, I'm doing a binary classification CNN with 700 images each class(total - 1400). Is it too small? How could I argue that it is small?

Note: I get 83% accuracy with my cnn.

There is no mathematical function that is able to relate size of model and performance.

Architecture(filter size, activation, depth, weights initialization) of a neural network can be thought as hyperparameter and hence to improve your model you need to employ hyperparameter tuning methods.

The other way to improve your model is to experiment with state-of-the-art features with your architecture. I believe your dataset being small enough give you a lot of room to experiment(because of the supposedly fast training time).


Steps taking too long to complete I'm trying to train a model which in my opinion is taking too long compared to other datasets given that it's taking about 9s to complete a step. I think that the problem is because the dataset is not being stored on ram, but I'm not sure of this.

The code is the following:

def load_data():

    train_datagen = ImageDataGenerator(rescale=1./255, shear_range=0.2, zoom_range=0.2, horizontal_flip=True)
    train_generator = train_datagen.flow_from_directory(path1, target_size=(200, 200), batch_size=32, class_mode="binary")

    test_datagen = ImageDataGenerator(rescale=1./255)
    test_generator = test_datagen.flow_from_directory(path2, target_size=(200, 200),batch_size=32, class_mode="binary")

    return train_generator, test_generator

Model:

Sequential model
2 Convolutional layers with 32 neurons, activation = relu.
1 Convolutional layer with 64 neurons, activation = relu.
Flattening and Dense layer, activation = relu.
Dropout of 0.5
Output layer (Dense) with sigmoid activation.
Adam optimizer.
Loss: binary cross entropy.

Fit:

model.fit_generator(x, steps_per_epoch=37, epochs=50, validation_data=y, validation_steps=3, callbacks=[tensorboard])

My dataset has 1201 images and 2 classes.
I built the model following this tutorial.
My GPU is a GTX 1060 3gb.
8gb of ram.
The images are being reshaped to 200x200.

If you could help me I'd appreciate it. Thank you very much!

Your images are probably too large for using just three convolutions. So the number of parameters is huge during flattening (and the net cannot learn global features of the images, but this is a different problem).

Try to

use more convolution and pooling operations
see if strided convolutions help
check the number of parameters using the summary method of Keras


Classify event announcement I would like  train a classifier to spot news articles which would spot articles that announce an event. The issue is that I do not have a large pre-labeled dataset for this task (I only have 200 examples). So here are my two questions:

Have you heard of a training set labeled for such task
I heard of few shot learning that can be helpful to train a classifier with few example would it be applicable to this case and is there any library/reading you would recommend.

Thanks in advance

Semi-supervised learning and in particular active learning could be considered in cases like this:

The general semi-supervised setting consists in training a model from an initially small training set by applying it iteratively to unlabelled instances. There are various methods to minimize the risk of training the model on wrongly classified instances.
Active learning is a variant of semi-supervised learning where the model queries the human expert for annotations, but the instances are carefully selected in order to minimize the amount of human labor.
There is also bootstrapping, where one would focus on the positive instances: apply the original model to the unlabelled data, the manually annotate only the instances which are predicted as positive (useful only in cases where the positive class is much smaller than the negative one).


Ridge and Lasso Regularization Recently, I started working on Ridge and Lasso regularization for Linear and Logistic Regression.
My doubts are given below:

Is the penalty the same (by same proportion) for all the coefficients or is it based on variable importance?
If it is the latter I believe we can directly apply regularization rather than spending time in feature selection.
Whether the Multi-collinearity is taken care by ridge and lasso regularization?

Thank you.

The penalty of both Lasso and Ridge is proportional to the magnitude of the weight. That is, the penalization added to the cost function is $\lambda ||\omega||_2$ or $\lambda ||\omega||_1$.

Wether it is more convenient to apply regularization or feature selection, Lasso already does some feature selection for you, as the estimated weights for Lasso are sparse (there will be many coefficients equal to 0).

About multi-colinearity, Ridge tends to eliminate variables that are colinear, while Lasso doesn't.

All that I have said is taken from An Introduction to Statistical Learning book, from James, Witten, Hastie and Tibshirani.


Is reinforcement learning a subset of unsupervised learning? According to this article:

Reinforcement learning on the other hand, which is a subset of Unsupervised learning ...

How true is this statement? Is there any scholarly discussion/writing on the classifications of Reinforcement learning?

Reinforcement learning differs from Unsupervised learning as it uses additional information regarding the expected behavior of the agent in the form of a reward function. However, it also differs from Supervised learning as it does not require any labelled data for training or testing.
So, it is neither of them. You may also refer to the wiki page of Reinforcement learning for further related details.


Pyspark coverting timestamps from UTC to many timezones This is using python with Spark 1.6.1 and dataframes.

I have timestamps in UTC that I want to convert to local time, but a given row could be in any of several timezones. I have an 'offset' value (or alternately, the local timezone abbreviation. I can adjust all the timestamps to a single zone or with a single offset easily enough, but I can't figure out how to make the adjustment dependent on the 'offset' or 'tz' column.

There appear to be 2 main ways of adjusting a timestamp:
using the 'INTERVAL' method, or using pyspark.sql.from_utc_timestamp.

Here's an example:

data = [
  ("2015-01-01 23:59:59", "2015-01-02 00:01:02", 1, 300,"MST"),
  ("2015-01-02 23:00:00", "2015-01-02 23:59:59", 2, 60, "EST"),
  ("2015-01-02 22:59:58", "2015-01-02 23:59:59", 3, 120,"EST"),
  ("2015-03-02 15:59:58", "2015-01-02 23:59:59", 4, 120,"PST"),
  ("2015-03-16 15:15:58", "2015-01-02 23:59:59", 5, 120,"PST"),
  ("2015-10-02 18:59:58", "2015-01-02 23:59:59", 4, 120,"PST"),
  ("2015-11-16 18:58:58", "2015-01-02 23:59:59", 5, 120,"PST"),
  ("2015-03-02 15:59:58", "2015-01-02 23:59:59", 4, 120,"MST"),
  ("2015-03-16 15:15:58", "2015-01-02 23:59:59", 5, 120,"MST"),
  ("2015-10-02 18:59:58", "2015-01-02 23:59:59", 4, 120,"MST"),
  ("2015-11-16 18:58:58", "2015-01-02 23:59:59", 5, 120,"MST"),
  ...
]

(I realize the offset and tz columns aren't consistent - this isn't real data)

df = sqlCtx.createDataFrame(data, ["start_time", "end_time", "id","offset","tz"])
from pyspark.sql import functions as F

these two options both do what is expected:

df.withColumn('testthis', F.from_utc_timestamp(df.start_time, "PST")).show()
df.withColumn('testThat', df.start_time.cast("timestamp") - F.expr("INTERVAL 50 MINUTES")).show()

But if I try to replace the "PST" string with df.tz, or the " 50 " string with df.offset.cast('string'), I get a type error:

  TypeError: 'Column' object is not callable

I've tried variations on this, but to no avail.


Looping problem in python I was trying to plot the cluster result of DBSCAN clustering. I cluster data into two cluster and when i write code to plot those it shows 'name error'. But I couldn't understand whats the problem.
Here's my code with error

for i in range(0, reduced_data.shape[0]):
    if dbscan.labels_[i] == 0:
        c1 = plt.scatter(reduced_data[i,0],reduced_data[i,1],c='r',marker='+')
    elif dbscan.labels_[i] == 1:
        c2 = plt.scatter(reduced_data[i,0],reduced_data[i,1],c='g',marker='o')
    elif dbscan.labels_[i] == -1:
        c3 = plt.scatter(reduced_data[i,0],reduced_data[i,1],c='b',marker='*')
    plt.legend([c1, c2, c3], ['Cluster 1', 'Cluster 2','Noise'])
    plt.title('DBSCAN finds 2 clusters and noise')
    plt.show()

Edited:

Rest of my code:

feature_cols = ['age','workclass','fnlwgt','education','education num','marital-status','occupation','relationship','race','sex','capital-gain','capital-loss','hours-per-week','native-country']
X = train[feature_cols]
y = train['label']

# split X and y into training and testing sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30,random_state=10)

X_train_scale = scale(X_train)
X_test_scale = scale(X_test)

reduced_data = PCA(n_components=2).fit_transform(X_train_scale)
reduced_data_test = PCA(n_components=2).fit_transform(X_test_scale)

from pylab import *
xx, yy = zip(*reduced_data)
scatter(xx,yy)
show()

dbscan = DBSCAN(eps=0.3, min_samples=10).fit(reduced_data)
labels=dbscan.labels_
print(labels)

n_clusters_ = len(set(labels)) - (1 if -1 in labels else 0)
print (n_clusters_)

The result of n_cluster is 2

and the main data before PCA reducing is like these:

In your specific case you only have 2 clusters, however this is not necessarily always going to be the case. I would allow for more flexibility.

I assume from your sample code that you are following what is shown in the docs. Following from what they are doing you should have the following

feature_cols = ['age','workclass','fnlwgt','education','education num','marital-status','occupation','relationship','race','sex','capital-gain','capital-loss','hours-per-week','native-country']
X = train[feature_cols]
y = train['label']

# split X and y into training and testing sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30,random_state=10)

X_train_scale = scale(X_train)
X_test_scale = scale(X_test)

reduced_data = PCA(n_components=2).fit_transform(X_train_scale)
reduced_data_test = PCA(n_components=2).fit_transform(X_test_scale)

from pylab import *
xx, yy = zip(*reduced_data)
scatter(xx,yy)
show()

# Compute DBSCAN
db = DBSCAN(eps=0.3, min_samples=10).fit(reduced_data)
core_samples_mask = np.zeros_like(db.labels_, dtype=bool)
core_samples_mask[db.core_sample_indices_] = True
labels = db.labels_

# Number of clusters in labels, ignoring noise if present.
n_clusters_ = len(set(labels)) - (1 if -1 in labels else 0)

I just changed your variable name dbscan to db to make it easier to go through the code from the source. From here you can plot all the different clusters that were identified by the DBSCAN method. You should also keep the mask which retains the labels as a list in order for us to access them easily when we plot.

We will identify what are the unique labels which DBSCAN identified and we will map a color to each of these labels.

# Plot result
import matplotlib.pyplot as plt

# Black removed and is used for noise instead.
unique_labels = set(labels)
colors = [plt.cm.Spectral(each)
          for each in np.linspace(0, 1, len(unique_labels))]

For each cluster as determined by the unique labels we will plot all the values associated with it as

for k, col in zip(unique_labels, colors):
    # -1 is an identifier for noise
    if k == -1:
        # Black used for noise.
        col = [0, 0, 0, 1]

    # Find out what instances belong to this cluster, k
    class_member_mask = (labels == k)

    # Pull out these instances
    xy = X[class_member_mask]

    # Plot all of these instances
    plt.plot(xy[:, 0], xy[:, 1], 'o', markerfacecolor=tuple(col),
             markeredgecolor='k', markersize=6)

plt.title('Estimated number of clusters: %d' % n_clusters_)
plt.show()


Marginalization of joint distribution I am trying to understand how you marginalise a joint distribution.

In my case I have a fair coin,  $P(C) = \frac12$ and a fair dice $P(D) = \frac16$. I am told I win a prize if I flip the coin and it lands on Tails and if the outcome of the dice $= 1$. I am told at least one of them is correct.

$$Q = (\text{Coin = Tails or Dice} = 1)$$

$$W = (\text{Coin = Tails and Dice} = 1)$$

So if I wanted to work out the probability $W =$ True $| Q =$ True I can use marginalisation to work this out given the joint distribution:

$$P(C), P(D), P(Q|C,D), P(W|C,D)$$

I am just not sure where to start any help would be really appreciated. I am pretty new to this.

Thanks in advance.

Guide:

To solve the problem, we have to assume that there is the outcome of the dice and the outcome of the coins are independent.

We let $C$ denotes the event that the coin lands on tail and $D$ be the event that the dice outcome is $1$.

You have been given that the coin lands on tails or  the dice lands on $1$, and you are interested in finding out that the coin lands on tails and  the dice lands on $1$.

You want to compute $P(W|Q)=\frac{P(W\cap Q)}{P(Q)}=\frac{P(W)}{P(Q)}=\frac{P(C)P(D)}{P(Q)}$.

To compute $P(Q)$ where $Q= C \cup D$. You can either use $$P(Q)= P(C)+P(D)-P(C\cap D)$$

or $$P(Q)=1-P(Q^c)=1-P(C^c \cap D^c)$$

Given all these formulas, hopefully you can solve for $P(W|Q)$.


How do you pronounce ROC? In regards to ROC curves, how do you pronounce ROC? I have always spelled out the letters like R-O-C but I sat through a sales demo today where they guy pronounced it as a word like "rock" as in "the rock curve".

So who's right?

I have personally only heard the letters being spelled out:

  "Here we see the R-O-C curve on this pronunciation model is far from optimal"


Interpretation of ROC AUC score i tried to evaluate 6 models and after plotting , this what i get :

So i'm wondering , if those results are "Right" ?

Thank's in advance.

Did you evaluate the results in the training set? Or in the test set?

Those results are outstandingly good! Suspiciously good.

I think you tried your results in the training set only, so your results reflect overfitting on your data, which means your model learned the set, it was not generalized (which means it is unapplicable to any other dataset you may encounter in the future, which is not useful).

For comparing ROC between methodologies you should model them being careful for overfitting and try them on a test dataset (a dataset which you never knew before, which can be obtained partitioning your dataset).

In that way your comparison is not measuring which model learns by memory your data.


How to choose between different types of feature scaling? The feature set for my multi-class multi-label classification task, using the MLPClassifier from scikit learn, contains mostly features where the values are in the same range of [0,1], but there are 3 out of 45 features where this isn't the case and feature scaling is required. So far I've tried out min-max normalization, mean normalization and z-score normalization on these features. However all scaling methods result in slightly different train and test performance and z-score standardization results in the fastest convergence but worst scores overall. To measure performance, Precision, Recall, F1 and MCC were used.
What is a decent strategy when choosing a type of feature scaling?

You've listed a few different error metrics, I would pick one metric that is best suited to your problem. Trying to maximize several metrics at once makes it difficult to tell if your model is getting better. In any case, if normalization leads to the best score - then that is your answer. Since all other variables were already in range [0,1], then that's probably what I would have started with just to keep it consistent.
I am curious how much worse the z-score standardization is performing compared to normalization. From what you've written about the data - I'd be surprised if standardization vs. normalization produced significantly different results when only 3 out of 45 features are impacted.
This blog explains the topic better than I can About Feature Scaling. This article has several examples of training a MLP with scaled and unscaled variables How to use Data Scaling Improve Deep Learning Model Stability and Performance.
I'd also consider some simpler algorithms. It might be good to also train a logistic regression or similar as a comparison, fewer parameters will be easier to configure and less prone to error. A neural net will give you a boost if you have a lot of training data or a very non-linear problem.


Why does my cost function return double the expected value? I tried to implement a function propagate() that computes the cost function and its gradient knowing that :

Forward Propagation:

You get X
You compute $A = \sigma(w^T X + b) = (a^{(0)}, a^{(1)}, ..., a^{(m-1)}, a^{(m)})$
You calculate the cost function: $J = -\frac{1}{m}\sum_{i=1}^{m}y^{(i)}\log(a^{(i)})+(1-y^{(i)})\log(1-a^{(i)})$

Here are the two formulas I was using:

$$ \frac{\partial J}{\partial w} = \frac{1}{m}X(A-Y)^T\tag{7}$$
$$ \frac{\partial J}{\partial b} = \frac{1}{m} \sum_{i=1}^m (a^{(i)}-y^{(i)})\tag{8}$$

Which I coded as :

def propagate(w, b, X, Y):
    """
    Implement the cost function and its gradient for the propagation explained above

    Arguments:
    w -- weights, a numpy array of size (num_px * num_px * 3, 1)
    b -- bias, a scalar
    X -- data of size (num_px * num_px * 3, number of examples)
    Y -- true "label" vector (containing 0 if non-cat, 1 if cat) of size (1, number of examples)

    Return:
    cost -- negative log-likelihood cost for logistic regression
    dw -- gradient of the loss with respect to w, thus same shape as w
    db -- gradient of the loss with respect to b, thus same shape as b

    Tips:
    - Write your code step by step for the propagation. np.log(), np.dot()
    """

    m = X.shape[1]

    # FORWARD PROPAGATION (FROM X TO COST)
    ### START CODE HERE ### (≈ 2 lines of code)
    A = sigmoid(np.dot(w.T,X)+b)                                    # compute activation
    cost = -1/m * np.sum(np.dot(Y.T,np.log(A)) + np.dot((1-Y).T,np.log(1-A)))                                # compute cost
    ### END CODE HERE ###

    # BACKWARD PROPAGATION (TO FIND GRAD)
    ### START CODE HERE ### (≈ 2 lines of code)
    dw = 1/m*np.dot(X,(A-Y).T)
    db = 1/m*np.sum(A-Y)
    ### END CODE HERE ###

    assert(dw.shape == w.shape)
    assert(db.dtype == float)
    cost = np.squeeze(cost)
    assert(cost.shape == ())

    grads = {"dw": dw,
             "db": db}

    return grads, cost

With the following example :

w, b, X, Y = np.array([[1.],[2.]]), 2., np.array([[1.,2.,-1.],[3.,4.,-3.2]]), np.array([[1,0,1]])
grads, cost = propagate(w, b, X, Y)

My output was :

dw = [[ 0.99845601]
 [ 2.39507239]]
db = 0.00145557813678
cost = 10.6046359582

Whereas the expected output was

You are supposed to perform calculations on cost function element wise. Try using

np.multiply

instead of

np.dot

which is an inner product for Matrices.


How to fix the index value that I use for X_test and y_test? I am writing code for SVR. Therefore, I have generated my code as the requirements. But I am stuck on writing codes for the index of for-loop of X_test and y_test. I have to write code as it should be associated with the line in the datasets just next to the X_train and y_train. So their index should be +1 of the ending index of X_train and y_train.

For Example:

In the first iteration (i.e. when i=0), we are using the first 1000 rows for training and the next row (i.e. the 1001st row) for testing
In the second iteration (i.e. when i=1), we are using the rows from 1 to 1001 for training and the next row (i.e. the 1002nd row) for testing
In the third iteration (i.e. when i=2), we are using the rows from 2 to 1002 for training and the next row (i.e. the 1003rd row) for testing and so on.

My full code:

import pandas as pd
import numpy as np

# Make fake dataset
dataset = pd.DataFrame(data= np.random.rand(2000,22))
dataset['age'] = np.random.randint(2, size=2000)

# Separate the target from the other features
target = dataset['age']
data = dataset.drop('age', axis = 1)

X_train, y_train = data.loc[:1000], target.loc[:1000]

X_test,  y_test  = data.loc[1001], target.loc[1001]

X_test = np.array(X_test).reshape(1, -1)
print(X_test.shape)

SupportVectorRefModel = SVR()
SupportVectorRefModel.fit(X_train, y_train)

y_pred = SupportVectorRefModel.predict(X_test)
y_pred

y_pred_list = []
y_test_list = []

for i in range(1, 2000):

    X_train, y_train = dataset.iloc[i:1000+i], target.iloc[i:1000+i]
    X_test, y_test = dataset.iloc[i], target.iloc [i]

    X_test = np.array(X_test).reshape(1, -1)
    print(X_test.shape)

    SupportVectorRefModel = SVR()
    SupportVectorRefModel.fit(X_train, y_train)
    y_pred = SupportVectorRefModel.predict(X_test)

    y_pred_list.append(y_pred)

    y_test_list.append(y_test)

print(y_test_list, y_pred_list)

I want to update my code on this below:

 X_test, y_test = dataset.iloc[i], target.iloc [i]

So how may I update this index line as an above requirement?

I've changed the range values in the loop and indexing for training and test data. (Also, it seems you are using the dataset instead of data by mistake.)

i:(999 + i) contains thought rows locations of which will be incremented by one as the loop progresses.

for i in range(0, 999):

    X_train, y_train = data.iloc[i:(999 + i)], target.iloc[i:(999 + i)]
    X_test, y_test = data.iloc[i + 1000], target.iloc [i + 1000]


FREAK feature extraction OpenCV I want to know how to use FREAK feature extraction in python,
I read the documentation but I need some examples.
I wrote this code :

freakExtractor = cv2.xfeatures2d.FREAK_create()
keypoints,descriptors= freakExtractor.compute(image,None)

but the descriptor are None, why?

I used keypoints detected by SURF instead of None parameter:

surfDetector = cv2.xfeatures2d.SURF_create()
keypoints = surfDetector.detect(image,None)
freakExtractor = cv2.xfeatures2d.FREAK_create()
keypoints,descriptors= freakExtractor.compute(image,keypoints)

but the output was very very large, and hard to process.
How can I fix this?
Thanks...

FREAK is only a feature descriptor, so in order to use it you must provide a set of keypoints for it to extract your features e.g., FAST, AGAST, SIFT’s keypoint detector, etc. The original paper uses AGAST.
You might need to process the keypoints since the region requirements are different (A keypoint for SURF might lay too close on the border to compute FREAK).
keypoints = SomeDetector(image)
freakExtractor = cv2.xfeatures2d.FREAK_create()
keypoints,descriptors= freakExtractor.compute(image,keypoints)

The FREAK: Fast Retina Keypoint paper says:

Rostenand and Drummond proposed in [A] the FAST criterion for corner detection, improved by Mair et al. in [B] with their AGAST detector. The latter is a fast algorithm to locate keypoints. The detector used in BRISK by Leutenegger et al. in [C] is a multi-scale AGAST. They search for maxima in scale-space using the FAST score as a measure of saliency. We use the same detector for our evaluation of FREAK.

[A] E. Rosten and T. Drummond. Machine learning for highspeed corner detection. Computer Vision–ECCV 2006, pages 430–443, 2006.
[B] E. Mair, G. Hager, D. Burschka, M. Suppa, and G. Hirzinger.
Adaptive and generic corner detection based on the accelerated segment test. Computer Vision–ECCV 2010, pages 183–196, 2010. 2
[C] S. Leutenegger, M. Chli, and R. Siegwart. Brisk: Binary
robust invariant scalable keypoints. 2011.

Sources: Here and This SO question


What is difference between Standard Normal Distribution and Mean Normalization approaches to feature-scaling? The tag feature-scaling seems to convey that one of the scaling methods is Standard Normal Distribution. Further, I read an Answer on this site saying that Mean Normalization is a form of feature scaling.
What is the difference between two approaches to scaling?
Note: I think that statistics and mathematics of normalization do differ.

The terms standardization and normalization are often used interchangeably. However, strictly speaking they do refer to distinct feature transformations.
Normalization
Normalization, also called feature scaling usually means scaling the data between 0 and 1. There are many approaches that can be used to achieve this. One common way is by
$x' = \frac{x - x_{min}}{x_{max} - x_{min}}$
Standardization
Standardization transforms the feature to have a mean 0 and a standard deviation of 1. This is also called z-scoring and can be achieved by
$x_i' = \frac{x_i - \bar{x}}{s}$
where $\bar{x}$ is the mean of the feature and $s$ is the standard deviation of the feature.


What is the more natural parsing, the one that leads to the preferred reading of the sentence I have those rules:

and those two possible parse trees:

I am asked for the next question:
What is the more natural parsing, the one that leads to the preferred reading of the sentence?
Can anyone explain to me, what is more natural in English and why ?
according to this
A determiner is a noun-modifier that expresses the reference of a noun or noun phrase in the context.

I don't see any possible more natural distinguish.

Can anyone explain to me, what is more natural in English and why ?

This is a classic example of PP-attachment ambiguity (PP = prepositional phrase). For a full overview of the problem and some traditional approaches, check out this paper. Here I'll cover the basics.
The quick explanation is that the first analysis corresponds to the interpretation
Twain ( bought (a book (for Howells)) )

whereas the second analysis means
Twain ( bought (a book) (for Howells) )

The second one is probably the most natural interpretation: somebody (Twain) buys something (a book) for somebody else (Howells). So, in this case, the subject may be just buying a gift for somebody else.
On the other hand, in the first sentence, somebody (Twain again) buys something (a book again as well) but now that something was intended for somebody else (a book for Howells) yet the buyer is keeping it for himself.
The explanation is that, when parsing this sentence, the for-prepositional phrase is consumed when it is attached to the closest noun (book). Therefore, it's no longer available later to become the for-phrase beneficiary argument of the verb buy, which therefore can no longer have its gift-making meaning and switches instead to its purely transactional transfer-of-ownership meaning.
The branching of the trees represents these two interpretations, that is, the different syntactic ways in which the same linear order of constituents can be arranged. This phenomenon is one of the traditional examples that linguistic structures do not map exactly to the observable temporal sequences we perceive.
The interesting thing in your example is that the "less typical" interpretation (the first one, corresponding to the "not-a-gift" meaning) is being assigned a higher probability, while I would expect it to be much less likely, since the probability of the "gift" meaning should be much higher when the for-phrase has a person name as the head (Howells) –people just don't go around buying books that belong to other people, but rather for them.
Hope this helps!


Clearest way to visualise temporal data? I've collected bus arrival times at my local bus stop from the past month - so I have every time my bus (a specific bus number) shows up at my bus stop for each day of the week (Monday, Tuesday, etc.).

I am struggling at determining the best and clearest way to display this data. Eventually I want to use a clustering algorithm to help understand the most likely time the bus will show up. So on a Monday I know if the bus is more likely to show up at 7.45am or 7.48 am.

I believe 8 charts will be best - one chart for each day of the week, and then one final chart that shows the average regardless of day of week. What would be my best chart type for clearly visualising this data?

How about using a boxplot, so you can also visualize the variance to some degree. Each boxplot could represent one weekday.

When you look only at the mean (and be at the busstop at this time) you have a good chance to miss the bus (about 50% for a well-shaped non-skewed distribution).

You want to make sure that you be at the stop at the right time (95% chance of getting the bus or so). Therefore variance is important.

The image is from this page: https://www.datamentor.io/r-programming/box-plot/

Alternatively, you could train a model on the arrival time, make predictions for each day, and plot the predicted value including a 95% confidence interval. But this probably is too much here.


Newton method and Vanishing Gradient I read the article on Vanishing Gradient problem, which states that the problem can be rectified by using ReLu based activation function.

Now I am not able to understand that if using ReLu based activation function solves the problem, then why there are so many research papers suggesting the use of Newton's method based optimization algorithms for deep learning instead of Gradient Descent?
While reading research papers, I was having the strong impression that vanishing gradient problem was the core reason for such suggestions but now I am confused whether Newton's method is really needed if Gradient Descent can be modified to rectify all the problems faced during machine learning.

Why there are so many research papers suggesting the use of Newton's
  method based optimization algorithms for deep learning instead of
  Gradient Descent?

Newton method has a faster convergence rate than gradient descent, and this is the main reason why it may be suggested as a replacement for gradient descent.

  Is Newton's method really needed if Gradient Descent can be modified
  to rectify all the problems faced during machine learning?

Existence of vanishing gradient problem depends on the choice of "activation function" and the "depth" of network. Newton method and gradient descent would both face this problem for a function like Sigmoid, since in the flat extremes of Sigmoid both first and second order derivatives are small and exponentially vanishing by depth. In other words, the problem is solved for both methods by the choice of function.

As a side note, 1st- and 2nd-order derivatives of Sigmoid go to zero at the same rate. Here is a graph of Sigmoid and its derivatives; zoom into the extremes.

Historical note. Newton method predates the vanishing gradient problem (which was faced after the introduction of Backpropagation in 60s) by centuries.


How does class_weights work in RandomForestClassifier I'm facing a problem with unbalanced classes, and have tried out a couple of methods like over and under sampling. However, my cross validation mean comes out to be only 0.4 and my confusion matrix shows that the prediction and recall is completely incorrect.

Ive read that the next step is to add weights to my class; I have two of them "Won" and "Lost". In Pandas, how am I supposed to assign weights to them? I know that there is a "class_weights" attribute, but I have no clue on how to use it.

Thanks

PS. My "Won" class is unbalanced, very small compared to the "Lost" one. I train by repeating the set of "Won"s twice and randomly sample an almost equal amount of "Lost"s. I've tried all sorts of combinations of the classes.

Maybe try to encode your target values as binary. Then, this class_weight={0:1,1:2} should do the job. Now, class 0 has weight 1 and class 1 has weight 2.


How to choose PCA or KernelPCA a priori? I am learning about dimensionality reduction and I understood that one of the most used techniques in ML is PCA.

If I understood correctly, I use PCA whenever I want to reduce the number of features which should be mostly linearly separable (independent ?).

When the features are not-linearly separable (linearly not independent?), a nonlinear technique is required to reduce the dimensionality of a dataset and therefore I use KernelPCA.

Question: Supposing that what I just wrote is correct, how can I know in advance if the features are linearly or non-linearly separable before i decide which technique to use? So far the only way I was able to "guess" is by plotting the features after a PCA and checking if i can separate the new features through straight lines/surfaces. If I am not able to do so, then I apply Kernel PCA. Is this approach even correct?

Note:

Feel free to modify my question where the "?" are :)
My features don't have labels. It's an unsupervised problem.

Lots of the dimensionality problems are trial and errors at first.

Only very few datasets are linear "manifolds" that can be described with PCA. But it's a good start to see if there are disjoint sets, or if you can figure out some structure in 2D or 3D.

KPCA is a good technique, but there are many different kernels you can try. You may want to start with other "better" techniques (better as more deterministic) like ISOMAP, Laplacian Eigenmaps or LLE.

One of the things to remember is that you need enough data to populate the embedded space.