What is the difference between trax vs tensorflow? What is the main difference between trax vs tensorflow?
both of them deep learning library and implemented by google team.
https://github.com/google/trax
https://github.com/tensorflow/tensorflow

Trax is a high-level interface to make it easy to apply existing models (like ResNet, LSTM, Transformer). Trax focuses almost exclusively on deep learning.
TensorFlow is a data flow computational framework that can be applied to a wide variety of problems. TensorFlow allows fine grain control over every step in the modeling process. TensorFlow is often used for deep learning, but given its flexibility it can be used for any numerical computation.


Best way to classify datasets with mixed types of attributes I would like to know what is the best way to classify a data set composed of mixed types of attributes, for example, textual and numerical. I know I can convert textual to boolean, but the vocabulary is diverse and data become too sparse. I also tried to classify the types of attributes separately and combine the results through meta-learning techniques, but it did not work well.

Christopher's answers seem very reasonable. In particular tree based methods do well with this sort of data because they branch on discriminative features. It's a little hard to say without knowing your specific application, but in general if you think that some of your features might be significantly more discriminative than others, you could try some dimensionality reduction techniques to clean this up a bit.

Also if you use a dimensionality reduction technique you end up getting a slightly more robust format for your feature vector (they generally end up being straight numerical vectors instead of mixed data types), which might let you leverage different methods. You could also look into hand engineering features. With properly hand engineered features Random Forest will get you very close to state of the art on most tasks.


distinguish users for recommender system How can you calculate better video recommendations for a SmartTV app which is used by multiple users in a household. I don't know which user is currently watching the video because the account for the app is shared. So the current user might get a recommendation based on video usage of an other user.

I was thinking of using principal component analysis to find out how many users live in the household and which user is likely to watch the video now. I could map the usage data to this specific user and then build recommendations for this hypothetical user.

What would be the best approach to solve this problem. Are there best practices or papers which describe a solution?

I was interestedin this problem awhile back ago.  I still have this paper and should serve as a good primer.

Guess Who Rated Thos Movie:...


What is query id ("qid") in XGBoost In XGBoost documentation it's said that for ranking applications we can specify query group ID's qid in the training dataset as in the following snippet:

1 qid:1 101:1.2 102:0.03
0 qid:1 1:2.1 10001:300 10002:400
0 qid:2 0:1.3 1:0.3
1 qid:2 0:0.01 1:0.3
0 qid:3 0:0.2 1:0.3
1 qid:3 3:-0.1 10:-0.3
0 qid:3 6:0.2 10:0.15

I have a couple of questions regarding qid's (standard LTR setup set of search queries and documents, they are represented by query, document and query-document features):

1) Let's say we have qid's in our training file. Does it mean that the optimization will be performed only on a per query basis, all other features specified will be considered as document features and cross-query learning won't happen?

2) Let's assume that queries are represented by query features. Should we still have qid's specified in the training file or we should just list query, document and query-document features?

UPDATE:

So far, I have the following explanation, but how correct or incorrect it is I don't know:

Each row in the training set is for a query-document pair, so in each row we have query, document and query-document features. If we specify "qid" as a unique query ID for each query (=query group) then we can assign weight to each of these query groups. If the weight in some query group is large, then XGBoost will try to make the ranking correct for this group first.

From a file in XGBoost repo:

weights = np.array([1.0, 2.0, 3.0, 4.0])
...
dtrain = xgboost.DMatrix(X, label=y, weight=weights)
...
# Since we give weights 1, 2, 3, 4 to the four query groups,
# the ranking predictor will first try to correctly sort the last query group
# before correctly sorting other groups.

and also:

In ranking task, one weight is assigned to each query group
(not each data point). This is because we only care about the
relative ordering of data points within each group, so it
doesn't make sense to assign weights to individual data points.

UPDATE 2:

Found this link.  Given

3 qid:1 1:1 2:1 3:0 4:0.2 5:0 # 1A
2 qid:1 1:0 2:0 3:1 4:0.1 5:1 # 1B
1 qid:1 1:0 2:1 3:0 4:0.4 5:0 # 1C
1 qid:1 1:0 2:0 3:1 4:0.3 5:0 # 1D
1 qid:2 1:0 2:0 3:1 4:0.2 5:0 # 2A
2 qid:2 1:1 2:0 3:1 4:0.4 5:0 # 2B
1 qid:2 1:0 2:0 3:1 4:0.1 5:0 # 2C
1 qid:2 1:0 2:0 3:1 4:0.2 5:0 # 2D
2 qid:3 1:0 2:0 3:1 4:0.1 5:1 # 3A
3 qid:3 1:1 2:1 3:0 4:0.3 5:0 # 3B
4 qid:3 1:1 2:0 3:0 4:0.4 5:1 # 3C
1 qid:3 1:0 2:1 3:1 4:0.5 5:0 # 3D

the following set of pairwise constraints is generated (examples are referred to by the info-string after the # character):

1A>1B, 1A>1C, 1A>1D, 1B>1C, 1B>1D, 2B>2A, 2B>2C, 2B>2D, 3C>3A, 3C>3B, 3C>3D, 3B>3A, 3B>3D, 3A>3D

So qid seems to specify groups such that within each group relevance values can be compared to each other and between groups relevance values can't be directly compared (inc. during the training procedure). So during training we need to have qid's and during inference we don't need them as input.

Thank you!

In ranking applications of information retrieval, training data consists of queries and documents matching them together with relevance degree of each match.

For example when searching in something in google, the training data may be prepared manually by human assessors (or raters, as Google calls them), who check results for some queries and determine relevance of each result.

It is not feasible to check the relevance of all documents, and so typically only the top few documents, retrieved by some existing ranking models are checked.

Training data is used by a learning algorithm to produce a ranking model that computes the relevance of documents for actual queries.

Rank profiles can have one or two phases:

Phase one should use a computationally inexpensive function to rank candidates
Phase two is run on a small candidate set

In short, the query selection and first phase ranking reduces the size of the computation - then machine-learned models can be used on the second-phase ranking on rerank-count documents per node. This makes the ranking scalable (see sizing):

Control the second phase candidate set size
Add content nodes to rank less documents per node

Let's now answer your questions:

  1) Let's say we have qid's in our training file. Does it mean that the optimization will be performed only on a per-query basis, all other features specified will be considered as document features and cross-query learning won't happen?

It means that in order to rank all candidates it will rank only the first ones in order to be computationally less expensive. So it will only look in the first query for the extensive ranking search.

  2)Let's assume that queries are represented by query features. Should we still have qid's specified in the training file or we should just list query, document and query-document features?

Yes, the rank_feature query is typically used so its relevance scores are added to other scores query.

Note that qid's are only used in training set https://github.com/Microsoft/LightGBM/issues/1398

Sources:
 - Wikipedia - Learning to rank
 - Vespa - Tank
 - Vespa - Phased Ranking
 - Elastic - Rank feature query


Determine Optimal CNN Complexity: Achieve ~100% Training Accuracy, Then Add Regularization? I was just wondering whether it is a decent idea to first try to make a CNN complex enough to achieve close to 100% accuracy on the training dataset, and then slowly add regularization, augmentation etc to make sure that the network doesn't overfit completely but instead is able to generalize better.

Yes, that is in my opinion the best methodology for approaching Deep Learning. This isn't supported by just me but by many researchers in the field. (I can't cite it but I'm pretty sure even Andrew Ng mentioned it in a lesson in the Deep Learning Specialization in coursera.)

The first step is to make sure you have a model with a sufficient capacity to be able to learn all the relationships (useful and not) in the training set.

Then you start applying regularization techniques (dropout, augmentation, early stopping, etc.), to prevent overfitting. During this step your training accuracy is expected to take a drop (or at least take longer to reach 100%), but the accuracy on the validation set should increase.

This methodology, however, is only applicable in the field of deep learning, because of the strong regularization techniques available.


What is the most effective unsupervised ML algorithm to use when outliers are present in data set? I am analyzing a portfolio of about 225 stocks and have gotten data for each of them based on their "Price/Earnings ratio", "Return on Assets", and "Earnings per share growth". I would like to cluster these stocks based on their attributes into 3 or 4 groups. However, there are substantial outliers in the data set. Instead of removing them altogether I would like to keep them in. What ML algorithm would be best suited for this? I have been told that K Means would not work so well since the outliers would skew the centroids of a particular cluster. Any and all thoughts welcome!

You could try a hierarchical clustering approach. As an example, K clusters could initially be found for the data points. Then, for each of the K clusters, an arbitrary number of clusters could be found from the data points within the cluster to further refine the clustering.


Difference between tf-idf and tf with Random Forests I am working on a text classification problem using Random Forest as classifiers, and a bag-of-words approach.
I am using the basic implementation of Random Forests (the one present in scikit), that creates a binary condition on a single variable at each split. Given this, is there a difference between using simple tf (term frequency) features. where each word has an associated weight that represents the number of occurrences in the document, or tf-idf (term frequency * inverse document frequency), where the term frequency is also multiplied by a value that represents the ratio between the total number of documents and the number of documents containing the word)?

In my opinion, there should not be any difference between these two approaches, because the only difference is a scaling factor on each feature, but since the split is done at the level of single features this should not make a difference.

Am I right in my reasoning?

Decision trees (and hence Random Forests) are insensitive to monotone transformations of input features.

Since multiplying by the same factor is a monotone transformation, I'd assume that for Random Forests there indeed is no difference.

However, you eventually may consider using other classifiers that do not have this property, so it may still make sense to use the entire TF * IDF.


How to identify similar data points? I am new to machine learning and I am struck at one thing. Please help.

I am developing an app to sort images based on user preferences. So initially I have n data points(images) with m features each. Now when user create an album and add some images in it then my algorithm should find similar images from the gallery and add it to the same album. This should work even if user creates more albums. And not all images are needed to be sorted, only the relevant one.

What algorithms should I use to achieve this?

Well, thats a question that institutes at univiersities, among others, are trying to solve since several decades and where millions of money currencies are spent in each country every year.

Short answer: for now, nothing general is known to perform good.

Long answer: it depends completely on your image domain. Do you want to recognize suns in a picture? That possible up to some false discovery rates. Do you want recognize familiar faces? Possible, see automated passport controls at airports. Emotion recognition? Good luck with that! Grouping by persons? Possible, but statues will be included. etc...

Something really general:  Grouping by colors? Totally viable! But will not mean much. Something that includes some hint to the structure? Well, the easiest way is to compute color histograms and use them as feature vectors for (like suggested) nearest neighbor calculation - (watch out for normalization, color space, and other sources of information distortion)

Basic answer: take some Computer Vision courses either online (e.g. https://youtu.be/skaQfPQFSyY?list=PL7v9EfkjLswLfjcI-qia-Z-e3ntl9l6vp)

There is also an intersting TED talk to that topic https://youtu.be/40riCqvRoMs

On the other side: don't let yourself down! Understand the #basics of data processing and break the structures! Develop that algorithm we are all hoping to use some day!


DBSCAN - Space complexity of O(n)? According to Wikipedia, "the distance matrix of size $\frac{(n^2-n)}{2}$ can be materialized to avoid distance recomputations, but this needs $O(n^2)$ memory, whereas a non-matrix based implementation of DBSCAN only needs $O(n)$ memory."

$\frac{(n^2-n)}{2}$ is basically the triangular matrix. However, it says that a non-matrix based implementation only requires $O(n)$ memory. How does that work? Regardless of what data structure you use, don't you always have to have $\frac{(n^2-n)}{2}$ distance values? It would still be $O(n^2)$ space complexity, no? Is there something I'm missing here? I'm working with a huge dataset and I would really like to cut down on memory usage.

You can run DBSCAN without storing the distances in a matrix. This has the drawback that each time you visit a point, you have to recalculate all the relevant distances, which requires more time. However, the space complexity stays $O(n)$, since the only things you have in memory at any single time are the positions of the n points, their various labels, the neighbors of the current point and the neighbors of a particular neighbour in the case that the point turns out to be a core point.


Does it violate the assumptions of linear regression to perform it on time series data? One of the assumptions of linear regression says that the errors must be independent i.e., the residuals must not depend on each other.

Let's say we are using linear regression to model the temperature on a given day. If it is 13:00 and 20 degrees, the temperature at 13:15 will be similar and thus depends on the time before it - it cannot suddenly fall to -20 in the space of 15 minutes. Likewise, the temperature at 20:00 will be more closely related to the temperature at 19:50 than 13:00.

Does the linear regression assumption 'independence of errors' mean that you cannot perform it on time series data?

You are correct when you say that there might be autocorrelation in the error terms when some observation in time $t$ is contingent on $t-1$. Find a rigorous discussion of the problem here.
Usually a solution to cope with this problem is to include additional variables to "control for" autocorrelation. This usually means adding lagged $y$-variables (which are called autoregressive variables or $AR(t)$). So you could simply add a lagged variable $y_{t-1}$ to your model, which reads like "todays $y$ is (among other things) explained by yesterdays $y$".
Say you have a model which is contingent on time (omitting $i$ for convenience):
$$y_t = \beta X_t + u_t.$$
You can add an autoregressive term like:
$$y_t = \beta X_t + \gamma y_{t-1} + u_t.$$
For a detailled discussion including testing procedures, please refer to the link above.


Why we prefer VIF if we can find multicollinearity from correlation matrix as well? Why we prefer VIF if we can find multicollinearity from the correlation matrix as well? What is the exact logic behind it?

The correlation matrix is not a reliable measurement for multicollinearity  because it only considers the pairwise effects. Unfortunately, multicollinearity is defined as:

  Phenomenon in which two or more predictor variables in a multiple regression model are highly correlated,

Do you see the point? You'll need to consider the correlation with all other variables in your data set, not just 1-to-1 pairwise comparison.

VIF addresses the issue.


Why apply a 50:50 train test split? I am going through the "Text classification with TensorFlow Hub" tutorial.
In this tutorial, a total of 50,000 IMDb reviews are split into 25,000 reviews for training and 25,000 reviews for testing.

I am surprised by this way of splitting the data, since I learned in Andrew Ng's course that for fairly small datasets (<10,000 examples) the "old-fashioned" rule of thumb was to consider 60% or 70% of the data as training examples and the remainder as dev/test examples.

Is there a reason behind this 50:50 split?

Is it common practice when working with text?
Does it have anything to do with using a "pre-trained" TensorFlow Hub layer?

Is it common practice when working with text?

No, you could split dataset as you wish, in general in real-world problem you should use cross validation.

  Does it have anything to do with using a "pre-trained" TensorFlow Hub layer?

No, it doesn't.


How to predict consumer purchase in next 6 months? I'm working on a model to predict a customer as being 'in-market' for a product in the next 6 months. The dataset has a wealth of information like lifestyle and demographic variables and previous transaction history. What is the best model to combine the demographic and lifestyle data with transaction history to predict a purchase in the next 6 months? Would a combination of a model that predicts likelihood to purchase with something like a survival model to predict when that purchase might occur work?

Are there any real world examples of something similar to this? Basically predicting time to purchase.

This is a forecasting problem, you are predicting a value in the future for a group of people which is the day they would make the next purchase. Sure, you would need the last time each person has bought something ( that's your target ) , periodicity of purchases;number of times that person bought something,obviously the id of the person as variables. And those are the things on top of my head, maybe someone who worked on forecasting problems could help you better.

On a second note : this is how you could approach the problem; maybe encode the dates they made the last purchase as date_num 1 to infinity, and each number in this target variable represents a consecutive day starting from a certain date(maybe the oldest date in your dataset). You would do a regression to try to predict that number using every potential predictor and round it to get an integer representing a unique day.


Theano logistic regression understanding code I am used to C/Java like programming, and sometimes I am getting a headache on understanding the Python notation.

On the logistic regression code available online, I am trying to understand this line of code:

-T.mean(T.log(self.p_y_given_x)[T.arange(y.shape[0]), y])

It is basically saying perform this average:
$\begin{align}
J(\theta) = - \left[ \sum_{i=1}^{b} \sum_{k=1}^{K}  1\left\{y^{(i)} = k\right\} \log \frac{\exp(\theta^{(k)\top} x^{(i)})}{\sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)})}\right]
\end{align}$ where $b$ is the batch size.

So is it related to theano code, or it is just a python notation ?
I am interested exactly on this piece of code:

T.log(self.p_y_given_x)[T.arange(y.shape[0]), y]

If you could please give me some explanation.

I agree the code may be hard to read at the first glance, but I find the comment pretty clear:

y.shape[0] is (symbolically) the number of rows in y, i.e., number of examples (call it n) in the minibatch
T.arange(y.shape[0]) is a symbolic vector which will contain [0,1,2,... n-1].
T.log(self.p_y_given_x) is a matrix of Log-Probabilities (call it LP) with one row per example and one column per class
LP[T.arange(y.shape[0]),y] is a vector v containing [LP[0,y[0]], LP[1,y[1]], LP[2,y[2]], ..., LP[n-1,y[n-1]]]
T.mean(LP[T.arange(y.shape[0]),y]) is the mean (across minibatch examples) of the elements in v, i.e., the mean log-likelihood across the minibatch.

Everything is Python notation (but sometimes using Theano functions).


How to perform Anomaly Detection on a force profile? I have a set of force profiles of an industrial machine. I'm trying to develop an algorithm that tries to understand when a new profile is "anomalous" with respect to the ones in "normal operating conditions".
In the picture below you can see the force profiles (function of time). I want the blue curve found checked as anomalous.
What approaches do you suggest? I'm thinking about using some statistical distance (like mahalanobis) to check the similarity of a new curve to the "mean" of the others.
Another point: it can be useful to use approaches like PCA in this case? Then I can try to use clustering techniques to separate anomalous profile from normal ones. I'm not sure since I have a great number of observations but of the same variable and PCA is a multivariate technique.
]1

I strongly recommend to start with a simple statistical data analysis. In this approach you take the Moving Average/Median of signals and if one signal shows a magnitude -/+ 3 times standard deviation, you mark it as anomaly. Please have a look at this answer for Python code.


Grid search model isn't recognized as fitted for Graphviz I find this really weird, and the code is really straight forward.
What am I doing wrong ?

from sklearn.model_selection import GridSearchCV
scoring_type="accuracy"
preprocess_data(X,y,0)
p_grid = {'min_samples_split':np.arange(2,10),'min_samples_leaf': np.arange(1,10)}
Tree_opt = GridSearchCV(estimator=DecisionTreeClassifier(random_state=42), param_grid=p_grid, scoring=scoring_type, cv=10)
Tree_opt.fit(X_train,y_train)

print("Best training params: {}".format(Tree_opt.best_params_))
print("Best training Score: {}".format(Tree_opt.best_score_))
dot_data = tree.export_graphviz(Tree_opt, out_file=None,feature_names=labels,class_names=class_names,filled=True, rounded=True,special_characters=True)
graph = graphviz.Source(dot_data)

graph
print(Tree_opt.score(X_test, y_test))

And I'm getting the following error:

NotFittedError                            Traceback (most recent call last)
<ipython-input-14-1c7fa906f99b> in <module>
      8 print("Best training params: {}".format(Tree_opt.best_params_))
      9 print("Best training Score: {}".format(Tree_opt.best_score_))
---> 10 dot_data = tree.export_graphviz(Tree_opt, out_file=None,feature_names=labels,class_names=class_names,filled=True, rounded=True,special_characters=True)
     11 graph = graphviz.Source(dot_data)
     12

~\Anaconda3\lib\site-packages\sklearn\tree\export.py in export_graphviz(decision_tree, out_file, max_depth, feature_names, class_names, label, filled, leaves_parallel, impurity, node_ids, proportion, rotate, rounded, special_characters, precision)
    394                 out_file.write('%d -> %d ;\n' % (parent, node_id))
    395
--> 396     check_is_fitted(decision_tree, 'tree_')
    397     own_file = False
    398     return_string = False

~\Anaconda3\lib\site-packages\sklearn\utils\validation.py in check_is_fitted(estimator, attributes, msg, all_or_any)
    949
    950     if not all_or_any([hasattr(estimator, attr) for attr in attributes]):
--> 951         raise NotFittedError(msg % {'name': type(estimator).__name__})
    952
    953

NotFittedError: This GridSearchCV instance is not fitted yet. Call 'fit' with appropriate arguments before using this method.

I also have this warning I don't think it's important:

C:\Users\Flow\Anaconda3\lib\site-packages\sklearn\model_selection\_search.py:841: DeprecationWarning: The default of the `iid` parameter will change from True to False in version 0.22 and will be removed in 0.24. This will change numeric results when test-set sizes are unequal.
  DeprecationWarning)

I think the fit works because I can get the first prints as follows:

Best training params: {'min_samples_leaf': 3, 'min_samples_split': 2}
Best training Score: 0.8809523809523809

Try using Tree_opt.best_estimator_.  The grid search object passes fit and score functions through to the best estimator when refit=True, but other (esp. more specific) methods need to be called directly from best estimator object.


Is there any ongoing research on known patterns ML algorithms can not mine? I have been doing ML for sometimes and have explored and implemented Deep Learning as for some projects. ML algorithms assume a certain pattern (Geometric/Bayesian/Distribution) in the data .This can usually be referred as Hypothesis Space in the literature, then we design a solution to best fit that hypothesis to our data.

I wanted to know are there some patterns (Hypothesis spaces), which do not yet have a reliable algorithm to be mined?

Like Decision Trees used the Information Theory to assume that Information Gain from a variable can help us better classify the data. This was a new approach than SVM which assumes a geometrical space to where a plane can classify the data. I hope there must me some new theories which are being worked on.

Let me know if I can improve this question.

The closest I can think of is research into "adversarial" transformations of data (usually images) that cause classifiers (usually CNNs) to fail. Here's a blog post which discusses a bit about how these transformations are constructed, some approaches that have been tried to address them, and what some of the consequences of their existence might be.

You might also be interested to read about the No Free Lunch Theorem.


Can CNN be used with 3d dataset My dataset is a simple table of 20 columns and 100,000 rows.It is not a image data as commonly used in CNN. What input shape should I provide in this case?

Right now I did-

input_shape = (21,109713,1)

model.add(Conv2D(32, kernel_size=(5, 5), strides=(1, 1),
                 activation='relu',
                 input_shape=input_shape))

which gives the error-

ValueError: Error when checking input: expected conv2d_1_input to have 4 dimensions, but got array with shape (109713, 19)

The question in the title is inconsistent with the question body. If you are intent on using a CNN albeit it not being the best idea for this type of data, then for the dataset you are describing a 1D CNN architecture would be most suitable.

To answer the question in the title, yes a 3D CNN is definitely possible and is actually commonly used on 3D data, for example: video (3rd dimension is time), 3D medical images, etc.

The instances in the dataset you described is a vector with 20 instances. This data is a single dimensional structure. You can thus use a 1D CNN as described here.

However, it would be best to use a DNN for such a data structure. As a CNN is essentially cross-correlation of neighboring features. In this case, your data cannot be assumed to have such similarities between adjacent features. A DNN will allow for cross-correlation between distant features to be captured more effectively.


Appropriate similarity measure that highlights symmetrical values of features I am trying to find an appropriate distance measure that reflects the differences of the vectors seen in the image below:
The green vector is compared with the blue one and the orange one.

Most of the distance measures (like the Euclidean for example) would yield the same value despite the "phase shift". Therefore the straight line would look "the same" as the blue and the orange line!

The features are ordered but I am also interested in the case they were unordered.

I was also thinking to "split" the vectors in the middle and treat them as bags of two subvectors (like in a multi-instance setting) but then I would not know how to combine the two resulting distances in one.

So I guess my question is two-fold:

Is there a distance measure that highlights these type of value symmetries in a single instance setting?
Is there a way to combine multiple distances (in a multi-instance setting) in a way that symmetry (and therefore order) is highlighted in the final distance result?

Thank you

Is this an issue of features?  From the explanation, it seems that the feature you are considering is distance. So the results are as expected (from a "bag of single feature distances").

You mention that order is important, so that can added as a feature.  Or since you mentioned "phase shift", time (and/or frequency) -- giving you a time series.  In both cases, you then can bring in all kinds of tools.  You could start with regression and go from there to more complex methods (simple sometimes is best).

That would allow you to yield similarity metrics between single or multiple vectors.

Vectors contained in a sliding time window vs. population, is something you maybe can also consider, we did a solution like that once.


Which metric to choose for tracking model performance? I am working on a binary classification problem with class proportion of 33:67.

Currently what I am doing is running multiple models like LR,SVM,RF,XgBoost for classification.

RF and Xgboost models perform better.

However I am reading online that AUC, F1-score or Accuracy comparison between models may not be a good metric (as they are sensitive) to measure the model performance.

I see these are the mostly used scoring metrics but can you let me know which metric should I look for measuring binary classification model performance?

Can you let me know why should we choose a different metric?

These are the metrics that I see in scikit-learn

TL;DR

I would suggest you to:

Balance your dataset (since you are not doing an anomaly detection task, that should be fine).
Measure Precision/Recall/F1 Score and argue for them depending on your problem.
Perform K-Fold Cross Validation to compare models

Balancing your dataset

You said you have a 33:67 ratio. Why keep it like that an not just make it 50:50 (e.g. by removing samples from the larger class)? In general this help your models be less biased to the larger class.

Special care is needed when doing anomaly detection, where ratios are usually around 1:99. But since you are not doing that, simple dataset balancing is advisable.

Precision / Recall / F1

F1 score is the default score to go to for classification. It should be ok to use across models as long as you keep your test set the same for all models. An even better thing to do to compare models, is to perform K-Fold Cross Validation on each model. (Sklean docs).

However, you could look at Precision or Recall instead of the F1 Score, but this will depend on the problem you are trying to solve.

  Precision = when I detect an instance of a class, I am sure it is the right class.

  Recall = I detect all instances of a class, even if I detect instances that don't belong.

For example: we are performing a binary classification of whether a patient has a deadly disease or not. Health Insurance: "let's optimize the precision, because we only want to pay treatment for patients that really have the disease". Doctor: "let's optimize for recall, because it is better to treat more people even if the don't have the disease to ensure that those that do have it are likely being taken care of".

If you think that precision are recall are equally important then that is the definition of the F1 score.


Why leaky relu is not so common in real practice? As leaky relu does not lead any value to 0, so training always continues. And I can't think of any disadvantages it have.

Yet Leaky relu is less popular than Relu in real practice. Can someone tell why?

Speaking from my experience, the performance of the two is almost the same. It might depend on the problem.

LeakyReLU was introduced to address the vanishing gradient problem, however it introduces yet another hyperparameter, the slope. If you want to squeeze out a little bit more performance of your model you can use LeakyReLU and tune the slope parameter, but that comes again of the cost of potential overfitting.


Prediction model for marketing to prospective customers (using pandas) I'm currently working on a part-time project which involves predicting the likelihood of customers going to buy a product using data analytics. The company I'm interning with has given me a customer CSV file with all current customers and their attributes and needs to make a prediction model to classify whether prospects are feasible to pursue or not.

However since they have given me a list of all their successful customers or leads, in marketing terms, is it possible to train a model like K-means with PCA (and k-fold cross validation?) and get results? I have to train my model to fit a value, say 10, which I will add to the CSV, and further test it.

I am using pandas. Another issue is that there is a lot of demographical data, but I managed to overcome it using get_dummies(). The number of columns escalated from about 10 to 47, though.

I'm just entering into the world of data analysis, hence I'm a bit clueless as to what path to take or whether what I'm doing is right.

The exact analysis is called Predictive Lead Scoring/Analysis, in marketing terminology.

EDIT 1

I followed what @HonzaB did and, hence did get a decision tree. However, since I had 40 columns, it looks like this

I had to take a screenshot of it, as it was over 2 MB.

Obviously it's really big, and I have to prune the tree somehow, but I not sure how to do so on pandas. Also, is there any way that I can just generate the best characteristics as a text file or something that can be understood without the help of a data scientist?

EDIT 2

I've read up on a question that is quite similar to what I need to do. Predictive modeling based on RFM scoring indicators. In it there is a link to a paper([Data Mining using RFM Analysis][3]) that talks about rule-based classification. Ideally this is what I need to do, and what is most suitable to the company's need.

I want to know if it's possible to do this on Python/pandas. Or is it possible to traverse the decision tree and generate the rules?

EDIT 3

I found another website Decision trees in python again, cross-validation that uses cross validation and hyperparameter optimisation to get a better solution. Also they have included Python code to get readable code. It's a feasible solution, however it's quite complicated and I can't understand how it works. Will it work?

PS I solved the "really big decision-tree" problem from Edit 1, by reducing max-depth. I didn't know at all.

First, I would ask the company if there is more information about the customer. You mentioned you have 10 original columns, which might not be enough to make a good prediction. Same goes for number of rows. Usually, more data, better the model, up to a certain limit.

Second, encode categorical features (demographical data in your case) is good thing to do. The increased number of columns dont have to bother you in your case.

For the task itself, yes, it is doable. Start easy, simply check importance of each feature (I would let PCA for later), pick few models and test them.

Also consider train simple decision tree. Your results can be easily visualized in way the business people understand. As oposed to black-box methods as K-Means.


Cleaning NaNs with averages pre or post split? I have a column with some NaNs in it and I want to replace those NaNs with the average/median/mode.
Technically, the validation/ test data has never been seen before - so how could I include it in the average? That would bias the values.
Do I "fit" the average to my training data only, just like scaling? Or do I take the average using the entire dataset?

Imputing all missing values before the split would mean data leakage - you would use test set information to influence the training set.
I had a similar problem recently, I ended up using complete cases of the source data for the test set and then imputing the training data using medians calculated by class.
Otherwise, you might split the data and then impute the test set using the training set average/median/mode.


Does using unimportant features hurt accuracy? I'm using the scikit-learn gradient boosting classifier found here.

If I run the classifier on the same data without seeding the random number generator, I get different feature importances, and notably, different ones are 0 each time. If I just look at the features where the feature importance is < .005, most of the same ones appear, but there is a little variation there too.

My question is, does this have an impact other than computation time? If it were to hurt accuracy, I imagine that would be due to over-fitting. However, if the classifier already identifies the relatively unimportant features, then doesn't that take care of the problem?

If it does hurt accuracy, how should I select which features to get rid of?

I used the gradient boosting classifier in a project a while ago. We had about 130 features and since the system had to be implemented on a microcontroller I tried to reduce the number of features as far as possible.

Here is a plot showing the performance (F1-score) of GBC models trained with the top n most important features:

From that image it looks like "good-enough performance" is already reached with only 30 out of 130 features, while the other 100 (unimportant) features don't seem to have much positive nor negative influence.

Only when you zoom in a bit on the Y-axis you see this:

This shows that the performance reaches its peak (0.93) at 90 out of 130 features. So 40 features are unimportant if you want the best model.

To answer your question "does this have an impact other than computation time?": it does have an impact on performance, since the performance decreases again with more than 90 features, but only slightly.

To answer your second question "If it does hurt accuracy, how should I select which features to get rid of?":

I did the following:

Select the top n features
Use a cross-validated grid search to train a GBC model on the top n features
Repeat with the top n+1,... features
Select the model / features with the best score

How to select the top n features in step 1):

I actually didn't use GBC for feature selection, but random forests. I don't know why, but my experiments have shown that selecting features with RF and then training a GBC model on them works better than selecting features with GBC and then training a GBC model on them.


Curvature - Linear Assumption They asked if the linear assumption is correct, but I just see a graph and based on the data I was taught that a curvature is nonlinear so nonlinear means that a graph is incorrect?

What is the assumption? I know this may seem stupid, but I still haven't grasped the question.

Yes, the linear assumption is incorrect since the graph is non linear


Python: Plotting equation with the Dirac Delta function I am attempting to plot the expected diffraction pattern from a sharp razor which has a formula of:

$$ I(p) = \frac 14 \delta^2(p) + \frac {1}{4\pi^2p^2}$$
where $p = \frac {\sin\theta}{\lambda}$

I initially want to just make a simple list of x values and feed them into the equation to get y values and plot those, like so:

x = [theta for theta in range(-90,90)]

y = [I(theta) for theta in x]

plt.plot(x,y)

However, I'm not sure how to write $I(\theta)$ because I don't know how to write a dirac delta function in python. How would one graph this function?

Use the scipy library function
in this example
import numpy as np
from scipy import signal
number_points =100
location_of_impulse= 60
it_theta=signal.unit_impulse(number_points ,location_of_impulse)
you'll have to do your own scaling
By the the way the following is a direct copy from the SCIPY sit and may be more illuminating
..........................................................................
import numpy as np
from scipy import signal
import matplotlib.pyplot as plt
imp = signal.unit_impulse(100, 40)
b, a = signal.butter(4, 0.2)
response = signal.lfilter(b, a, imp)

plt.plot(np.arange(-50, 50), imp)
plt.plot(np.arange(-50, 50), response)
plt.margins(0.1, 0.1)
plt.xlabel('Time [samples]')
plt.ylabel('Amplitude')
plt.grid(True)
'


If in t-SNE digaram of binary classification both classes follow the similar curve what does t-SNE diagram show? If in t-SNE digaram of binary classification both classes follow the similar curve  what does  t-SNE diagram show for instance: Figure1 or  Figure2

To have the intuition, let's make simplify the problem. Let's say that you are using PCA and that you are getting a linear function.

If you don´t get any cluster it means that it is not possible to separate them by linear combinations of the features that you have. And since it is a linear function, as you move in this direction there is a change in the variance.

t-SNE is similar to PCA but it does non linear combinatios of features. And the dimensionality reduction that you have mantains the variance in a non linear way.
So just change the world linear for non-linear in the second paragraph to answer your question.

By the way, this doesn't mean that if you make a classification problem with Machine Learning you are going to have bad results. ML can still work in this case.


How to find a confidence level given the z value I have a task from which I would like to find the confidence level given the z value. I have a sample population. From that population, given its distribution, I would like to find the confidence level of a given value of that population. In other words, given a a value of the population, I would like to know if it is within 95% (confidence level) of the whole population or 68% or 50% and so on. Usually, we can find the z value and confidence interval and given confidence level as explained here How to find the confidence Interval. But I would like to find the confidence level given the z value (which in this case is a given value from the population).

How can I tackle this? If possible it should be in python or in R

OK, for a 95% confidence interval, you want to know how many standard deviations away from the mean your point estimate is (the "z-score").  To get that, you take off the 5% "tails".  Working in percentile form you have 100-95 which yields a value of 5, or 0.05 in decimal form.

Divide that in half to get 0.025 and then, in R, use the qnorm function to get the z-star ("critical value").  Since you only care about one "side" of the curve (the values on either side are mirror images of each other) and you want a positive number, pass the argument lower.tail=FALSE.

So, in the end, it would look like this:

qnorm(.025,lower.tail=FALSE)

yielding a value of 1.959964

You then plug that value into the equation for the margin of error to finish things up.

If you want to go the other direction, from a "critical value" to a probability, use the pnorm function.  Something like:

pnorm(1.959964,lower.tail=FALSE)

which will give you back 0.025


For a regression model, can you transform all your features to linear to make a better prediction? I was thinking. Would it be a good approach to check your features one by one (assuming you have a manageable amount of them)  and see the relationship they have with your target variable, if they have a non linear relationship then transform each of those features using their appropriate function for each case to make them linear? In my mind if you do this your are guaranteed to have a better Linear model and also you are able to perform hypothesis testing on each feature to see the relevance of them, giving you the chance to perform some feature selection as well.

I know that the interpretability of model will be thrown out of the window, but the model will give a much better performance. Basically you could potentially end up with a model with only engineer features (assuming that all of them have a non linear relationship)

Would this approach be acceptable and it is worth exploring?

Your idea is good, but you are not the first with this idea.

You can use Generalized Additive Models (GAM) with regression splines to check and/or add non-linearity in a linear regression setup. There is a clear advantage over looking at just descriptive figures one-by-one (with manual feature generation), since you can estimate a whole model with extreme flexibility.

Alternatively, you can simply do a linear regression with a lasso penalty. Add polynomials to your $X$, and let the lasso „shrink“ irrelevant features to zero.

The book „Introduction to Statistical Learning“ covers these topics in Sections 6,7.

BTW: even with polynomials, your model can retain interpretability if you care for this. Polys are relatively easy to interpret. However, under a GAM approach, interpretation is a little more difficult.

Maybe as a note: you want to make features linear (which can be a problem since you can only apply linear transformations to the data). The approaches proposed above aim at making your (linear) model more flexible to cope with non-linearity in data.


The main purpose of batch normalisation is not for dealing with overfitting but if you have small batches while training it can have regularization effect. In the paper that it was introduced for dealing with covariat shift, it was mentioned that it should be used before activation function. Consequently, you can use it both in convolutional layers and dense layers, after employing weights and before activation functions. But I've seen people using that after activation function. It can also be used there.


Comparing TFIDF vectors of different shapes I'm working on a project using TF-IDF vectors and agglomerative clustering -- the idea is that the corpus of documents increases over time, and when a new document is added, the mean cosine similarity with each cluster will be calculated to find the best match.
However, I'm under the impression that it is costly and inefficient to re-calculate the TF-IDF vectors of every document each time a new doc is added to the corpus.
I'm trying to discern if it's possible to generate the TF-IDF vectors on an individual basis for each document (see below), and then calculate the cosine similarity by temporarily reshaping/modifying the two vectors.
doc_1 = ['some', 'text', 'here']
doc_2 = ['other', 'text', 'here']

vectorizer = TfidfVectorizer()

vector_1 = vectorizer.fit_transform(doc_1)
vector_2 = vectorizer.fit_transform(doc_2)

# somehow add the features from vector_1 to vector_2 that do not exist there, and vice versa...

# this way vectors can be calculated on the fly without having to regenerate vectors with an ever-expanding corpus

It's my understanding that running the sklearn TF-IDF vectoriser returns not only the vectors but a mapping between the unique words in the corpus and the vectors they correspond to, so I think this approach would be possible, but I'm not sure?
Finally, as I'm new to data science, am I approaching this issue horribly? Is there a much cleaner or efficient way of solving the problem of non-parametrical clustering (i.e. no n_clusters) with an ever-expanding corpus.

Your thinking is correct, it is unefficient to recalculate TFIDF on the whole collection every time a document is added.
There are two parts in TF-IDF:

$TF(w,d)$ is the Term Frequency of a word $w$ in a particular document $d$, it's independent from other documents. It's just defined as the proportion of $w$ in $d$.
$IDF(w)$ is the Inverse Document Frequency: it is defined as the log of the inverse of the document frequency, where the document frequency of a word is the proportion of documents in the collection which contain this word:

$$IDF(w)=log\left(\frac{|D|}{| \{d \in D \text{ such that } w\in d \} |}\right)$$
$IDF(w)$ depends on all the documents so it's costly to re-calculate. However updating IDF whenever there is a new documents can be implemented efficiently:

Maintain a global map of the document frequency (not the IDF) for every word. This map is easy to update: for every distinct word $w$ in the new document, increment $m[w]$.
Whenever a TF-IDF weight is needed, collect the required document frequency and apply the IDF formula.

I don't know whether this can be done with scikit API or not, but it's not too big a task to implement it yourself (and it's a good little exercise to understand TFIDF ;) )


Document parsing modeling and approach? I'm relatively new to data science / machine learning (yes, I know) and am experimenting with text analysis. I only want a relatively naive approach and am looking to know whether my approach is valid procedurally even if I may not get perfect results. I want to compare my model to a fulltext tf-idf search of specific skills/keywords against an index of resumes.

The task: Classify job descriptions, rather cluster them in an unsupervised way. Eventually I want to match resumes against them as a basic implementation of my idea. I was thinking that a naive implementation of word embeddings and kmeans would be fine. Basically, extract document_vectors from specific job descriptions, then cluster the job descriptions based on these vectors. This would create a ghetto "labeling" system so that I could then apply a classification algorithm Then I would use resume raw_text and extract a new vector, applying a simple classification it into one of these clusters (regression).

To summarize

1. Use a large database of job descriptions
2. Tokenize the text, and apply doc2vec and generate document vectors
3. Cluster the documents and group similar job descriptions
    NOTE: as to how many clusters to use, that will have to be tuned
4. Tokenize the raw_text of a resume
5. Use regression (or some technique) to find the closest cluster based on the resume vector compared to the cluster vector.

Now, I realize there are most likely some huge holes in the way I'm doing this (I'm using intuition based on no training). For example, my assumption that I can even match resume to job description is based on the idea that these are even comparable text structures - and how good the matching is is also dependent on how good the clustering of documents is.

I would like to know if this approach is good for something naive or whether I'm missing anything critical. I'm using nltk, scikit-learn, and gensim Word2Vec

What you say seems reasonable.

You can also try to create the document embeddings as you mention and use k-Nearest Neighbors to find the nearest job descriptions to a given resume based on some distance, without any clustering of job descriptions. There are some approximate implementations of k-Nearest Neighbors that work very fast like https://github.com/nmslib/nmslib.


Single Layer Perceptron with three classes I need some help with a single layered perceptron with multiple classes.

What I need to do is classify a dataset with three different classes, by now I just learnt how to do it with two classes, so I have no really a good clue how to do it with three.

The dataset have three different classes: Iris-setosa, Iris-versicolor and Iris-versicolor.

The url with the dataset and the information is in : http://ftp.ics.uci.edu/pub/machine-learning-databases/iris/iris.data.

I really appreciate any help anyone can give to me.

Thanks a lot!

The standard way to do this is called 'one versus all'... you train three perceptrons. First 1 target = is class a?, 2nd perceptron target = is class b? 3rd = is class c. You just train each perceptron separately, and then take max of the three perceptrons to decide class


Keras model producing same output I've seen a couple questions that have a similar problem, but none of them solved mine. I'm trying to fit a neural network in Keras to a dataset with 22 input features for binary classification. The problem is that I only have 195 training samples. I know it's a small dataset, but I don't know if it's possible to fit a model with reasonable accuracy (I'm aiming for >95% accuracy). The problem I'm having is that my model is only outputting 1 and getting 75% accuracy because my dataset is 75% positive cases. Here's the code I have:

data = pd.read_csv("") #filename omitted, but it loads properly
scaler = MinMaxScaler()
X = scaler.fit_transform(X)
Y = data['status']
X = data.drop(['status', 'name'], axis = 1)
xTrain, xTest, yTrain, yTest = train_test_split(X, Y, train_size = 0.8)
model = Sequential()
model.add(Dense(48, input_shape=(22,), activation = 'relu'))
model.add(Dropout(0.5))
model.add(Dense(1, activation = 'softmax'))
optim = keras.optimizers.adam(lr=0.0001)
model.compile(optimizer = optim, loss = 'binary_crossentropy', metrics = ['accuracy'])
model.fit(xTrain, yTrain, epochs = 20, batch_size = 5, validation_data = (xTest, yTest))

I've tried adding more hidden layers, increasing the number of training epochs, and increased and lowered the optimizer's learning rate, but the accuracy stays the same. Here's the link to the dataset: https://www.dropbox.com/s/c4td650b4z7aizc/fixed.xlsx?dl=0

EDIT: I fixed my problem by applying SMOTE to balance my dataset. The accuracy went up to 96% after I did this and added more hidden layers.

It happens often, that some model only predicts one class. The reason usually is, that the model is unable to distinguish the two classes well, and resorts to one (often the majority) class.
With your data, you may have a hard time to fit a NN with reasonable results. I suggest you check boosting which usually works okay with small data. Make sure you tune parameters well.
Here is an application of lightgbm to the iris data.
You could also check Logit with L1 regularization.


Predicting complete time series for given parameter sets I am searching for an approach that is able to predict a complete time series for a given parameter set.
Imagine a robotic arm which has a starting position and a target position. There is  a sensor mounted to the top of the arm. For random start and target positions I can record the sensor values as a time series. Now, I would like to predict the sensor values for a previously unknown start and target position.
Currently, the only approaches I can find try to forecast the next values for a given time series. Other approaches aim for generating new sequences for a given set of sequences, which is close to what I would like, but without the possibility to connect the sequences to parameter sets.
Does anyone have an idea?

Check the literature on functional data. Your function here comprises the sensor values over a complete path of the robotic arm. From a random sample of such functions you are wanting to predict a new function with specified start and target positions. A good intro is Ramsay and Silverman (2005).


Some questions about GoogleNet paper This phrase is from the "Rethinking the Inception Architecture for Computer Vision" paper. it says :

  Higher dimensional representations are easier to process locally within a network. Increasing the activations per tile in a
  convolutional network allows for more disentangled features. The
  resulting networks will train faster.

What does the phrase

  Higher dimensional representations are easier to process locally within a network"

mean?

Does it simply mean, a higher representation is achieved more easily by having more feature-maps in a layer? again indicating, having lots of feature-maps per layer, (more width) is more beneficial?
If so, does the next part indicate the same thing where it says :

  Increasing the activations per tile in a convolutional network allows
  for more disentangled features.

what does disentagled features mean in that regard?

"Higher dimensional representation" refers to having more feature maps like you suggested (check the inception module proposed in figure 7).

As to what disentangled features mean, I believe they mean decorrelated: the more features (with different filters) the inception module extracts, the more and the faster the network learns. Because the network will have access to more information leading to detecting which features are salient earlier in the training.


Encoding feature with tree-path data What are the common techniques for encoding discrete features with tree-path value (e.g. file/folder path) without expanding tree into multiple features corresponding to each node?

Update.

Overall, I am looking for encoding which can be used with different algorithms (e.g. linear regression, not only clustering).
Something like Huffman coding but usable with machine learning.

For example, consider this tree described with lisp-like notation:
(a (b b1 b2) (c c1 c2 c3))

It could be encoded by creating feature for each node
but this will be very expensive for any sufficiently large tree.

a  b  b1 b2 c  c1 c2 c3 Path
1  1  0  0  0  0  0  0  /a/b
1  0  0  0  1  0  1  0  /a/c/c2

Instead of creating feature for each node,
there could probably be a feature for each depth level.

L1 L2 L3 Path
1  1  0  /a/b
1  2  6  /a/c/c2

where
L2 can have value of 0,1,2 for none,b,c correspondingly;
L3 can have value of 0,1,2,5,6,7 for none,b1,b2,c1,c2,c3 correspondingly
(with gap between b2, and c1 so that there is numerical distance between nodes with different parents).

There are some obvious problems with it though,
e.g. c1 and c3 are further away from each other than c1 and c2.

I understand that encoding really depends on how you want to interpret the tree and relation between nodes. I was hoping there might be some article/paper which describes different approaches and trade-offs.

Clustering

I think a tree is a perfectly appropriate data structure in this case. You don't need an embedding to do clustering; there are similarity-based approaches too, and defining a similarity function in your case is straightforward: I'd say it's a function of the common depth between the two items in consideration. You will probably want to define a root to "normalize" the scores. For example, given the root "a/b/" and the paths:

a/b/c/d/foo1
a/b/c1/foo2
a/b/c1/foo3
a/b/c/d/foo4

The shared depth between foo1 and foo2 and 0, discounting the root. For foo1 and foo4 it is 2. For foo2 and foo3 it is 1. You can define the similarity as a transformation of this through the function $f:x \to 1-\exp(-ax)$, where $a$ is a parameter you can use to tweak the clusters.

For implementation, in python, you can try sklearn.

General

You can featurize a path string by creating a set from its parent folders, which can then be represented as a sparse bit string by hashing the set elements. For example, "a/b/c/d/foo1" becomes ("c", "c/d"), if we let "a/b" be the root, as before. (The notion of a root is not strictly necessary here except to ensure that the baseline similarity is zero.) The path similarity is simply then the set similarity or, after conversion to bit strings, the $L_p$ distance.


Regression Algorithms in Production I am interested in predicting if a doctor would prescribe a specific drug and have chosen Logistic Regression as a starting point.

I have a few questions:

Is feature selection the first step to take to choose relevant
variables?
Is Logistic Regression only for binary output? For each doctor, can I get probability of drug prescription (e.g. doctor 1 = 0.87, doctor 2 = 0.56)?
How can my model be deployed into production? Is it a huge task?

I would not start with (manual) feature selection. Use Lasso instead to "automatically" shrink/select features (this is Logit with shrinking of features basically). Logit (or Logit with Lasso as here) is for binary cases, but you can also do "Multinominal Logit" (option multi_class='multinomial' in sklearn), which is for more than two classes. Usually you use sklearn in Python for such things. Also see the examples in the sklearn docs.

Make sure you have a test and trainings set. Also make sure that you do not use data from your test set for training. Train on the train set only and use the test set to see how your model performs on data NOT seen during training.

It is not clear what you mean when you say "move to production". This depends on your problem. You simply need to make predictions here, but the implementation is of course contingent on the environment.

It is okay to play around with data. However, if you really want to go for serious data science, you should have a look at the methods behind all this magic. I recommend "Introduction to Statistical Learning". It is a really good book with many code examples and it is not very technical.

Note that there is no silver bullet. Lasso or Logit may be okay, but other methods may be better. This really depends on the problem/data.

Here is a little sample code for Lasso:

# Split test/train
from sklearn.model_selection import train_test_split
xtrain, xtest, ytrain, ytest = train_test_split(X, y, test_size=0.20, random_state=7)

from sklearn.metrics import roc_auc_score
from sklearn.metrics import confusion_matrix
from sklearn.linear_model import Lasso
from sklearn.linear_model import LassoCV

# Perform lasso CV to get the best parameter alpha for regulation
lasso = Lasso(max_iter=10000)
lassocv = LassoCV(alphas=None, cv=10, max_iter=10000)
lassocv.fit(xtrain, ytrain.values.ravel())

# Fit lasso using the best alpha
lasso.set_params(alpha=lassocv.alpha_)
lasso.fit(xtrain, ytrain)

# Look at results (coefficients)
la1 = pd.Series(abs(lasso.coef_), name="lasso")
la2 = pd.Series(X.columns, name="names")
dflasso = pd.concat([la2,la1], axis=1)
dflasso = dflasso.sort_values(by=['lasso'], ascending=False)
print(dflasso)

# Look at AUC
print("AUC Lasso: %.3f" %roc_auc_score(ytest.values, lasso.predict(xtest)))

# Predict probs
lasspreds0 = lasso.predict(xtest)
# Classes
lasspreds = np.round(lasspreds0)

# Confusion matrix
tnlog, fplog, fnlog, tplog = confusion_matrix(ytest, lasspreds).ravel() #y_true, y_pred
print("True negative: %s, False positive: %s, False negative: %s, True positive %s" %(tnlog, fplog, fnlog, tplog))
print("Share false %.2f" %(((fplog+fnlog)/(fplog+fnlog+tplog+tnlog))))

# Look at probs
print("Min. prob. of belonging to class 0: %.3f" %lasspreds0.min())
print("Max. prob. of belonging to class 0: %.3f" %lasspreds0.max())

EDIT:

Please note that the sklearn Lasso as described above does not do a logistic regression, which means predictions can be smaller zero or larger one. To use Lasso with Logit (ensuring predictions are zero or one), one can use LogisticRegression:

from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_iris
X, y = load_iris(return_X_y=True)
log = LogisticRegression(penalty='l1', solver='liblinear')
log.fit(X, y)


In Incremental Learning will the model be updated automatically? I came across Incremental Learning algorithms paper, where incremental algorithms are compared. I have problem with general understanding. Will the model be updated /adapts itself automatically when new data comes in?

Does it know by itself that new  data has arrived and it learns?

In general, can anyone explain how training, testing, and model adaption is carried out with such incremental algorithms?

Incremental learning is analogous to online learning. It is based on the assumption that your model can receive a continuous stream of data from which it can keep learning indefinitely. Training is therefore based on Mini-Batch Gradient Descent optimization: you feed batches of data into the Network as soon an new data comes in.

Hope this helps, otherwise let me know.


Using pretrained LSTM and Bert Models in CPU Only Environment - How to speed up Predictions? I have trained two text classification models using GPU on Azure. The models are the following

Bert (ktrain)
Lstm Word2Vec (tensorflow)

Exaples of the code can be found here: nlp
I saved the models into files (.h5) for later use. The files are big e.g. 27,613kb for the lstm and 1.2 gb for bert.
I loaded the models and in a computer where only CPU is available. They both work fine but the model.predict(text) function is super slow predicting the class of the text e.g. on average 1 tweet sized message per second.
Adding GPU on the computer is not an option. I wonder if there is another way to make it run faster? e.g. train the models in a different way (without compromising accuracy) or save the model in a different file format?

There are a couple of options:

Optimize tensorflow for your specific CPU. Sometimes the official versions of tensorflow are not compiled with support for some instruction sets (e.g. SSE4.1, SSE4.2, AVX, AVX2, FMA). Usually, there is a tensorflow runtime warning message stating so. This prevents some computations to take place in parallel. You can either download a version that is optimized for your CPU (e.g. from intel) or you can compile tensorflow yourself.

Prune the model. BERT is a multi-head attention model, and it is possible to remove some of its attention heads while retaining most of the quality. Here's a blog post that shows how to do it for tensorflow.


nlp - opinion mining vs sentiment analysis I have been told that nlp possibly holds the key for allowing researchers to infer the affective state of a person when writing. For instance, by using nlp analysis on online note taking you could infer whether a student is stressed. I work in educational cognitive science, so if this were the case it would be a valuable resource for me, however I am struggling to find evidence that this is indeed the case.

I have begun preliminary research and am trying to learn a bit about the technical aspects of nlp - I'm taking an online course by Jurafsky and Manning, another by Michael Collins - and I have been reading about what what can be inferred using nlp, specifically around opinion mining and sentiment analysis.

My question is two part:

Firstly, most resources I have come across say something along the
lines of nlp can be used for opinion mining and sentiment analysis
and we will talk about the implications for opinion mining'. Can
someone point me in the direction of more sentiment analysis oriented
resources?
Secondly, as I understand it, sentiment analysis is deriving the
stated sentiment within text, e.g. I was happy with, I loved, was tasty, enjoyed, hated, frustrated, etc. Can nlp, or something else,
be used to derive unstated affect? And is this the same thing as
sentiment analysis?

[Apologies if this is asked in the wrong exchange. I've been trying to find the best fit, but there were a few candidates and I wasn't sure which would be most appropriate.]

Can someone point me in the direction of more sentiment analysis
  oriented resources?

There are tonnes of exceptionally good papers on sentiment analysis for your reference. Especially the ones on social network sites like Twitter.

However, I think the book by Manning and his course are a nice start as references to NLP and opinion mining.

  Can nlp, or something else, be used to derive unstated affect?

Yes, it can be. This would be my approach:

(Consider I already have a hand-classified training data) Using the bag of words approach, the most used words and phrases for a mood (stressful, jovial, etc) can be captured, and the subsequent training data can be ranked accordingly as (60% stressful, 40% jovial).

Additionally, you can also define thresholds for getting a single mood. Like:

if mood[score] > .5:
    current_mood = mood


How to normalize complex-valued data? I'm taking the abs of all elements, compute the mean, subtract it off from the original values. I just feel that this is not correct and can change the vectors.
I'm also dividing by the standard deviation, but I'm quite confident about this, knowing that this is pure rescaling of complex values.

Any guidelines on how to do this?

PS: My concern eminates from thinking of each complex valued element as a vector in 2D plane, and this subtraction could change where it is pointing to.

First off, it's always helpful to think geometrically about what complex numbers are, and what arithmetic operations achieve.

In your function, you are using the mean and standard deviation of the absolute value of these complex numbers. That means that if you perform your operation to the absolute value of your data:

(tmp - tmp.mean()) / tmp.std()

you will end up with normalized data of mean 0 and standard deviation 1.

Going back to thinking geometrically, when you perform your original operation:

(x_source - tmp.mean()) / tmp.std()

you are essentially moving your data's mean tmp.mean() units to the left, then scaling horizontally by 1/tmp.std().

Notice none of this is vertical shift or scaling, so something smells funny.

What I would do: I would normalize each coordinate independently.

Finding the mean is fine -- the mean of complex data points is the same as means of components:
$$
\bar z = (\bar x , \bar y)
$$
So you can subtract the mean of the $x$ value's from each input value's $x$-coordinate. Ditto for $y$.

Then you divide the real component by the standard deviation of the real component, and ditto for the imaginary component.

It could also be appropriate to divide by the standard deviation of the (new) norms. This would ensure good properties involving your data lying within a circle of a certain radius.

Code:

real_data = real(x_source)
imag_data = imaginary(x_source)

real_data = ( real_data - real_data.mean() ) / real_data.std()
imag_data = ( imag_data - imag_data.mean() ) / imag_data.std()

x_source_norm = real_data + i * imag_data


Is there an R package for Locally Interpretable Model Agnostic Explanations? One of the researchers, Marco Ribeiro, who developed this method of explaining how black box models make their decisions has developed a Python implementation of the algorithm available through Github, but has anyone developed a R package? If so, can you report on using it?

Yes, there is now a port to R, which is available here.
It purports to provide LIME explanations for any classifier that implements a predict() method accepting a type = 'prob' argument. I have not yet tested it myself.


Data scaling for training and test sets when we are scaling the data i needed some clarification. so for preventing data leakage we split the train and test sets and then perform the scaling on them separately, correct?
so while scaling or label encoding the data in the train and test datasets,

how do we ensure that the scaling on test set is according to the train set because fit_transform on the train and test sets scale the features differently. so do we just fit on the train set first and transform the train and test set later?
can we save the scaler so that the new data scaling is done as per the scaler used during training
the same goes for label encoders and binarizer

i mean what is the industry standard of feature transformation?

For point 1. and 2., yes. And this is how it should be done with scaling. Fit a scaler on the training set, apply this same scaler on training set and testing set.
Using sklearn:
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit_transform(X_train)
scaler.fit(X_test)

Regarding binarizing, I think you should not have this problem. Only if you want to choose the threshold dynamically regarding a dataset. If so, you should only use training dataset to choose it.


Audio not saving in google colab # all imports
from IPython.display import Javascript
from google.colab import output
from base64 import b64decode

RECORD = """
const sleep  = time => new Promise(resolve => setTimeout(resolve, time))
const b2text = blob => new Promise(resolve => {
  const reader = new FileReader()
  reader.onloadend = e => resolve(e.srcElement.result)
  reader.readAsDataURL(blob)
})
var record = time => new Promise(async resolve => {
  stream = await navigator.mediaDevices.getUserMedia({ audio: true })
  recorder = new MediaRecorder(stream)
  chunks = []
  recorder.ondataavailable = e => chunks.push(e.data)
  recorder.start()
  await sleep(time)
  recorder.onstop = async ()=>{
    blob = new Blob(chunks)
    text = await b2text(blob)
    resolve(text)
  }
  recorder.stop()
})
"""

def record(sec=5):
  display(Javascript(RECORD))
  s = output.eval_js('record(%d)' % (sec*1000))
  b = b64decode(s.split(',')[1])
  with open('audio.wav','wb') as f:
    f.write(b)
  return 'audio.wav'  # or webm ?

The above is the code for audio recording in google colab. But the audio file is not saved in google colab and showing after running the code. kindly help?

You are only defining the record function but never actually running it. Adding record() to the end of the code works for me in Google Colab and saves the audio to the audio.wav file.


Query regarding neural network model I used the Neural Network Toolbox in matlab to train my data. I used four training algorithms, Scaled Conjugate Gradient (SCG), Gradient Descent with momentum and adaptive learning back-propagation (GDX), Resilient Back Propagation (RBP) and Broyden-Fletcher-Goldfarb-Shanno quasi-Newton back propagation (BFG). I have fixed the seeds at different points and obtained the accuracy. This is what I get:

The first column contains the size of the feature set. I have added features and increased the size of feature set to analyse the performance. Initially I have ranked the features and then taken the top 8 feature as one set, the top 16 feature as the next set and so on. The first number before the '-' , is the performance of the algorithm on the training set, the second number after the '-' is the accuracy of the testing set. The train and test set has been divided into 60 and 20 respectively. The other 20 is the validation set. The learning algorithms each have been run with the same seed values to fix the accuracy.  I have fixed the seed to obtain each of the results btw, Like I have used rng(1), rng(10), rng(158), rng(250) and averaged the results to obtain one single pair of train-test accuracy, and I have done this for each pairs. As you can see I am getting the same accuracy for all feature set size for each of the individual training algorithm. The same data shows perturbation in SVM chen I change the set size. What does this mean?

To debug this case, I suggest you try the following steps:

Reduce the features step-by-step until you end up with using just 1 feature and see whether the accuracy changes or not.
Add a sine-wave and a random noise to the feature set and see whether it effects any of these optimization algorithms.
Re-evaluate how you selected or derived these features, check if these are highly correlated.
Are your classification targets highly imbalanced? If so, then under/over sample them to achieve a more balanced training set. Then check the performance of you algorithm after training over this balanced dataset.

As already highlighted by Jan van der Vegt, its extremely odd that changing the no of features from 8 to 40 has no impact on test set accuracy.


When to use cross-validation? Cross-validation
Hi, I'm deploying machine learning models in my MSc thesis using Weka. I have noticed that when I use 10-fold cross-validation in the training dataset I get low evaluation metrics compared to training the model on the entire dataset without cross-validation, this can make sense if I get better performance for the model when I test it on the testing dataset, however, it keeps the same performance. For example, I trained a random forest model for a regression problem and these are the results I have got:
1. Without cross-validation:
Training:
R = 0.97; Mae = 1.31; Rmse = 1.78
Testing:
R = 0.91; Mae = 2.70; Rmse = 3.57
2. With 10-fold cross-validation:
Training:
R = 0.76; Mae = 3.55; Rmse = 4.77
Testing:
R = 0.91; Mae = 2.70; Rmse = 3.57

As you can see the performance of the model on the testing dataset is the same, can someone explain to me what is the purpose of using it if the model is the same in either case?

In a short answer, It seems that you used the same hyperparameters for both cross-validation's last training and the simple training process (without cross validation).
In the long answer

In K-fold cross-validation, you tune the hyperparameters according to the mean of K performance in order to obtain the maximum generalizing ability of the model.
Then you train the model last time with the chosen hyperparameters using the whole training set. If you have a test set you use that for the final evaluating of your model.
To sum up, the last step of the training part is training the model with the whole training set even if you use k-fold cross-validation or not.
So it seems that with cross-validation and without cross-validation you use the same hyperparameters as "max_depth", "minimum sample leaf etc."


How to use a set of pre-defined classifiers in Adaboost? Suppose there are some classifiers as follows:
dt = DecisionTreeClassifier(max_depth=DT_max_depth, random_state=0)
rf = RandomForestClassifier(n_estimators=RF_n_est, random_state=0)
xgb = XGBClassifier(n_estimators=XGB_n_est, random_state=0)
knn = KNeighborsClassifier(n_neighbors=KNN_n_neigh)
svm1 = svm.SVC(kernel='linear')
svn2 = svm.SVC(kernel='rbf')
lr = LogisticRegression(random_state=0,penalty = LR_n_est, solver= 'saga')

In AdaBoost, I can define a base_estimator and also the number of estimators. However, I want to use these 7 classifiers. In other words, n_estimators=7 and these estimators are above ones. How can I define this model?

One possible solution is using a Stacking classifier as follows:
from sklearn.ensemble import StackingClassifier
# Assumed you already import  all the models you are using

dt = DecisionTreeClassifier(max_depth=DT_max_depth, random_state=0)
rf = RandomForestClassifier(n_estimators=RF_n_est, random_state=0)
xgb = XGBClassifier(n_estimators=XGB_n_est, random_state=0)
knn = KNeighborsClassifier(n_neighbors=KNN_n_neigh)
svm1 = svm.SVC(kernel='linear')
svn2 = svm.SVC(kernel='rbf')
lr = LogisticRegression(random_state=0,penalty = LR_n_est, solver= 'saga')

estimators = [
    ('rf', rf),
    ('svm1', svm1), ('svn2', svn2), ('xgb', xgb), ('knn', knn), ('lr', lr), ('dt', dt)
]

Option 1:
stacker = AdaBoostClassifier()
model = StackingClassifier(
    estimators=estimators, final_estimator= stacker
)

Option 2:
base_model = StackingClassifier(
    estimators=estimators
)
model = AdaBoostClassifier(base_estimator = base_model)

Option 2 will be without question the most expensive* of both and as far as I understand, option 2 fits better what you are looking for.

Strategy:


result of correlation function in r I would like to ask if the matrix that is returned from the R cor() function applied on a dataset, is considered a confusion matrix. Thank you

No, it is not a confusion matrix.

Please have a look at the docs

This is how the confusion matrix is created in R.


Feature Transformation on Input data I was reading about the solution to this OTTO Kaggle challenge and the first place solution seems to use several transforms for the input data X, for example Log(X+1), sqrt( X + 3/8), etc. Is there a general guideline on when to apply which kind transformations to various classifiers?

I do understand the concepts of mean-var and min-max normalization. However, for the above transformations, my guess is that Log and Sqrt are used to compress the dynamic range of the data. And the x-axis shift is just to recenter the data. However, the author chooses to use different methods of normalization for the same input X when feeding into different classifiers. Any ideas?

We love the normal form

In most cases we try to make them act like normal. Its not classifiers point of view but its feature extraction view!

Which Transformation?

The main criterion in choosing a transformation is: what works with the data? As above examples indicate, it is important to consider as well two questions.

What makes physical (biological, economic, whatever) sense, for example in terms of limiting behaviour as values get very small or very large?
This question often leads to the use of logarithms.

Can we keep dimensions and units simple and convenient? If possible, we prefer measurement scales that are easy to think about.

The cube root of a volume and the square root of an area both have the dimensions of length, so far from complicating matters, such transformations may simplify them. Reciprocals usually have simple units, as mentioned earlier. Often, however, somewhat complicated units are a sacrifice that has to be made.

When to Use What?

The most useful transformations in introductory data analysis are the reciprocal, logarithm, cube root, square root, and square. In what follows, even when it is not emphasised, it is supposed that transformations are used only over ranges on which they yield (finite) real numbers as results.

Reciprocal: The reciprocal, x to 1/x, with its sibling the negative reciprocal, x to -1/x, is a very strong transformation with a drastic effect on distribution shape. It can not be applied to zero values.  Although it can be applied to negative values, it is not useful unless all values are positive. The reciprocal of a ratio may often be interpreted as easily as the ratio itself:    Example:

population density (people per unit area) becomes area per person
persons per doctor becomes doctors per person
rates of erosion become time to erode a unit depth

(In practice, we might want to multiply or divide the results of taking the reciprocal by some constant, such as 1000 or 10000, to get numbers that are easy to manage, but that itself has no effect on skewness or linearity.)

The reciprocal reverses order among values of the same sign: largest becomes smallest, etc. The negative reciprocal preserves order among values of the same sign.

Logarithm: The logarithm, x log10 x, or x log e x or ln x, or x log 2 x, is a strong    transformation with a major effect on distribution shape. It is    commonly used for reducing right skewness and is   often appropriate    for measured variables. It can not be applied to zero or negative    values. One unit on a logarithmic scale means a multiplication by the    base of logarithms being used. Exponential growth or decline.

$y = a exp (bx)$

is made linear by
- $ln y = ln a + bx$
       so that the response variable y should be logged. (Here exp() means    raising to the power e, approximately 2.71828, that is the base of    natural logarithms).
       An aside on this exponential growth or decline equation:
       $x = 0$, and $y = a exp(0) = a$ so that a is the amount or count when    x = 0. If a and b > 0, then y grows at a faster and faster rate (e.g.    compound interest or unchecked population growth), whereas if a > 0    and b < 0, y declines at a slower and slower rate (e.g. radioactive    decay).

Power functions :
$y = ax^b$ are made linear by $log y = log a + b log x$
so that both variables y and x should be logged. An aside on such power
functions: put $x = 0$, and for $b > 0$,
$y = ax^b = 0$    so the power function for positive b goes through the origin, which    often makes physical or biological or
economic sense. Think:    does    zero for x imply zero for y? This
kind of power function is a    shape    that fits many data sets
rather well.

Consider ratios y = p / q where p and q are both positive in       practice.

Examples are:

Males / Females
Dependants / Workers
Downstream length / Downvalley length

Then y is somewhere between 0 and infinity, or in the last case, between 1 and infinity. If p = q, then y = 1. Such definitions often lead to skewed data, because there is a clear lower limit and no clear upper limit. The logarithm, however, namely
log y = log p / q = log p - log q,
is somewhere between -infinity and infinity and p = q means that log y = 0. Hence the logarithm of such a ratio is likely to be more symmetrically
distributed.

Cube root: The cube root, x 1/3. This is a fairly strong transformation with a substantial effect on distribution shape: it is weaker than the logarithm. It is also used for reducing right skewness, and has the    advantage that it can be applied to zero and negative values.  Note that the cube root of a volume has the units of a length. It is commonly applied to rainfall data.

Applicability to negative values requires a special note. Consider
(2)(2)(2) = 8 and (-2)(-2)(-2) = -8. These examples show that the
cube root of a negative number has negative sign and the same
absolute value as the cube root of the equivalent positive number. A
similar property is possessed by any other root whose power is the
reciprocal of an odd positive integer (powers 1/3, 1/5, 1/7, etc.)
This property is a little delicate. For example, change the power just a
smidgen from 1/3, and we can no longer define the result as a product of
precisely three terms. However, the property is there to be exploited if
useful.

Square root:The square root, x to $x^(1/2)$ = sqrt(x), is a transformation with a moderate effect on distribution shape: it is
weaker than the logarithm
   and the cube root. It is also used for reducing right skewness, and also
   has the advantage that it can be applied to zero values. Note that the
   square root of an area has the units of a length. It is commonly applied
   to counted data, especially if the values are mostly rather small.

Square: The square, x to $x^2$, has a moderate effect on distribution shape and it could be used to reduce left skewness. In
practice, the main reason for   using it is to fit a response by a
quadratic function $y = a + b x + cx^2$. Quadratics have a turning
point, either a maximum or a minimum, although the turning point in a
function fitted to data might be far beyond the limits of the
observations. The distance of a body from an origin is a quadratic if
that body is moving under constant acceleration, which gives a very
clear physical justification for using a quadratic. Otherwise
quadratics are typically used solely because they can mimic a
relationship within the data region. Outside that region they may
behave very poorly, because they take on arbitrarily large values for
extreme values of x, and unless the intercept a is constrained to be
0, they may behave unrealistically close to the origin.

Squaring usually makes sense only if the variable concerned is zero
or positive, given that $(-x)^2$ and $x^2$ are identical.


How to compute unseen bi-grams in a corpus (for Good-Turing Smoothing) Consider a (somewhat nonsensical) sentence - "I see saw a see saw"

The observed bi-grams would be:
"I see""see saw""saw a"and,"a see".

My aim is to smoothen out the probability mass of the bi-gram probabilities by using Good-Turing smoothing. For this, I need to find the count of unseen bi-grams, i.e., bi-grams with a frequency count of 0.

How do I do this?

1) Would this be a list of all bi-grams formed by using 2 non-consecutive words? For example, "I saw", "saw saw", "a I", etc.?

2) Would repetitions of the same word be included as bi-grams? Eg. "I I", "see see", etc.?

I just remembered that we create a table with all possible words as the header of each row and of each column. As a result, the list of all bi-grams would be all possible bi-grams formed by concatenating any 2 words.


Is it wrong to use Glorot Initialization with ReLu Activation? I'm reading that keras' default initialization is glorot_uniform.

However, all of the tutorials I see are using relu activation as the go-to for hidden layers, yet I do not see them specifying initialization for those layers as he.

Would it be better for these relu layers to use he instead of glorot?

As seen in OReilly's Hands-On Machine Learning with Scikit-Learn & Tensorflow:

| initialization | activation                    |
+----------------+-------------------------------+
| glorot         | none, tanh, logistic, softmax |
| he             | relu & variants               |
| lecun          | selu                          |

As a general answer for hyperparameter tuning, you have to try both and see what works better for your problem. I suspect that some (if not most) of general tuning rule have been observed on a given problem / with a given architecture. (for exemple the He paper is about vision, including convolutional layers).

As for keras choice, sometimes, for practical reasons, it is easier to implement an unique default option than to adapt the default option to the activation. Given your Hends-on machine learning citation, it's not hard to see why they would implement Glorot as the default.


How do you search for content, not words? Given a string that describes a situation or method, are there algorithms that create fingerprints out of it, compare it with a corpus to then point to pages where a similiar concept is being discussed?
The simple form is word search.
You search for word X and X appears in a text whose excerpt is shown to you.
This is the next step.
You describe X without writing "X" and you get text excerpts as a result where X is also described using different phrases (or even words) with or without an explicit mention of "X".
How do you search for content, not words?
Hint: I am looking for technical terms for this problem to find research papers.

The general principle about finding text based on meaning similarity is called distributional semantics. The main idea is that words with a similar meaning statistically tend to co-occur with the same context words. This is the basis for many standard tasks like topic modeling, word sense disambiguation/induction, and generally for everything related to semantics.
The problem you're describing is essentially information retrieval: given a query expressed as one or several words, find the "documents" in a corpus which are semantically similar to this query.
[edit] Depending what the data looks like, you might be interested in the more specific task of semantic textual similarity.


Splitting a neural network in 2 microservices I have a neural network, that is already trained locally that can detect objects in the scene.

But I have to split the neural network into 2 parts, let's say it has 16 layers, and I want to have one microservice handle the first 8 layers and give the output of the 8th layer to the next microservice and it takes the data from there and proceeds with the 9th layer (1st layer in the 2nd microservice).

Sending an image to the 1st microservice will give the result from the 2nd microservice, is this feasible using TensorFlow?

Yes, you can split the model into two parts after training.  Not sure, what would be the advantage, but it is possible.  High level steps :

Define and train the model
Save trained model
Create two copies of model (Layers 1 to 8 and new input layer + 9 to 16)
Transfer weights to these copies of model

After this, service 1 can run first few layers and provide the output to service 2.

Example code to split the model (Code uses Keras as wrapper over TensorFlow):

import keras
from keras.models import Model, load_model
from keras.layers import Input, Dense
from keras.optimizers import RMSprop
import numpy as np

# Create original model and save it
inputs = Input((1,))
dense_1 = Dense(10, activation='relu')(inputs)
dense_2 = Dense(10, activation='relu')(dense_1)
dense_3 = Dense(10, activation='relu')(dense_2)
outputs = Dense(10)(dense_3)

model = Model(inputs=inputs, outputs=outputs)
model.compile(optimizer=RMSprop(), loss='mse')
model.save('test.h5')

# Load the model and make modifications to it
loaded_model = load_model('test.h5')
loaded_model.layers.pop()
loaded_model.layers.pop()

# Create your new model with the two layers removed and transfer weights
new_model = Model(inputs=inputs, outputs=dense_1)
new_model.compile(optimizer=RMSprop(), loss='mse')
new_model.set_weights(loaded_model.get_weights())

new_model.summary()
new_model.save('test_complete.h5')

Source : https://github.com/keras-team/keras/issues/8772


Should I create metafeatures for my XGBoost training set? Say I've got two (not necessarily independent) features A and B for my dataset. Should I create metafeatures from them? say for example the ratio: $$\frac{A}{B}$$

Yes.

You should think about the physical meaning of each proposed metafeature, and whether it's relevant to the problem at hand. For example, suppose you're interested in patient temperature. You might add in $height^2$ and $height^3$, as they are roughly proportional to surface area (through which patient loses heat to a cool environment) and proportional to volume (tissue respirating to generate heat). Of course, you might choose to discard that last one as silly if patient $weight$ is part of your dataset. If your subjects are mammals ranging from mouse up to elephant, that cubic entry might still hold some predictive power.

Multiplying a pair of prices to obtain a column with units of dollars squared would not be a good idea.

In the familiar iris dataset, multiplying length by width to get sepal area or petal area definitely makes sense, as does dividing them, to obtain an aspect ratio.

If a metafeature doesn't improve training error, then of course discard it and move on.


Multiple Linear Regression with k-fold Cross Validation I would first like to create few multiple regression models based on if the models violate any multiple regression assumptions and how well it fits the training data. Then I would like to compare how well these models predict new observations by using 5-fold Cross Validation. From my understanding 5-fold Cross Validation shuffles then splits my data into 5 groups and chooses 1 for the testing set, and the other 4 for the training set. A given model is tested and the prediction error is recorded. This is repeated until all 5 groups are used as a testing set. Finally, the prediction errors are averaged.

My question is, when I am first determining the multiple regression equation (checking for assumptions, applying transformations, variable selection, etc.) which set of data should I use as my training set? Do I use the entire data set? Do I use one of the 5 training sets created by the 5-folds CV method? Do I repeatedly try to fit the regression model for all 5 training sets? If so, how would I extract each training set using the caret package?

As Peter said, you need to split your dataset into two subsets: Training, and Test sets. Generally, 80% of data is allocated for Training set (20% for the Test set).
Thereafter, depending on the language/package you use (caret in your case), you use 5- or 10-fold cross-validation to train your model, and finally, you check the prediction ability of the model using the Test set.

I quickly checked the Caret Package website and ripped the required code for you.

1- For Training-Test split:

4.1 Simple Splitting Based on the Outcome

library(caret)
set.seed(3456)
trainIndex <- createDataPartition(iris$Species, p = .8,
                                  list = FALSE,
                                  times = 1)

2- For training with 10-fold cross-validation:

5.3 Basic Parameter Tuning,
5.5.4 The trainControl Function

fitControl <- trainControl(## 10-fold CV
                       method = "repeatedcv",
                       number = 10,
                       ## repeated the CV ten times
                       repeats = 10)

Usually, we do the cross-validation only once (repeats = 1); but to check the consitency of the results, you may need more repeats.