Detecting person on the picture if we know it is there I implemented an algorithm for pedestrian classification using Keras and Tensorflow and it is quite good, it has 96 percent accuracy on test data. It gives you information about existence of a person on the picture. Now I want to use that to detect where the person is. How to do that? For example, I want to build rectangle around the pedestrian but I don't know how to do that if I already know that there is a person on the picture.

What you are trying to do is called object localisation. If you have multiple of that object in the scene it is called object detection. There are numerous things that you can try that I refer to some of them.

YOLO paper which may be the perfect thing for your task which has the state of the art performance these days.
There is another algorithm which is old and is not efficient but is a simple way of detecting objects. It is not that much accurate. It is called Sliding Window.
The previous algorithm is time-consuming due to the fact that it calculates repetitive calculations. In OverFeat: Integrated Recognition, Localization and ... paper they have tried to overcome that mistake.

There are other approaches too because object detection is an important task in computer vision and every day someone makes another approach. You can see the citation of the mentioned papers which means there are abundant studies about the problem.


Does it make sense to use train_test_split and cross-validation when using GridSearchCV to play with hyperparameters? I was wondering if my methodology makes sense. I am using GridSearchCV with cross-validation to train and tune model hyperparameters for a bunch of different model types (e.g. Regression Trees, Ridge, Elastic net, etc.). Before fitting the models I leave out 10% of the sample for model validation using train_test_split. (see Screenshot). I select the models with the best parameters to make predictions on the unseen validation set.

Am I missing something, as I haven't seen someone doing this when evaluating model accuracy while tuning for model parameters?

Yes it makes sense. Your "validation" set is typically called the "test" set (at least in my experience). CV is creating train/validation sets to pick the hyperparams.

The most complete process would be to then build a model one more time with the best hyperparams on the 90%, and evaluate on the 10%. This evaluation gives you a reliable estimate of the generalization error. It can be slightly lower than the error you got from the the CV process for the same parameters; the hyper-parameter fitting can itself 'overfit' a bit.

Some people skip this last step, if you don't care about estimating generalization error. Whatever it is, it's still the best you can do, according to your tuning process. The upside of skipping it is, I suppose, 10% more training data.


Clustering based on geolocation pair I am trying to process a large set of location data where a list of start and end coordinate is given. For example,

[
  [(start_lat1, start_lon1), (end_lat1, end_lon1)],
  [(start_lat2, start_lon2), (end_lat2, end_lon2)],
  [(start_lat3, start_lon3), (end_lat3, end_lon3)],
  [(start_lat4, start_lon4), (end_lat4, end_lon4)],
  [(start_lat5, start_lon5), (end_lat5, end_lon5)],
  ...
]

My goal is to create clusters so that if different pair of start and end locations are close, they will form a cluster of pair with those start and end locations. For example, the average for the clustered pair will look something like this,

[
  [(start_lat_C1, start_lon_C1), (end_lat_C1, end_lon_C1)],
  [(start_lat_C2, start_lon_C2), (end_lat_C2, end_lon_C2)
  ...
]

I was following https://geoffboeing.com/2014/08/clustering-to-reduce-spatial-data-set-size/ this, but the tutorial only works for a single coordinate point. Any help about how to approach for paired clustering would be much appreciated.

Define your own distance function.

I suggest you simply use

dist(x,y)=haversine(x[0],y[0])+haversine(x[1],y[1])


Unsupervised Hierarchical Agglomerative Clustering I've read a number of papers where the authors talk about "Unsupervised Hierarchical Agglomerative Clustering". They seem to imply that the algorithm determines the number of clusters based on a hyper-parameter:

We define the hetereogeneity metric within a cluster to be the average
of all-pair jaccard distances, and at each step merge two clusters if
the heterogeneity of the resultant cluster is below a specified
threshold

When I search for python implementations of Agglomerative Clustering I keep coming up with sklearn, which requires the number of clusters to be specified aprior. In most examples this is computed by plotting a dendogram and then determining by what appears to be eyeballing the chart how many clusters - for example https://towardsdatascience.com/machine-learning-algorithms-part-12-hierarchical-agglomerative-clustering-example-in-python-1e18e0075019 I'd argue it's impossible from the chart alone to determine if 3 or 5 is the optimal (based on largest vertical distance). I believe this is Wards method but I'm not sure it's the same as ""merging clusters where the heterogeneity is below a threshold" and
Is this possible in sklearn, or is there another python implementation which does this? I feel at the very least there should be a way to process the dendogram programmatically rather than plotting it?

I think I've figured out how to implement the algorithm described in the paper I'm studying. I suspect they used scipy.cluster.hierarchy.
Anyway, my process is:

Generate a distance matrix y from my list of examples x.
Compute the linkage using scipy.cluster.hierarchy.linkage
Generate flat clusters using scipy.cluster.hierarchy.fcluster

The last step is where the threshold mentioned is applied. I still have a question around how to use fcluster to generate clusters based on heterogeneity
What I've found confusing is there are a lot of tutorials on how to determine the number of clusters for sklearn.cluster.AgglomerativeClustering which use scipy.cluster.hierarchy.linkage then scipy.cluster.hierarchy.dendrogram to plot a dendrogram and which is then used to visually identify how many clusters are required.


Understanding the effect of num_words of Tokenizer in Keras Consider the following code:

from keras.preprocessing.text import Tokenizer

tokenizer = Tokenizer(num_words = 5000)
tokenizer.fit_on_texts(texts)
print('Found %d unique words.' % len(tokenizer.word_index))

When I run this, it prints:

Found 88582 unique words.

My question is, isn't num_words the parameter that controls the number of words in the mapping dictionary known as tokenizer.word_index? Then why it still holds 88582 words when I explicitly asked it to keep only 5000 words?

The problem is with the way things are documented. Check this link:
https://stackoverflow.com/questions/46202519/keras-tokenizer-num-words-doesnt-seem-to-work


Application of machine learning in your job For those of you who may implemented any machine learning application in your business context, can you share your experience and results obtained?

Amidst all the hype on AI I am interested to hear from actual applications.

Using machine-learning techniques, nowadays computers can automatically see and understand better than years ago. Tasks like recognition, localization, detection, semantic segmentation and related things have better performance today. This progress has promoted other fields too. Automatic cars, using autonomous driving benefit these advantages for pedestrian detection or car detection.

Recommender systems are so popular today. They suggest you exactly what you need based on your actions. On Facebook, you can find your old friends without even trying to. You can see a lot of movies which you may be sure that will like them.

During searching in search engines, you can see exactly what you are looking for without even knowing how the learning algorithm is ranking the results to show you the appropriate result.

Your email service finds spam and junk mails automatically and delete them for you and you can be confident that you won't loose any important thing.

OCR systems transform images of books and make them as text files. Using this, you can search in texts and find what you want without searching a lot of images.

Tasks which are difficult for finding algorithms, like walking of robots, are so better than before, using data and machine-learning.

There have been designed agents which can defeat human brain in famous games, like chess. Even in AAA computer games, the agents use reinforcement learning to learn how to defeat the human rivals using experience of each specific player based on their behavior.

Machine learning is EVERYWHERE.


Code Vectorization of gsub in R How can I vectorize this code in R?

data <- data.frame(A = rep(5, 5), B = rep(0, 5))
data$abstract <- c("no abstract available", "A", "A", "B", "no abstract available")

for (row in (1:nrow(data))){
  data [row,"abstract"] <- gsub("no abstract available"," ",data[row,"abstract"])
}

You have plenty of alternatives for this problem

Using sapply

data$abstract <- sapply(data$abstract,
                        function(x){gsub(pattern = "no abstract available",
                        replacement = " ", x)})

Using mapply

data$abstract <- mapply(gsub, pattern = "no abstract available",
                        replacement = " ", data$abstract)

Using the stringr package

library(stringr)
data$abstract <- str_replace(data$abstract, "no abstract available",
                             " ")

Also, check this this question on StackOverflow for more information, like solutions with match and the qdap package.


Setting BATCH SIZE when performing multi-class classification with imbalanced dataset I have a question regarding BATCH_SIZE on multi-class classification task with imbalanced data. I have 5 classes and a small dataset of around 5000 examples. I have watched G. Hinton's lectures on Deep Learning and he states that each mini batch should ideally be balanced (meaning each batch should contain approximately same number of data points for each class). This can be approximated by shuffling data and then drawing random batch from it.
But, in my mind this will only work if we have some what large and BALANCED dataset. In my case, I think that setting BATCH_SIZE to be >=16 it might have a bad impact on learning, and network will not be able to generalize. Is it better to maybe use SGD and update weights after each sample has been processed (i.e. online training)? P.S. have in in mind that I am using label smoothing (i.e. class weighted loss)

There are two common options:

Stratified sampling within each batch. Regardless of batch size, make sure each group is equally represented. The downside this approach is that it would significantly slow down training.

Train with increased batch size (say 32-256) and over the course of the epochs the random fluctuations will "average" out.


Reinforcement learning for continuous (rather than discrete) actions I'm familiar with traditional reinforcement learning where the algorithm must choose a categorical action (e.g., best move in a game or the highest click-through-rate ad among a set of ads). Does reinforcement learning work for problems with continuous actions? Are there any easy to understand references that you recommend? I was hoping to be able to use reinforcement learning to assign credit limits, which are represented in dollars (i.e., continuous amounts).

Edit: I've come across this paper from Google's DeepMind, which so far seems relevant.

The DDPG described in that paper is very good to start working in control problems. There are many implementations of this algorithm online. However, bare in mind though that the algorithm is very unstable and needs many hours of monitoring and tuning the parameters of the networks. I would suggest you to create a very simple scenario (simulator) and put the DDPG to interact with it. It is very important to understand what your networks are learning and which state features are important or no. Furthermore the reward function will play a very important role in the performance of your algorithm.

Another important point is the way you formalize your problem will determine if you are trying to solve a one-step decision making problem (e.g. given state features ---> credit limit value, reward) or a sequential decision making problem (e.g. given state features ---> credit limit value, reward, new state features and so on). This will specify if you need to modify the Q learning part of the Critic and remove the $maxQ(s',a')$ part (one step decision case).

Regarding pretraining, as Neil suggested you can use regression to pretrain Actor and Critic networks so the weights are initialized to better than random values. A very interesting approach from Deepmind is the one that is described in this paper Learning from Demonstrations for real-world RL, and most likely helped them optimize the energy consumption at their server centers.


Is it a good practice to evaluate model performance by comparing the metrics of rescaled (inverse transformed) predictions and true target values? I am now working with a Linear Regression for a time-series regression problem (I am sorry that I cannot say too much about the problem and feature vector due to NDA).
I scaled both the input values and target (X and y) with scikit-learn StandardScaler().
The reasoning of scaling both X and y because since it is a time series regression problem, we are using the previous y value (target value) as the feature (X) for the next datapoint.
Thus, there is a relation between y and X.
So, I thought that in order to maintain the dependence of X and y (the previous target value), I need to scale them both.
After training the model, I got a low RMSE on my validation and test sets. Note that, the validation and test sets contain the data with scaled/normalized X and y values.
However, when I tried to rescale (inverse_transform) the predictions, and evaluate the RMSE of the predictions against the true target value of the test set (the true target value is already rescaled as well), I got a high RMSE error.
My question is, is it a good practice to evaluate RMSE/accuracy of rescaled (inverse transformed) predictions against the true target values?

The error as a percentage of the true value in each case is probably the same - it just seems bigger because the standardised values are a lot smaller than the unstandardised ones (standardised values will be mostly between -1 and 1 and the unstandardised values will be whatever they are in your dataset). Try measuring the fit of your linear model using Pearson’s correlation coefficient, checking how your predicted values correlate to the true values. This should be the same for both the scaled and unscaled cases and will give you a better idea of whether your model is good. You could also plot your predicted values against the true values in a scatter plot so that you can see roughly what’s going on.


How to define a custom metric function in R for Keras? I just started using keras and would like to use unweighted kappa as a metric when compiling my model. Following the instructions from here, I tried to define my custom metric as follows:

library(DescTools)  # includes function to calculate kappa
library(keras)

metric_kappa <- function(y_true, y_pred) {
   CohenKappa(y_true, y_pred)
}

model %>% compile(
   loss = 'categorical_crossentropy',
   optimizer = optimizer_rmsprop(),
   metrics = metric_kappa
)

However, there must be something wrong with my definition since I get a segmentation fault. Any idea on how to solve this and how to properly set up a custom metric would be highly appreciated.

This answers suggests a wrapper function . Does that work in your usecase ?

https://stackoverflow.com/a/51415066/655855


Overcome memory limitation when downloading from database into Orange I would like to run the association rule mining algorithm of the Orange library on a dataset that is stored in a PostgreSQL database. The table 'buildingset' contains the itemsets for each user, thus each record is related to a user, and each field is related to an item. The values are either 1 (smallint) or missing. The table has about 14,000 records and 31 fields.

When I try to run the algorithm on this dataset, I get the following error:

  ValueError Traceback (most recent call last):
    File "/opt/orange/orange3/Orange/canvas/scheme/widgetsscheme.py", line 722, in process_signals_for_widget
      handler(*args)
    File "/home/bdukai/.local/lib/python3.4/site-packages/orangecontrib/associate/widgets/owassociate.py", line 444, in set_data
      self.X = data.X
    File "/opt/orange/orange3/Orange/data/sql/table.py", line 353, in X
      self.download_data(AUTO_DL_LIMIT)
    File "/opt/orange/orange3/Orange/data/sql/table.py", line 333, in download_data
      raise ValueError("Too many rows to download the data into memory.")
  ValueError: Too many rows to download the data into memory.

Thus is there any way overcome this limitation, without upgrading the hardware?

You can use "download data to local memory" checkbox in SQL Table widget, it should allow you to work with up to 1000000 rows.


Are month and weekday rulers of surveys? How can I test the validity? I am analyzing a survey and came across an interesting pattern.
Weekday and month seems to have a very high say in the score given by customers.
There can be many reasons for this, though it is unexpected, but how can I test whether it is significant?

From the charts Tuesday is by far the worst day to ask the customers, and February is the worst month.
Combined it goes crazy…don’t ask the customer anything on Tuesdays in February..
How can I statistically back up or reject these hypothesis?

It looks like a good case for ANOVA, which can be seen as a generalization of Student t-test for more than two groups.

For visualizing the differences I'd suggest using boxplots (or violin plots) for every group: it's more informative than only plotting the mean which is sensitive to outliers.


Don't use clustering at all.

Define a threshold, and theshold the data. Done.

Keep it simple and easy to understand.


How to evaluate performance of a new feature in a model? I am working on a binary classification where I have 4712 records with Label 1 being 1554 records and Label 0 being 3558 records.

When I tried multiple models based on 6,7 and 8 features, I see the below results. Based on the newly added 7th or (7th & 8th) feature, I see an AUC improvement only in one of the models (LR scikit and Xgboost).

I also come across articles online that says AUC or F1-score aren't strict scoring rules. We could use log-loss metric but it's only applicable for logistic regression. but we can't use log-loss metric for Xgboost or RF or SVM right? So, is there any common metric which I can use to compare. Am I missing something here?

Does this mean that new feature is helping us improve the performance? But it decreases the performance in other models?

Please note that I split the data into train and test and did 10 fold CV on train data.

So, how do I know that this newly added 7th feature is really helping in improving the model performance?

update based on answer

from statsmodels.stats.contingency_tables import mcnemar
# define contingency table
table = [[808,138],    # here I added confusion matrix of two models together (I mean based on TP in model 1 is added with TP in model 2 etc)
         [52, 416]]
# calculate mcnemar test
result = mcnemar(table, exact=True)
# summarize the finding
print('statistic=%.3f, p-value=%.3f' % (result.statistic, result.pvalue))
# interpret the p-value
alpha = 0.05
if result.pvalue > alpha:
    print('Same proportions of errors (fail to reject H0)')
else:
    print('Different proportions of errors (reject H0)')

Informally democracy.

So how many classifiers did it improve, only 1 than dont add it.

Formally there are a couple of statistical tests.

Cochran's Q test

Is a generalisation of the McNemars test for comparing Machine Learning models.

or read this formal paper where they discuss it.


Recommendations for algorithm I have a very large dataset consisting of approximately 15,000 rows of survey data. The surveys are quantitative summaries of interviews with investors regarding their rankings of financial asset managers. They rate a manager on a scale from 1-5 and we have approximately 5 years of data for each investor on asset manager.

The data is indexed as:

Investor name | Manager name | Year | Rating parameter 1 | R. parameter 2 | ... | R. parameter 10 |

To give you an idea, I made an example below. Currently the data is saved in an Access-database with a ton of SQL JOINT's - but I can convert the data pretty much how I want to. So compatibility is not an issue.

My question is:
I have a wild idea that somehow the answers are correlated between years. So maybe an increase in parameter 1 and parameter 2 is correlated with a fall in parameter 10.

I'm looking for ways to test my hypothesis without having to do it manually (this would be quite tedious). Maybe through machine-learning or subgroup discovery (I have not idea if this is correct).

Are any of you familiar with tools or algorithms that can help me test my hypotheses? Naturally I'll have to make sure my analyses are statistically significant.

I'm looking very much forward hearing what you think.

Example of data

Buffet Holdings | Goldman Sachs AM | 2014 |1|5|5|1|4|5|1|1|1|5|
Buffet Holdings | Goldman Sachs AM | 2015 | 4 | ........| 1 |
Gates Ltd. | JP Morgan AM | 2014  | 4 | ........| 2 |
Gates Ltd. | JP Morgan AM | 2015 | 4 | ........| 3 |
Buffet Holdings | JP Morgan AM | 2014 | 3 | ........| 5 |
Buffet Holdings | JP Morgan AM | 2015 | 2 | ........| 1 |

A correlation matrix would help discover any pair-wise correlation (between any 2 variables). I think that would be a great place to start, as it is just one line of code for initial analysis.

After that, if you want to investigate "an increase in parameter 1 and parameter 2 is correlated with a fall in parameter 10", you could write code to form interaction variables like sum / product / divide and build a correlation matrix between the interaction variables vs. the original variables.

Personally, I found the corrplot R package useful in visualizing the correlation matrix. Hope this helps.


Choice of objective function for log transformed data for feedforward NN I am processing some data using a feedforward neural network in Keras. I have noticed that if I log transform the data, the model trains better, however the error metric on the transformed data doesn't necessarily reflect very well what's going on with the original dataset.

At the moment I am using Mean Average Percentage Error metric (mape), and a CV error of 3.8% on log transformed data set corresponds to about 25% when I reverse-transform the predictions and check against the original outputs.

Is there a better choice of cost/objective function for situations like this?

I am assuming that when you say log transforming the data, it is log transformation of outcome variable. In such case, you may want to check root means squared logarithmic error.


How are Decision Trees averaged in Random Forest? We all know that a Random Forest is an ensemble of Decision Trees, whose results are averaged.

Every source I find simply talks about "averaging trees", but how does this "averaging of trees" happens? Can anyone exaplain it in detail?

It depends on the type of variable that the Random Forest is predicting, and perhaps on the specific implementation of Random Forest.  Following is an overview of the simplest techniques.

Continuous Target

In the case of a continuous target variable $y$, each of the trees in an ensemble will generate a prediction $\hat{y}_i$.  The most naive way to combine the results from trees in an ensemble is to take the mean of all predictions.

Suppose you have an ensemble of 10 trees. Then the combined prediction $\hat{y}$ would be computed as
 $$\hat{y} = \frac{1}{10} \sum^{10}_{i=1} \hat{y}_i$$

This is the approach taken by forest regressors in scikit-learn.  Notice the lines

    # Parallel loop
    lock = threading.Lock()
    Parallel(n_jobs=n_jobs, verbose=self.verbose,
             **_joblib_parallel_args(require="sharedmem"))(
        delayed(_accumulate_prediction)(e.predict, X, [y_hat], lock)
        for e in self.estimators_)

    y_hat /= len(self.estimators_)

The "Parallel Loop" part sums up the predictions made by each tree in the forest, and the last line divides by the total number of trees so that y_hat is the mean of the predictions from each tree.

An alternative approach is to take a weighted average of the predictions
$$\hat{y} = \sum^{10}_{i=1} \omega_i \hat{y}_i$$

Where each tree has a weight $\omega_i$.  The weight could be determined, for example, by the tree's performance.  This approach might slightly improve accuracy in some cases, but it's easier to overfit.

Categorical Target

In the case where your random forest is predicting the value of a categorical variable, you can allow each tree to act like a member of a "committee" and cast a vote.

Suppose a categorical variable has three possible values, $A$, $B$, or $C$, and your random forest has 10 trees.  You generate a prediction with each tree, and that prediction counts as one vote.

For example, suppose 5 of your trees predict $A$, 3 of them predict $B$, and 2 of them predict $C$.  The combined prediction will be $A$.

Notice that you may run into cases where there is a tie.  Suppose 5 of your trees predict $A$, and the other five predict $B$.  Which prediction do you go with?

One option is to predict the majority class in the case of a tie.  In the example above, if the training data contains more $B$ examples than $A$ examples, then we would predict $B$.

Another option for resolving ties is to weight votes by the accuracy of the tree.  Suppose again that you have five trees that vote for $A$ and five trees that vote for $B$.  If you observe that, on average, the trees voting for $B$ are more reliable, you may give their votes extra weight and predict $B$.

Hopefully that helps!


Feature extraction for sentiment analysis I am working on a group project for my capstone course and we have been tasked with creating a sentiment analysis tool with Python business logic and (L/W)AMP everything else.

We have good feedback for every part of our project plan except for feature extraction. One of our advisors insists that we should have ~15 different kinds of features.

Currently we only use unigrams and are having a hard time finding others that are practical to implement with our small set of data (~50 items) and within our time limit (~2 weeks to fully implement).

What are feature extraction techniques that are useful for sentiment analysis and work on smaller datasets? They should be able to be implemented quickly or already exist in a Python library.

Have a look at these papers-

Semantic Similarity - 1
Semantic Similarity - 2

They came up with a good solution. Have a look maybe you will find something of your use in it. Plus try exploring these too

vaderSentiment 0.5
(VADER Sentiment Analysis. VADER (Valence Aware Dictionary and sEntiment Reasoner) is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media, and works well on texts from other domains.)
Demo- Sentiment Analysis with Python
(This is a demonstration of sentiment analysis using a NLTK 2.0.4 powered text classification process. It can tell you whether it thinks the text you enter below expresses positive sentiment, negative sentiment, or if it's neutral. Using hierarchical classification, neutrality is determined first, and sentiment polarity is determined second, but only if the text is not neutral.)

Hope it helps! :)


What is the definition of knowledge within data science? "Knowledge" is crucial within several fields like Knowledge Discovery, Knowledge Distraction, Natural Language Processing, Data Mining, Big Data, etc etc etc.

What is the definition of knowledge within these fields?

Is there 1 common definition, or does it depend on the exact context?

The definition of knowledge varies based on the context, but can be broadly defined as actionable information based on further analysis (statistical, heuristic, or otherwise) of those patterns, associations, or relationships identified in the raw data.

Put another way, knowledge is defined by what information will most prove actionable based on the needs of the client or user (sometimes called the semantics).


Text Classification with deep learning I  have questions of users and I want to classify them automatically without manually labelling them. What deep learning method would be good for text classification just from text (so unsupervised).

Does those algorithms have to rely on word embedding?

Since you are specifically asking about deep learning techniques, nothing strikes me out of the bat other than Autoencoders.

You can try using autoencoders for clustering, basically, you need to stick with clustering. Since you don't have the labels.

To answer your question whether they rely on embeddings that are out there, it depends on your data, if you have a domain-specific data or you have the data in a weird language then you should go for creating your own embeddings.

I found this article really helpful, though it has been done on images, you can try it on text, using Conv1D. Also yes, you can try out word embeddings like word2vec or fasttext. There is this good article where they use gensim for doing attaching the embeddings, in case your data is just plain ordinary English.

there's also this article from keras blog where the author uses pre-trained GloVe embeddings.

Hope this helps.


TensorFlow Time Series Tutorial Enhancement Gone Wrong I’ve been following this time series tutorial for Tensorflow…

https://www.tensorflow.org/tutorials/structured_data/time_series

And it was going good, and seemed to work ok.  I substituted with my own dataset (about 1.1m records with 4 features) and it also seemed to work well, but memory was getting REALLY tight, so I thought I would try and implement the improvement mentioned at the bottom that said…

  In addition, you may also write a generator to yield data (instead of
  the uni/multivariate_data function), which would be more memory
  efficient. You may also check out this time series windowing guide and
  use it in this tutorial.

This kind of made sense, as it seems to produce a sliding time window with single steps so the data becomes HUGE.  After reading through this…

https://www.tensorflow.org/guide/data#time_series_windowing

It looked like I could use tensor flow to do the windowing, which I assumed would get executed dynamically therefore not much memory would be used.  After reading, I came up with the following code to try and replace the multivariate_data function (functions are borrowed from the tutorial, though I changed the dense_1_step to get a single set of label features back instead of a window…

def make_window_dataset(ds, window_size=5, shift=1, stride=1):
  windows = ds.window(window_size, shift=shift, stride=stride)

  def sub_to_batch(sub):
    return sub.batch(window_size, drop_remainder=True)

  windows = windows.flat_map(sub_to_batch)
  return windows

def dense_1_step(batch):
  # Shift features and labels one step relative to each other.
  return batch[:-1], batch[-1:]

# get training samples (features and labels)
train_ds = make_window_dataset(tf.data.Dataset.from_tensor_slices(dataset[:TRAIN_SPLIT]), window_size=past_history+1, shift = 1, stride=1)
dense_labels_train_ds = train_ds.map(dense_1_step)

# get validation samples (features and labels)
val_ds = make_window_dataset(tf.data.Dataset.from_tensor_slices(dataset[TRAIN_SPLIT:]), window_size=past_history+1, shift = 1, stride=1)
dense_labels_val_ds = val_ds.map(dense_1_step)

# batch and shuffle training data
train_data_single = dense_labels_train_ds.cache().shuffle(BUFFER_SIZE).batch(BATCH_SIZE).repeat()
val_data_single = dense_labels_val_ds.batch(BATCH_SIZE).repeat()

But for some reason it's all gone pear shaped for the following reasons...

The training loss and validation loss are now NOTHING alike and in fact I don’t think the net is even learning properly anymore
The memory usage seems much better UNTIL you execute the fit() and then it just blows up about the same anyway.
I get the warning ‘Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.’

I’m sure I screwed something up here, but to me it looks like it's doing what it should be doing.  I'm still on a steep learning curve with this, but if anyone with more experience could point me in the right direction I would be very grateful.

Thanks

Ray

It seems I was returning multiple features as labels.  I had to modify the dense_1_step function to return a single feature...

def dense_1_step(batch):
  # Shift features and labels one step relative to each other.
  return batch[:-1], batch[-1:,1][0] # take second feature only

To make it the same as the output from the multivariate_data function.


How to compute denominator in Naive Bayes? Suppose we have class C_k and input feature vector x in dataset

How to calculate probability p(x)?

in the Examples section of the Wikipedia article there is a nice example.  The calculation of $p(\mathbf{x})$ can be done via

$$p(\mathbf{x}) = \sum_k p(C_k) \ p(\mathbf{x} \mid C_k)$$

Note that using the conditional independence assumption of the Naive Bayes one can write

$$ p(\mathbf{x} \mid C_k) = \Pi_{i} \, p(x_i \mid C_k) $$


What initial steps should I use to make sense of large data sets, and what tools should I use? Caveat: I am a complete beginner when it comes to machine learning, but eager to learn.

I have a large dataset and I'm trying to find pattern in it. There may / may not be correlation across the data, either with known variables, or variables that are contained in the data but which I haven't yet realised are actually variables / relevant.

I'm guessing this would be a familiar problem in the world of data analysis, so I have a few questions:

The 'silver bullet' would be to throw this all this data into a stats / data analysis program and for it to crunch the data looking for known / unknown patterns trying to find relations. Is SPSS suitable, or are there other applications which may be better suited.
Should I learn a language like R, and figure out how to manually process the data. Wouldn't this comprimise finding relations as I would have to manually specify what and how to analyse the data?
How would a professional data miner approach this problem and what steps would s/he take?

I will try to answer your questions, but before I'd like to note that using term "large dataset" is misleading, as "large" is a relative concept. You have to provide more details. If you're dealing with bid data, then this fact will most likely affect selection of preferred tools, approaches and algorithms for your data analysis. I hope that the following thoughts of mine on data analysis address your sub-questions. Please note that the numbering of my points does not match the numbering of your sub-questions. However, I believe that it better reflects general data analysis workflow, at least, how I understand it.

Firstly, I think that you need to have at least some kind of conceptual model in mind (or, better, on paper). This model should guide you in your exploratory data analysis (EDA). A presence of a dependent variable (DV) in the model means that in your machine learning (ML) phase later in the analysis you will deal with so called supervised ML, as opposed to unsupervised ML in the absence of an identified DV.

Secondly, EDA is a crucial part. IMHO, EDA should include multiple iterations of producing descriptive statistics and data visualization, as you refine your understanding about the data. Not only this phase will give you valuable insights about your datasets, but it will feed your next important phase - data cleaning and transformation. Just throwing your raw data into a statistical software package won't give much - for any valid statistical analysis, data should be clean, correct and consistent. This is often the most time- and effort-consuming, but absolutely necessary part. For more details on this topic, read this nice paper (by Hadley Wickham) and this (by Edwin de Jonge and Mark van der Loo).

Now, as you're hopefully done with EDA as well as data cleaning and transformation, your ready to start some more statistically-involved phases. One of such phases is exploratory factor analysis (EFA), which will allow you to extract the underlying structure of your data. For datasets with large number of variables, the positive side effect of EFA is dimensionality reduction. And, while in that sense EFA is similar to principal components analysis (PCA) and other dimensionality reduction approaches, I think that EFA is more important as it allows to refine your conceptual model of the phenomena that your data "describe", thus making sense of your datasets. Of course, in addition to EFA, you can/should perform regression analysis as well as apply machine learning techniques, based on your findings in previous phases.

Finally, a note on software tools. In my opinion, current state of statistical software packages is at such point that practically any major software packages have comparable offerings feature-wise. If you study or work in an organization that have certain policies and preferences in term of software tools, then you are constrained by them. However, if that is not the case, I would heartily recommend open source statistical software, based on your comfort with its specific programming language, learning curve and your career perspectives. My current platform of choice is R Project, which offers mature, powerful, flexible, extensive and open statistical software, along with amazing ecosystem of packages, experts and enthusiasts. Other nice choices include Python, Julia and specific open source software for processing big data, such as Hadoop, Spark, NoSQL databases, WEKA. For more examples of open source software for data mining, which include general and specific statistical and ML software, see this section of a Wikipedia page.
UPDATE: Forgot to mention Rattle, which is also a very popular open source R-oriented GUI software for data mining.


Modern methods for reducing dimensions and feature engineering I am training a binary classifier in Python to estimate the level of risk of credit applicants. I extracted a little over a thousand independent variables to model the observed behavior of four million people. My target is a binary column that tells me whether or not a person defaulted on a loan (1 for event, 0 for non event).
I am asking this question because I feel overwhelmed by the dimensionality of the problem. I want to know some common and modern ways used to:

drop features (dimensionality reduction)
create new features based on combinations of other features (feature engineering)

So far, I dropped features based on their Information Values and kept only the most relevant ones. From the remaining set of features, I calculated each pair's correlation coefficient and for every highly correlated pair, I kept the feature with the highest information value of the two.
I now want to make new features based on this subset of remaining variables, such as ratios and multiplications (i.e. Number of open accounts divided by number of closed accounts). However, I think that my elimination process can be improved. My current method is very old school, as I have mostly relied on univariate analysis to drop features (one variable against the target).

I'm not really knowledgeable about the modern techniques but I can tell you about the old ones: ;)
First there are two main approaches to dimensionality reduction: feature selection and feature extraction. You're using the former, which consists in discarding some of the original features. The latter consists in some kind of "merging" of similar variables, it can be worth trying especially if you have redundant features.
As you rightly noticed, feature selection based on individual features is rarely optimal. There are methods which can take the full set of features as a basis for selection, in particular genetic feature selection.


Problem with binning I am trying to change continuous data points to categorical by using binning. I know two techniques, i) equal width bins ii) bins with equal number of elements.
My questions are:

Which type of binning is appropriate for which kind of problem?
I use pandas for my data analysis task and it has pd.cut method for arbitrary binning which I use for equal wdith bins and pd.qcut method for bins with equal number of elements. The second function always produces very complicated bin boundaries (like, [(-28.004,795.8976],(795.8976,900.342]]). Is there any way to "control" the bin boundaries so that they look more meaningful to non-technical persons?

Thanks in advance.

The two methods you're citing belong to what is called unsupervised binning, including as you said equal width and equal frequency binning. On the other hand, supervised binning broadly tries to make sure bins are made in majority of instances sharing the same class label.
For both types of unsupervised binning, i.e. equal frequency and equal width, the best way is still to give it a try and select based on the observation of the resulting histogram you get. If your data is not properly divided by bins of equal frequency, maybe equal width bins would help, and vice versa.
For what concerns Pandas execution, you can pass a precision argument to qcut, this should return more "comprehensible" bin limits, as shown below
>>> array = np.random.randn(10)

>>> pd.qcut(array, q=4, precision=3)
Categories (4, interval[float64]): [(-1.8889999999999998, -0.732] < (-0.732,
-0.136] < (-0.136, 0.973] < (0.973, 1.543]]

>>> pd.qcut(array, q=4, precision=0)
Categories (4, interval[float64]): [(-3.0, -1.0] < (-1.0, -0.0] < (-0.0, 1.0]
< (1.0, 2.0]]


What is the difference in plotting ROC curve with probability scores vs binary decisions As the title reads what is the difference? Plotting the ROC w.r.t probability scores gives the stair cased version.
But in my opinion I find that using binary decision is better because the ROC curve is just a summary of the different confusion matrices w.r.t different thresholds. And the confusion matrix shows the binary decisions.
Is their an advantage or certain situation where plotting the ROC curve with the prob.scores is better than the binary decisions?

How do you go from probabilities to binary decisions? The default choice is to use a threshold of 0.5 but maybe a threshold of 0.3 or 0.7 would have given better results (depending on you metric). The ROC curve gives you more information as it allows to see the results for each probability threshold.
Usually you set some metric to optimize (F1 score for example) and you set the threshold based on this metric. Then you plot the confusion matrix and any other metric that is useful to you


TF-IDF: How to handle terms not part of the corpus I'm working on a ML.Net based feature to extract keywords from a document corpus using TD-IDF.

Given this test corpus (one document per line):

This is an example to compute n-grams.
N-gram is a sequence of 'N' consecutive words/tokens.
ML.NET's ProduceNgrams API produces vector of n-grams.
Each position in the vector corresponds to a particular n-gram.
The value at each position corresponds to,
the number of times n-gram occured in the data (Tf), or
the inverse of the number of documents that contain the n-gram (Idf),
or compute both and multiply together (Tf-Idf).

And this test document:

foobar This is an example to compute n-grams

I'm getting the following result:

example             = 2,0794
example|compute     = 2,0794
compute             = 1,3863
compute|n           = 0,0000
n                   = 0,0000
n|grams             = 0,0000
grams               = 0,0000
n|gram              = 0,0000
gram                = 0,0000
gram|sequence       = 0,0000

My question is about the term foobar which is not part of the training corpus, and therefore not part of the result set. Is there a recommended way of dealing with out-of-vocabulary terms when using TF-IDF?

In my opinion, there is no way to deal out-of-vocabulary terms in TF-IDF as it only works using the test corpus. It's like you are trying to find certain word like technology in sports corpus.


Neural network model for sparse multi-class classifier on Tensorflow The problem I'm trying to solve is the following:

the data is Movielens with N_users=6041 and N_movies=3953, ~1 million ratings.

For each user, a vector of size N_movies is defined, and the values of the vector are 1 if the user rated the movie before time T, 0 if not. For instance if the user rated movie 3 and 5 the input vector is [0,0,1,0,1].

The goal is to predict the movie the user will rate in the future (between time T and T+delta T). The labels are vector of size N_movies, and if the user rate the movie 4 the labels vector is [0,0,0,1,0].

I'm currently trying to get some initial results based on Fully connected layers, but it seems that it cannot optimize the loss at all. The representation might be too sparse, but it seems that the neural network should be able to learn at least some features.

Is it possible to make this model work, is there any problem with the loss function, or the optimizer?

from __future__ import print_function
import tensorflow as tf
from tensorflow.contrib import rnn
import csv
import bisect
import glob
import re
import numpy as np
import random
import data
import config

cfg = config.Config()
graph_data = data.Graph_data(cfg)

X = tf.placeholder("float", [None, cfg.N_movies])
Y = tf.placeholder("float", [None, cfg.N_movies])

def Dense(x):
    hidden_layer_1 = tf.layers.dense(inputs=x, units=500, activation=tf.nn.relu)
    hidden_layer_2 = tf.layers.dense(inputs=hidden_layer_1, units=50, activation=tf.nn.relu)
    output_layer = tf.layers.dense(inputs = hidden_layer_2, units= cfg.N_movies, activation=tf.nn.softmax)
    return output_layer

logits = Dense(X)
cross_entropy = tf.reduce_sum(- Y * tf.log(logits), 1)
loss_op = tf.reduce_mean(cross_entropy)
optimizer = tf.train.GradientDescentOptimizer(learning_rate=cfg.learning_rate)
train_op = optimizer.minimize(loss_op)

init = tf.global_variables_initializer()

with tf.Session() as sess:
  sess.run(init)

  for step in range(1, cfg.training_steps+1):
    batch_x, batch_y = graph_data.train_next_batch(cfg.batch_size)
    sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})
    if step % cfg.display_step == 0 or step == 1:
      loss = sess.run(loss_op, feed_dict={X: batch_x,
                                                                  Y: batch_y})
      print("loss = ",loss)

The model seems fine to me. Did you try to optimize the hyperparameters? You could use trial and error methods for optimiring the parameters such as,

Learning rate
Number of epochs
Batch size etc,.

And instead of vanilla gradient descent optimizer, you could try other optimizers such as Adagrad, Adam or RMSProp etc,.

Best,
Sangathamilan Ravichandran.


Learning Rate based on error of the network I am not an expert and do not have theoretical justification for that, but it seems to me that the smaller network error is, the smaller learning rate should be.

Is there an algorithm to dynamically update learning rate based on total error of the network without relying on any hyper-parameters ?

Your intuition is on point, and shrinking the learning rate like this is often referred to as "annealing". But linking the learning rate to error magnitude neglects certain problematic error surface topologies. An excellent motivating example is the Rosenbrock "Banana" Function, which is often used as a test case for optimization algorithms. The "banana" is a low error valley which hides the global minimum. If an optimization path finds its way into this valley, the path to the global minimum is along a nearly flat gradient.

If you use an optimization algorithm that naively shrinks the learning rate relative to the error, you're going to get stuck as soon as you hit the valley. On the one hand: congrats! You've achieved a low error solution. But you're not necessarily anywhere near the global minimum. So how can we do better?

An approach used by modern gradient-based methods like Adagrad, RMSProp, and Adam is to separately assign learning rates to each parameter, and tie the learning rate to the magnitude of the respective parameter's update. The Stanford CS231n lecture notes explains:

  Adagrad is an adaptive learning rate method originally proposed by Duchi et al..

  # Assume the gradient dx and parameter vector x
  cache += dx**2
  x += - learning_rate * dx / (np.sqrt(cache) + eps)

  Notice that the variable cache has size equal to the size of the gradient, and keeps track of per-parameter sum of squared gradients. This is then used to normalize the parameter update step, element-wise. Notice that the weights that receive high gradients will have their effective learning rate reduced, while weights that receive small or infrequent updates will have their effective learning rate increased.


Perform clustering from a similarity matrix I have a list of songs for each of which I have extracted a feature vector. I calculated a similarity score between each vector and stored this in a similarity matrix. I would like to cluster the songs based on this similarity matrix to attempt to identify clusters or sort of genres.
I have used the networkx package to create a force-directed graph from the similarity matrix, using the spring layout. Then I used KMeans clustering on the position of the nodes from that graph and this resulted in clusters that made sense. However, I'm not sure that this is the correct approach as it is fundamentally linked with the positions given by the spring layout.
I have also attempted to run Spectral Clustering on the similarity matrix, however, it is too slow.
Is using the positions derived from a graph generated from the similarity matrix, and then using the spring layout piped into KMeans to extract clusters, fundamentally flawed? If so, what could be a potential alternative way to cluster elements given a similarity matrix?

I am not sure I fully understand your question, but generally the metric you use to build a DAG needs to be understood in terms of how you interpret relevant results.
That said, a cluster map sounds like a good match for your use case. That is, a correlation matrix with sorted values according to linkage clustering on your datapoints. See below an example:

You can easily experiment with something like this using seaborn library and seaborn.clustermap.


Using SMOTE for Synthetic Data generation to improve performance on unbalanced data I presently have a dataset with 21392 samples, of which, 16948 belong to the majority class (class A) and the remaining 4444 belong to the minority class (class B). I am presently using SMOTE (Synthetic Minority Over-Sampling Technique) to generate synthetic data, but am confused as to what percentage of synthetic samples should be generated ideally for ensuring good classification performance of Machine Learning/Deep Learning models.

I have a few options in mind:- 1. The first option is to generate 21392 new samples, with 16904 majority samples of class A and remaining 4488 minority samples of class B. Then, merge the original and synthetically generated new samples. However, the key drawback I believe is that the percentage of minority samples in my overall dataset (original+new) would remain more or less the same, which I think defeats the purpose of oversampling the minority samples. 2. The second option is to generate 21392 new samples, with 16904 majority and remaining 4488 minority samples. Then, only merge the original data with the newly generated minority samples of the new data. This way, the percentage of minority (class B) samples in my overall data would increase (from 4444/21392 = 20.774 % to (4444+4488)/(21392+4488) = 34.513 %. This I believe is the purpose of SMOTE (to increase the number of minority samples and reduce the imbalance in the overall dataset).

I am fairly new to using SMOTE, and would highly appreciate any suggestions/comments on which of these 2 options do you find better, or any other option which I may consider alongside.

First of all, you have to split your data set into train/test splits before doing any over/under sampling. If you do any strategy based on your approaches, and then split data you will bias your model and that is wrong simply because you are introducing points on your future test set that does not exist and your scores estimations would be imperfect.

After splitting you data then, you will use only SMOTE on train set. If you use SMOTE from imblearn, it will automatically balance the classes for you. Also, you can use some parameter to change that if you dont want perfect balancing, or try different strategies.

https://imbalanced-learn.readthedocs.io/en/stable/over_sampling.html#smote-adasyn

So, basically, you would have something like this:

from sklearn.model_selection import train_test_split
from imblearn.over_sampling import SMOTE

X_train, X_test, y_train, y_test = train_test_split(X, y, split_size=0.3)
X_resample, y_resampled = SMOTE().fit_resample(X_train, y_train)

Then you continue fitting your model on X_resample, y_resample. Above, X is your features matrix and y is your target labels.


Organize TSNE data into grid I have some data reduced by TSNE into a 2D representation, which shows clear spatial features.

However, I'd like to format this into a grid – not just snapping data to the nearest grid square but spreading everything out to fill up a grid, preserving (as much as possible) the existing spatial relationships.

So far, I've only found this article which might close to what I need? This process might already have a name and I'm just one step from an easy Google solution, but at the moment I'm stuck!

There seem to be a few options, but I found rasterfairy which is very easy to install and use. Has the added bonus of being able to fit to a rectangular grid, but also circular and other arbitrary shapes.

A very nice IronPython notebook example: https://github.com/Quasimondo/RasterFairy/blob/master/examples/Raster%20Fairy%20Demo%201.ipynb

And some example results:


Which type of regression has the best predictive power for extrapolating for smaller values? I have a data set which deals with response variable in the order of 10-20. The scatter plot for such a regression appears linear, but the problem being when I predict for test cases using values very small compared to the trained sample the predicted response variables appear in negative values. Please note that the values of the data set predicted should not be negative.

Here is the 3d scatter plot of my data

Is there a type of regression which has better predictive power which can perform such operations without such an error?

OK, so you apply linear regression to create a model for your data, and when you use that model to predict new values, the output values don't satisfy a constraint (namely, being positive). I can think of only a few different things that might be going on here:

The new input values you are giving to the model are not within the allowable range for the problem. This is unlikely given your setup - presumably you know your situation well enough to determine what input values are allowed - but if this does happen to be the case, you need to change the way in which you obtain inputs.
A linear model is not accurate over the full range between your original input data and new input data. Without more domain knowledge, it's impossible to tell what sort of model would be a better substitute. If this is a situation where you can come up with some sort of theoretical prediction about what general form of model should describe the data, you should use that form. If not... try an exponential I guess?
A linear model is accurate, it's just not this particular one. You might have to do something like forcing the regression to include a particular point (e.g. the origin), for example. That would mean that you're not actually doing a full linear regression, but rather fitting a model with one fewer free parameter - e.g. in slope-intercept form, you're basically forcing the intercept to zero and only fitting the slope. Or the equivalent if using another point. This should be doable with some regression library that you can use. But if you use a linear model, it will predict negative values somewhere. You'll have to live with declaring those regions of parameter space to be outside the domain of the model (or possibly finding a way to make sense of negative values).

Based on the little information you've provided here, there's really not much more I can say. Maybe if you described your situation in more detail, I could offer more specific advice.


Optimal selection of k in K-NN I am currently reviewing some concepts related to Machine Learning, and I started to wonder about the hyperparameter selection of K-NN classifier.
Suppose you need to solve a classification task with a number of classes equal to M: I was thinking that the best choice for the parameter K of our classifier should be for K > M.
In this way, we are avoiding all the pathological cases in which a sample may be in the middle of all the M classes and then have a tie. For instance, consider the following example in which we have M=3 and each geometrical shape represents a class:

Assume that K<=M: for sure you will have a tie for a sample in the middle of samples 1, 2 and 3. This tie could be avoided if K > M.
Clearly this is just a toy example, but I think it is sufficient to illustrate my thoughts. I have tried to look for an answer but I wasn't able to find any resource mentioning this, am I wrong in some way or this reasoning may be sound?

Your idea isn't wrong, however in k-NN there always might be a case where you have the same number of votes for 2 or more classes (e.g. you have $k=6$ and you have 3 samples of one class vs 3 of another). With your solution you are just overcoming one very small case of ties and the $k$ that you choose might not be the optimal $k$ for classification, which is your primary objective (i.e. you might hurt your classification performance with a sub-optimal $k$ so that you don't get one type of ties).
Because there are also other times of ties, you always need to have a tiebreaker condition. I have seen a lot of strategies for this, like reducing $k$ by one until you break the tie, selecting the class with the minimum distance, selecting the class with the fewest samples overall, etc.
Since you already have such a condition, you could also handle your situation, without selecting a sub-optimal $k$.


Suggestions for labeling data for named entity recognition Is it good to label the data based on sub category than parent category? For example: for drugs data ... label the drugs dose as drug_dose or label the drug dose as different type of dose like injection, capsule, spray..which is a better option?

While labeling, label with sub-categories. This enables you to work with larger set of tools / algos.

With Algos that work better with linear labels (and don't know about label hierarchies), you can pre-process to transform and use only least granular labels (E.g.: CoreNLP)
With some of Deep Learning models, you can use sub category tags as it is.

Since tagging / labeling is expensive and (potentially one-time) exercise; it is better to tag more granular categories.


Sequential pattern mining in SPMF: Giving words as input My dataset is the result of multiple joins from large transactional database stored in a MySQL database. There are some columns with text value and double values.  I am trying to do a first level sequential pattern mining on this dataset using SPMF.

Is it possible to give words as input in SPMF? The documentation tells that only positive integers are allowed.
If I want to give words or decimal numbers, which software/package shall I use?

Thanks.

Thanks for using SPMF. I'm the founder of that library. Currently, SPMF does not support text files containing some text as input. Thus, users need to do some preprocessing to convert the text file to the format used by SPMF.  But in the future, it is my plan to add the feature of handling text files natively in SPMF.  It will likely be added in the next few weeks, depending on my schedule.

By the way, if you have any question about SPMF, you can also directly ask them on the forum on the SPMF website. Then, I will answer you more quickly.


What is the classical way to visualize 3D filters in convolutional neural networks? Suppose that at a layer $N$ within a CNN, my "image" is a 200$\times$200$\times$10 array. Thus, if I convolve such an array with, for example, 15 filters of size 3$\times$3$\times$10, I will end up with a new "image" whose shape is $A\times{B}\times{15}$ where $A$ and $B$ depend on the stride of the convolution.

I am thus wondering how such 3D (3$\times$3$\times$10 in my toy example) convolution filters are usually displayed as 2D images (i.e. sum or mean, ...)?

With CNNs, a common strategy is to visualize the weights. They are usually most interpretable on the first CONV layer which is looking directly at the raw pixel data, but it is possible to also show the filter weights deeper in the network.

If you want to have a look at the activation maps, you will have to display them 1 by 1 and they will be grayscale (e.g. you would display 15 AxB grayscale filters for your layer N). Check this site for more ideas: http://cs231n.github.io/understanding-cnn/

Having said that, it is unclear what you can expect to "see" on layers that are looking at a 200*200*10 cube and what action you can take afterwards.

EDIT - Additional link
This article describe a new technique that tries to show the "meaning" included in the neural network layers: https://distill.pub/2018/building-blocks/ "For instance, by combining feature visualization (what is a neuron looking for?) with attribution (how does it affect the output?), we can explore how the network decides between labels like Labrador retriever and tiger cat."


non-separable assignments of the vertices of a hypercube I have a question regarding exercise 14.17 in An Introduction to Information Retrieval by Manning et al.

The problem is:

  "Assuming two classes, show that the percentage of non-separable
  assignments of the vertices of a hypercube decreases with
  dimensionality M for M > 1. For example, for M = 1 the proportion of
  non-separable assignments is 0, for M = 2, it is 2/16. Solve the
  exercise either analytically or by simulation."

The total number of assignments of vertices of an N-dimensional hypercube is: $2^{(2^N)}$

And as I found in here the number of separable assignments is O($2^{(N^2)}$).
So the percentage of non-separable assignments is increasing with N (which is the opposite of what is said in the exercise).

What am I missing here?

You appear to be correct. I believe this is a simple proof of the opposite fact:

Let $s(n)$ denote the number of separable sets in the $n$-cube. Let $Q_0, Q_1$ denote the subcubes of one smaller dimension, with say last coordinates equal to 0, 1 respectively. The key fact is that any separable set of the whole cube intersects with each of $Q_0, Q_1$ in separable sets. (The separating hyperplane intersects the codimension-1 spaces to form separating hyperplanes.) So
$$s(n)\leq [s(n-1)]^2.$$

Now the proportion $r(n)$ satisfies
$$r(n)=
\frac{s(n)}{2^{2^n}}\leq
\frac{[s(n-1)]^2}{(2^{2^{n-1}})^2}=
[r(n-1)]^2<
r(n-1),$$
where the last inequality holds because $r(n-1)<1$ for $n>2$.


What is the difference between BERT architecture and vanilla Transformer architecture I'm doing some research for the summarization task and found out BERT is derived from the Transformer model. In every blog about BERT that I have read, they focus on explaining what is a bidirectional encoder, So, I think this is what made BERT different from the vanilla Transformer model. But as far as I know, the Transformer reads the entire sequence of words at once, therefore it is considered bidirectional too. Can someone point out what I'm missing?

The name provides a clue. BERT (Bidirectional Encoder Representations from Transformers): So basically BERT = Transformer Minus the Decoder
BERT ends with the final representation of the words after the encoder is done processing it.
In Transformer, the above is used in the decoder. That piece of architecture is not there in BERT


What is exactly meant by neural network that can take different types of input? There is a scientific document that implements a convolutional neural network to classify 3 different types of data, although how exactly, is unknown to me.

Here's the explanation of network architecture:

  This section describes architecture of our neural net which is
  depicted in Fig. 3.
  Our network has three types of inputs: Screenshot (we use upper crop of the page with dimensions 1280 × 1280, however this net can
  work with arbitrarily sized pages), TextMaps (tensor with dimensions
  $128 \times 160 \times 160$) and Candidate boxes (list of box coordinates of
  arbitrary length).

  A screenshot is processed by three convolutional layers (the first two layers are initialized with pretrained weights from BVLC
  AlexNet). TextMaps are processed with one convolutional layer with
  kernel size $1 \times 1$ and thus its features capture various
  combinations of words. These two layers are then concatenated and
  processed by final convolutional layer.

What exactly is implied by Our network has three types of inputs above? Is it possible for convolutional neural network to pass different types of inputs differently?

From my understanding, neural network for Screenshot input would be created like this:

def CNN(features, labels, mode):
  input_layer = tf.reshape(image, [-1, 1280, 1280, 1])

  # Conv+ReLU
  conv_relu_1 = tf.layers.conv2d(
      inputs=input_layer,
      filters=96,
      kernel_size=[11, 11],
      padding="same",
      activation=tf.nn.relu)

  # MaxPool
  pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[3, 3], strides=2)

  # Conv + ReLU
  ...

So let's say this is first neural network, then should I create another neural network for TextMaps and concatenate results? Or does every magic just happen in a single neural network?

In short, can I create neural network that takes different types of input individually or do I use different neural networks for each of them and then group their outputs?

Thank you!

In short, can I create neural network that takes different types of
  input individually or do I use different neural networks for each of
  them and then group their outputs?

Yes, you can. Check the Functional API of Keras, on how to define multi input/output networks. Then you can create different models for the processing of each input and fuse them together into a single multi-input model using the keras.models.Model() class.

In the following example, you can see that the main_input is processed differently than the aux_input and both are thereafter merged together to be propagated through the rest of the layers of the network.


NA in LR model summary(R) So, i was trying to improve mr LR model performing multiple linear regression on a dataset. I had a categorical variable region

Region(variable):
Midwest
Northeast
South
West

I made dummy variable for each of them and it did improved my model a bit.

Previous Model Summary

After adding these variable(Which i made using a different variable)

I am getting NA's in the coefficient for west, which i don't understand why. can someone explain?

You've given all four regions a dummy variable, so these are perfectly multicollinear, and the (unpenalized?) regression doesn't have a unique solution.  R automatically drops a column in this situation and reports the NA.

https://stats.stackexchange.com/q/212903/232706
https://stackoverflow.com/q/7337761/10495893
https://stats.stackexchange.com/q/25804/232706

(You can see where R calls C and subsequently FORTRAN functions, then inserts the NAs at
https://github.com/wch/r-source/blob/0f07757ad10ca31251b28a2c332812e63c0acf38/src/library/stats/R/lm.R#L117
A nice article that helped me find that: http://madrury.github.io/jekyll/update/statistics/2016/07/20/lm-in-R.html )


RMSE in Weka Time Series Forecasting I am using Weka Time Series Forecasting to forecast the trend of the topic NLP in 2018. For that I used evaluate on training option in Weka to obtain Root Mean Squared Error (RMSE). The RMSE value I obtain is 0.785.

What does this imply?
Is there a threshold value (e.g., RMSE < 0.2) that we can consider to
accept/reject this prediction?

RMSE(Source)

  The RMSE is the square root of the variance of the residuals. It indicates the absolute fit of the model to the data–how close the observed data points are to the model’s predicted values. RMSE is an absolute measure of fit. As the square root of a variance, RMSE can be interpreted as the standard deviation of the unexplained variance, and has the useful property of being in the same units as the response variable. Lower values of RMSE indicate better fit.
  RMSE is a good measure of how accurately the model predicts the response, and is the most important criterion for fit if the main purpose of the model is prediction.

The best measure of model fit depends on the researcher’s objectives, and more than one are often useful.

  The RMSE for your training and your test sets should be very similar if you have built a good model. If the RMSE for the test set is much higher than that of the training set, it is likely that you are badly overffiting the data.


Clustering with 0 or Null values I want to do some clustering for a dataset where I am looking at 10,000 peoples usage of certain electronic devices. I have 11 columns; the first column is simply a URN representing each person in the study. Then the other ten columns are for the weekly usage, in minutes, of each device (e.g. phone, laptop, t.v. etc).

No individual actually has all ten devices. They normally tend to have 2-4. For this reason they will have a value of n for each of the devices they use, and then a "0" if they do not have said device. Therefore, I obviously can't omit rows which have a 0 in them, and the 0 does not necessarily mean missing data, as opposed to actually meaning they do not own the device.

I am wondering how a typical basic cluster analysis would be carried out in a situation like this? / what would be an appropriate way to approach such a study? (I am using R for the analysis, for reference)

Thanks for the help.

K-means is the wrong algorithm for this problem.

It assumes dense, continuous input data, which you haven't.


Machine Learning: Balanced training set but highly unbalanced prediction set? How to adjust? I am trying to train a model to detect gender in a dataset of CEO speeches. Here are the datasets that I have:

Final Dataset: 20K CEO voices analyzed (around 95% male)
Testing dataset (?): 1K CEO voices analyzed from the final dataset, less unbalanced because I added more females on purpose (80% male)
Training dataset: 6K voices analyzed from audiobooks and TED talks (55% male).

For now, I have been trying different models by training and splitting dataset #3 (70%, 30%). I get good accuracy (95%) using this method. However, when I apply the trained model to dataset #2, I get an accuracy of 85%.

I am not sure what to do. Should I undersample women in the training dataset so that its distribution is more similar to the final dataset?

Over- or undersampling should be your second choice. Currently, the best method to deal with class-imbalances is to use the weights argument which sklearn- and keras-classifiers support (see for example the DecisionTreeClassifier).

On a general sidenote, I would recommend focusing on the f1-score,  AUC, and the confusion matrix to evaluate the model's performance, as accuracy might be not as informative in this case.


GridSearch on imbalanced datasets Im trying to use gridsearch to find the best parameter for my model. Knowing that I have to implement nearmiss undersampling method while doing cross validation, should I fit my gridsearch on my undersampled dataset (no matter which under sampling techniques) or on my entire training data (whole dataset) before using cross validation?

Do grid search on the same Level of "imbalancedeness" that you plan/are able to do your Training and Evaluation on.
So that means that if you saw that imbalanced data set does not skew your model predictions or results in other unwanted Outcomes, done use the maximal dataset possible. But on the other Hand if your model is strongly overfitting because of imbalanced dataset then optimisation with grid search will make him overfit more in that direction.


Machine Translation using seq2seq model I am trying to perform Hindi To English translation using s2s model, following - https://chunml.github.io/ChunML.github.io/project/Sequence-To-Sequence/

I have used https://github.com/karimkhanp/Seq2Seq/tree/master/data dataset which contains 37726 sentences for training.

Training took around 44 hours on my 8GB machine. I considered 3 layers, 10 epoch, 20000 vocap size

Other parameters

ap.add_argument('-max_len', type=int, default=200)
ap.add_argument('-vocab_size', type=int, default=20000)
ap.add_argument('-batch_size', type=int, default=100)
ap.add_argument('-layer_num', type=int, default=3)
ap.add_argument('-hidden_dim', type=int, default=1000)
ap.add_argument('-nb_epoch', type=int, default=10)
ap.add_argument('-mode', default='train')

But now when I test using hindi sentence, it give UNK for all words. Though I take same sentence for testing as in training even it says UNK in results.

Test sentences:

डेली हिन्दी न्यूज - बुंदेलखंड का प्रथम अन्तरजालीय स्थल
मैं भारत से प्यार करता हूँ
ज़बूर जो कि दाउद को प्रदान की गयी
प्रशासनिक विभाजन
वे इस देश के प्रथम UNK -LRB- अफ्रीकी UNK -RRB-
नेपाली विदेश
वेल्श खिलाड़ी इंग्लैंड के लिए खेलने के लिए पात्र हैं
उनके बड़े भाई अजीत तेंडुलकर ने उन्हें खेलने के लिये प्रोत्साहित किया था ।
फिर एक मिनट के बाद किताब छत की ओर उछालकर उन्होंने कहा - चलो ।

Result :

of of of
of of of
the of of of

lrb lrb rrb rrb

the the of of of
the the of of of of
the the the of of of of

I could not understand the issue behind this result. I have used some sentences which are same as training sentences. Atleast answer should be correct for them.

Did I do anything wrong? I really appreciate any help

Answer -

if MODE == 'train':
    k_start = 1
    t1 = time()
    # If any trained weight was found, then load them into the model
    if len(saved_weights) != 0:
        print('[INFO] Saved weights found, loading...')
        epoch = saved_weights[saved_weights.rfind('_')+1:saved_weights.rfind('.')]
        model.load_weights(saved_weights)
        k_start = int(epoch) + 1

    i_end = 0
    for k in range(k_start, NB_EPOCH+1):
        # Shuffling the training data every epoch to avoid local minima
        indices = np.arange(len(X))
        np.random.shuffle(indices)
        X = X[indices]
        y = y[indices]

        # Training 1000 sequences at a time
        for i in range(0, len(X), 1000):
            if i + 1000 >= len(X):
                i_end = len(X)
            else:
                i_end = i + 1000
            y_sequences = process_data(y[i:i_end], y_max_len, y_word_to_ix)

            print('[INFO] Training model: epoch {}th {}/{} samples'.format(k, i, len(X)))
            model.fit(X[i:i_end], y_sequences, batch_size=BATCH_SIZE, nb_epoch=1, verbose=2)
        model.save_weights('checkpoint_epoch_{}.hdf5'.format(k))
    print("Time taken to train the data in hour=>", (time()-t1)/3600)

# Performing test if we chose test mode
else:
    # Only performing test if there is any saved weights
    if len(saved_weights) == 0:
        print("The network hasn't been trained! Program will exit...")
        sys.exit()
    else:
        # import pdb
        # pdb.set_trace()
        X_test = load_test_data('test', X_word_to_ix, MAX_LEN)
        X_test = pad_sequences(X_test, maxlen=X_max_len, dtype='int32')
        model.load_weights(saved_weights)
        # print(X_test)
        # print(model.predict(X_test))
        predictions = np.argmax(model.predict(X_test), axis=2)
        # print(predictions)
        # print(y_ix_to_word)
        sequences = []
        for prediction in predictions:
            sequence = ' '.join([y_ix_to_word[index] for index in prediction if index > 0])
            print(sequence)
            sequences.append(sequence)

Clearly, the optimization of your loss function led to it only predicting the most common words in English. The probability of other words are obviously small.
I am no expert, but I would check three things. 1. How long does it take for the loss to converge? With the output you show it is likely it does not take a long time. 2. Maybe try to filter out so you have different words in the sentences. 3. Check if it can help to optimize another loss function which would have a bias towards more diverse sentences.
I hope this can give you a few ideas at least.


Can I install Tensorflow and Keras on Cloud? I will like to install Tensorflow and Keras on my PC. I use 32 bits OS. I learnt Tensorflow is not compatible with 32 bits. I cannot upgrade my OS to 64 bits since my hardware does not support it. I have learnt one or two things about cloud computing but I only know about how I can use it for storage purposes.
My question is, can I install Tensorflow and keras on cloud and somehow use them on my PC? Or is there a way round this as I cannot afford to get another PC for now and I cannot afford to stop learning as well?
NOTE: I use Windows 7 32bits
NOTE: I read about how one could install Tensorflow from source but the writer discourages it as he says it won't be worth the headache in the long run as the software would not be able to implement many programs because of space.
NOTE: I have heard about Pytorch but I don't think it is supported by 32 bits. In addition, the materials I am using for learning machine learning only focus on Tensorflow.
Thanks.

I think your best option is to use the cloud, since your PC is not capable, and has limited resources.
You can set up a Google Colab or Kaggle account to run Jupyter notebooks online. They will support most libraries, including Tensorflow, Keras, and Pytorch. Both will also provide limited GPU resources.

Google Colab has a free version you can use, just need to keep an eye on your usage.
Kaggle is free, you don't need to enter a competition to use the service. Kaggle also has discussion boards and public notebooks to look at for ideas.


Why standard distribution for ML Data normalization: It ensures that each input (each pixel value, in this case) comes from a standard distribution.
This standardization makes our model train and reach a minimum error, faster!
my question is about standardization, I don't understand the relation between input data being between 0 and 1 to gradient decent being faster or even being correct(predicts well), why is that, I'm very ok with a deep math proof.

The standardization of the pixel values comes from the same logic standardization data to make Gradient Descent (or some other similar optimization algorithm) faster. When you provide an image to the NN, it is no different from other data types, meaning you provide a sample with different features (or let's say variables). When an image is a case, your features are your pixels. That is, if you have a 32x32 image this means you have 1024 features. If your input is RGB, it creates 3 dimensions which make your feature size 32x32x3 that is, 3072. In other words, your variable in such a case is a specific pixel of all the images in your dataset. Your image is just a single row of data with 3072 features like in a simple regression problem where for example, house price is predicted using 5 features location, size, age of the building, number of rooms, and floor.
Now, before going into details, let's first think why distribution might be different. Depending on your image type, your specific channel values may have a bigger variance than others. For example, if your data samples, images, are about plants, trees, or forest then your Green and Blue channel values will probably vary a lot, but your Red channel will always have a value close to 0. Thus, the features that represent these channels will have a different distribution than others.
Another thing that needs to be considered is pixels' (even if it is a grayscale image) having dissimilar distribution due to natural reasons. For example, consider the dataset where all images are identity card photos of people, where the background is white. In such a case, some of the pixels, probably the top left and top right ones will have a variance close to zero due to having almost the same color value for all images.
These two cases cause the possibility of a specific pixel having, for example, values 0-10, another one having 3-240, another one have 126-255, etc. In such a case, what needs to be handled is the same problem as Andrew NG talks about in this video. Since the ranges of your values will vary in 3072 (for example) dimensions, gradient descent will need more steps to find optimal value in one dimension when compared to the other dimension. This is because your step size is fixed. Consider, you have 100 rungs in one ladder and 12 rungs in another ladder. With normal human step size 1, which one would be faster to climb? The latter.
To briefly summarize, since for all features in the NN same parameters are used, to expect those parameters affect the features in the same way (or objectively), these features should display similar characteristics. If not, some features might advantage more from these parameters when compared to others.


Dealing with population instability I am working on using machine learning to correctly predict a binary classification using an input dataset that I receive about once a month.

The idea is that I train, test and validate the classifier on one month of data, and use it to classify another month. Whether I repeatedly add the next month of data to the training dataset and upset my classifier is open for debate.
I've spent a little while on this and have something that works well but it works better on some months than others. I am using either RandomForest or LinearSVC(penalty=’l1’) because I have a lot of features, around 400,000 observations and limited computing power.

I do not have much insight into the processes that produce the input dataset and based on previous analysis, I do not trust their quality control. What worries me is that one month I might receive a dataset that is very different to the data I have been training on and I wouldn’t really know unless the distribution of classes was quite different.

I am sure that I cannot be the first person to have this worry but I can’t find much information about it, so maybe I am misguided here. I’ve found the Population Stability Index, which is the right kind of idea, but I am unsure about it because it is dependent on bin size.

So my question(s):

How is a worry like this normally dealt with?
Can you point me to some ideas/case studies that will help me?
Is there anything I can do on the classification side that will make
my classifier more robust to population instability?

There could be significant problems from using one time chunk in a time series to predict the next chunk. For example, if you were using data from September and October to predict retail shopping expenditure in November and December, you would be very wrong. Instead, you need to note the annual trend of increased shopping between Thanksgiving and Christmas to make correct predictions.

So, while this sort of skirts the questions you asked, I think that if you are having difficulty using one month to predict the next month, you need to start looking at longer-term trends, especially annual ones. I would investigate this first before looking into population instability.


In supervised learning, are more data entries always better? I am doing a supervised learning problem and have 600,000 rows of data. I divided it into a training and test set and achieved a high accuracy that I was happy with. However, I had thrown away 300,000 entries as they contained significant missing data. When I redo the analysis except use mean, median, and mode imputation methods to fill in the missing data entries and repeat the training/testing - my accuracy drops by 4%.
Why is this? I thought more data would be better or at the least stay the same. Does this imply the imputation was inaccurate? Or perhaps I was biasing the sampling process by only selecting records with no missing entries the original time? How could I possibly know which case since I have no way of knowing how accurate the imputed values are with the true values that were missing?

That is a clue your data is missing-not-at-random.
The missing data is a result of a systemic process that also effects predicting target performance.


Appropriate Machine Learning algorithm for modeling clustered time-varying binary outcome I'll just dive right in. I have a decent-size (100K observations) dataset of time-varying continuous and categorical predictors. Categorical predictors, actually, usually do not change, however, continuous one change every day. Another level of complexity - the one that I am struggling with - is the fact that the data are clustered at several levels (measurements coming from the same individual over time, with multiple individuals in the data set).
So, I have something like:
id | day | cont_predictor | cat_predictor | daily_outcome
1  | 1   | 1.4            | 1             | 0
1  | 2   | 1.7            | 1             | 0
1  | 3   | 1.9            | 1             | 0
1  | 4   | 2.9            | 1             | 1
2  | 1   | 4.0            | 0             | 0
2  | 2   | 4.1            | 0             | 0
3  | 1   | 5.7            | 0             | 0
3  | 2   | 4.2            | 0             | 0
3  | 3   | 3.5            | 0             | 1

I am looking for advice on what algorithm is best suited for modeling daily_outcome. This variable is highly imbalanced (30:1) but also the observations are time-varying and are clustered on an individual level (id). I could work with these data using a mixed effect model for longitudinal logistic regression. However, I am most interested in optimizing prediction, therefore I'd really like to use a machine learning approach.
So far, I have tried logistic regression and random forest, to start with, however, they are not performing well when predicting daily_outcome = 1, even after balancing and oversampling techniques including SMOTE. I am assuming it's because these models do not account for the structure of the data, where observations coming from one individual are highly correlated (I've examined those - very high intraclass correlations).
What are some machine learning algorithms that worked for you in the past? Looking for advice from people who worked with this kind of data (seems like this would be a fairly common problem to work on, but it's my first time working on it).
Thank you so much in advance!

At first, choosing the best model is probably not as important as getting the data-set setup properly. Most of data science and modelling is spent on building the best data-set to use. Random Forest Classification, SVM, XGBClassifier, even Logistic regression can perform adequately if given the properly prepared data-set to analyze. Also, the metric you are comparing you model to should be appropriate for imbalanced data. Just a FYI, accuracy is not one of them. Here are just a few suggestions to get you started:
Data preparation/cleansing:

you can try more methods to balance, such as down-sampling (think
this is usually better than up-sampling as the data you are using
was real data, not synthetic). SMOTE is okay also, but can be a bit
more complicated to use properly
you can try using a weighting function in your model to give more
importance to the minority value, if you think a mistake in
predicting that value is more important or more costly than a
mistake in the majority, or just want to balance your predictions
evenly between the classes
You can use PCA, SVD or LDA to remove some of the correlations
between your features
Normalize the data as some algorithms still don't handle large
differences in features very well (ex: feature one mean is .001 vs
feature 2 mean is 1000). This is a good practice in any modelling.
Look for fliers. Sometimes fliers can cause problems. Look for
values that don't make sense, or are too far out of normal range.

Metric to use:

I usually use ROC_AUC to find which model is best at predicting both
majority and minority classes.
Sensitivity and Specificity are also 2 good metrics to use to see
how good your model is at predicting each class


How to find similarity of two series over time containing periodic trends? Considering the data is received from a streaming source each second.How to distinguish if both the line graphs 'look' same/different in real time, statically, like the picture given below

Edit:
1.Not sure if there is something as real-time  periodic correlation mechanism then cross-correlation would perhaps be an ideal solution ?
2. Comparing slope of the two Line is the last option I would go with.
3. If statistically there is no way to solve this then I would look at machine learning to solve this

Maybe it is a huge oversimplification, but you can try: What if you do just arithmetic difference between the two signals (or divide one signal to another one) and look at the result?

I would expect that you will have spikes where signals become very different and you can make a threshold to find where they are on time.


Interpreting the evaluation result of multiple linear regression I am learning the multiple linear regression model.
I've built a model and using R command:

summary(model)

I got this result:

Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 253.2 on 44 degrees of freedom Multiple
R-squared:  0.3336, Adjusted R-squared:  0.2579  F-statistic: 4.405 on
5 and 44 DF,  p-value: 0.002444

How can I interpret this result in order to have a decision regarding the goodness of the model? In specific, what is the 44 degrees of freedom means for this case?

Also, why do we have adjusted and multiple r squared parameters?

I am going to answer your questions one after another.

First, what do the 44 degrees of freedom mean?

It simply means that the model you built is constructed by using 44 independent variables. For example a model that looks like y = ax + b has 1 independent variable (i.e. a) and thus 1 degree of freedom. A model that looks like y= ax1 + b*x2 + c would have 2 independent variables (i.e. a and b) and thus 2 degrees of freedom.

Second, what is the Multiple R-squared?

Here for the purpose of interpretating it Multiple R-squared is equivalent to the (simple) R-squared you would have for a linear regression model with 1 degree of freedom. Multiple R-squared tells us the share of the observed variance that is explained by the model. For example if you have a Multiple R-squared of 0.79 it means that your model explains 79% of the observed variance in your data.

Third, what is the Adjusted R-squared and why do we need it?

There are several problems with Multiple R-squared.

Problem 1: Every time you add a predictor to a model, the R-squared increases, even if due to chance alone. It never decreases. Consequently, a model with more independent variables (more degrees of freedom) may appear to have a better fit simply because it has more independent variables.

Problem 2: If a model has too many predictors and higher order polynomials, it begins to model the random noise in the data. This condition is known as overfitting the model and misleadingly high R-squared values and a lessened ability to make predictions.

Problem 1 is caused by Problem 2. And this is where Adjusted R-squared is coming in handy. Adjusted R-squared is an attempt at fixing these problems by factoring in the number of independent variables. The adjusted R-squared tells you the percentage of variation explained by only the independent variables that actually affect the dependent variable.

where:

n is the number of data points you have,
and k is the number of independent variables used to explain their distribution, excluding the constant

If you add more and more useless variables to a model, adjusted r-squared will decrease. If you add more useful variables, adjusted r-squared will increase.
Adjusted R-squared  will always be less than or equal to R-squared . You only need R-squared when working with samples. In other words, R-squared  isn’t necessary when you have data from an entire population.

Here is an interesting series of articles which will help you understand how to use R-squared to interpret the results of your model even better.

Regression Analysis: How Do I Interpret R-squared and Assess the Goodness-of-Fit?
Multiple Regression Analysis: Use Adjusted R-Squared and Predicted R-Squared to Include the Correct Number of Variables


Early stopping and bounds Say I am training neural networks using a train set and set aside a validation set V. I obtain models h's after each epoch along with the validation losses(0-1 loss) $\hat{L}(h_1,V)$, $\hat{L}(h_2,V)$ ... if I use the early stopping rule suggested here(top answer).

Is the resulting $\hat{L}(h_*,V)$ an unbiased estimate of the true loss?
How can I bound the true loss using $\hat{L}(h_*,V)$ ?

I'm guessing no for the first one since the stopping rule depends on the partition of my data set.

Afaik the bounds that can be applied depends on the size of my hypothesis set and I'm not entirely sure if it's finite, countable or uncountable in this case.

Is the resulting $\hat{L}(h_*,V)$ an unbiased estimate of the true loss?

No. You have taken multiple measurements, each with some uncertainty, and chosen the maximum or minimum value.

  How can I bound the true loss using $\hat{L}(h_*,V)$ ?

In the general case you cannot. It will depend on how much over-fitting is occurring within the model on the training set, size of cv set, amount of times it has been used, and how similar the model's performance was on each use. There is also sampling bias in the cv set, and that interacts with the selection process.

What is generally done if you need unbiased estimate at the end of production is a train/cv/test split. The cv set is used for model selection, and once you have a single model selected, you estimate its loss - or other key metric - on the test set. It is important to use the test set minimally and not in order to select models, if you want it to be an unbiased measure. Otherwise you repeat the problem.

Another approach which maintains confidence in cv-based metrics is to use k-fold cross validation. Taking taking the mean of a metric in k-fold cross validation is still biased once you have used it a few times, but the bias is reduced somewhat. You can take that idea further with nested cross-validation, which allows you to get an unbiased estimate of model performance in a general fashion (i.e. using the same hyper-parameters) from more of your data.


How has data generation evolved over the years (and centuries)? I would like to know how data generation has evolved through the years.

My end goal would be to generate a logarithmic line plot showing the ridiculous increase in data generation. Please, does someone have a reference for this? I couldn't find any. It's OK if you point me to a book (available on eBay) that has this information.

Thanks!

Edit: when I talk about data generation, I talk about combined information generated by both machines and humans. I know this is not very well defined, but, for example, before the inventions of computers, you can consider books and movies.

You could search on Google Trends all the mentions to generative models. Similarly, you can do something like this on ArXive papers and/or Google Scholar.


Artificial Neural Networks and Efficient Parameter Optimization I have a bunch of test measurements data and a semi-empirical model that has 18 parameters which I have to find so that the model fits my data well. So far I've managed to find and optimise the parameters using Optimisation and Global Optimisation algorithms in MATLAB.

Now I would like to explore different approaches for the parameter estimation. I have read some papers where the approach with NNs is described. I am new to NNs and have no idea if this is even possible.

I would create a two layer network with 18 input and output neurons. I am not sure what kind of transfer function would be appropriate for the problem.

The formula I have to fit and find the parameters look like this:

$ y = D \sin(C \arctan(Bx - E(Bx - \arctan(Bx))))$

where $B, C, D, E$ are macro-coefficients and the micro-coefficients are used to express the variation of each of the primary coefficients with respect to some other data.

This is how my data looks like.
How would you create network in MATLAB for this problem? Can you give me some hints and a push in the right direction to tackle this problem?

Thanks in advance.

If you were fitting a large number of different models, and you had sequences of training data for each different model (with the params already known in those cases), then you might be able to use a recursive neural network (RNN) to provide param estimates for new data sequences. However, that does not seem to be your situation.

As I understand your question, you have a mathematical model with some free parameters, and one set of data that you would like to use to estimate those parameters.

A neural network is not really of any use to you here. That is because the neural network would replace your semi-empirical model with the NN, and the free parameters become the weights of the NN. The trained NN would not match your desired model, but would generically fit to the data. It could be used to predict more function outputs given the inputs - the NN might even do better than your preferred model, which would be an interesting result implying that your model is incomplete.

Often we don't know (or perhaps even care) about an underlying parsimonious model. In this case, ML techniques that fit params of a generic model to data using some objective function like squared error can be very useful tools. However, when we do have a good idea about a simple or explanatory  model, those same techniques are less useful, and instead we can use optimisers direct on the model - in some cases these could be the same optimisers used to train neural networks, provided the model is differentiable.